Preheat the oven to 400°F (200°C).
Make the marinade: in a bowl combine 2 tablespoons olive oil, 2 tablespoons Dijon mustard, 3 chopped garlic cloves, 1 chopped shallot, chopped rosemary, 1 teaspoon smoked paprika, and a pinch each of kosher salt and black pepper.
Add the chicken to the marinade and toss to coat evenly.
Heat 1 tablespoon olive oil in a large oven-safe skillet over medium-high heat. Sear the chicken until golden, about 3–5 minutes per side.
During the last 2 minutes of searing, add 3 tablespoons butter and the lemon slices to the pan to baste the chicken. Remove the chicken and lemon slices to a plate and reserve any pan juices in the skillet.
To the same skillet, add the orzo, the remaining 1 shallot (chopped) and 1 clove garlic. Cook, stirring, until the orzo is lightly toasted, about 2 minutes.
Add the chicken broth, chopped kale, and 2 tablespoons lemon juice to the skillet; season with salt and pepper. Bring to a boil over high heat.
Slide the seared chicken and lemon slices back into the skillet with any accumulated juices. Transfer the skillet to the preheated oven and bake uncovered for about 15 minutes, or until the chicken is cooked through.
Meanwhile, make the feta sauce: combine 4–6 ounces feta, 1/4 cup plain Greek yogurt, 1 grated garlic clove, the juice of 1 lemon, 1/4 teaspoon smoked paprika, and a pinch of crushed red pepper flakes in a blender. Blend until smooth, thinning with additional lemon juice if desired.
Remove the skillet from the oven. Spoon the orzo and chicken onto plates and top with the feta sauce and chopped fresh dill.