Preheat the oven to 400°F (200°C) and position the rack in the middle.
Bring a large pot of salted water to a boil and cook the jumbo shells until al dente according to package directions; drain in a colander and let cool slightly.
In a mixing bowl, combine the shredded cooked chicken, softened cream cheese, lemon juice (about 1 tablespoon), garlic powder, grated Parmesan, and salt and pepper; stir until smooth and well combined.
Grease an ovenproof skillet or baking dish with the olive oil to prevent sticking.
Spoon the chicken mixture into each cooled shell using a teaspoon or small scoop, then arrange the filled shells snugly in the prepared skillet or baking dish.
Sprinkle the grated mozzarella evenly over the arranged shells.
Bake for 20–25 minutes, until the filling is heated through and the cheese is melted; if you want browned cheese, broil for 1–3 minutes while watching carefully.
Remove from oven and serve immediately.