Whisk together 1/2 cup of the teriyaki sauce and 1/2 teaspoon cayenne until combined.
Place chicken thighs in a gallon-sized sealed plastic bag or a glass dish and pour in the marinade, coating each piece evenly. Seal or cover and refrigerate for 6 to 24 hours.
Position an oven rack 6 inches below the broiler and preheat the oven to 400°F. Coat an oven-safe nonstick skillet with the 2 tablespoons olive oil.
Arrange the marinated thighs in a single layer in the skillet and transfer to the oven. Cook for 12 minutes. Discard any remaining marinade used for soaking.
Meanwhile, combine the remaining 1/4 cup teriyaki sauce with 1/2 teaspoon cornstarch in a small saucepan. Whisk over medium heat until it comes to a boil and thickens, about 1–2 minutes, then remove from heat.
Using an oven mitt, remove the skillet, flip each thigh, switch the oven to broil, and place the skillet 6 inches below the broiler. Broil 8–10 minutes until the thighs are cooked through and caramelized with slight char.
Brush or drizzle about 2–3 teaspoons of the thickened sauce over each thigh, sprinkle with chopped scallions, and serve.