Preheat oven to 375 F. Line a baking sheet with parchment pepper.
Heat oil, fennel seeds, cayenne pepper, garlic, zest and olives in a small pot over the lowest heat possible. Cook, swirling occasionally, until you start to see the garlic and spices sizzle and toast in the oil, 5 to 8 minutes. (Watch carefully to ensure it does not burn). Keep cooking at the lowest heat setting until the spices are toasted and the garlic is golden brown, another 3 to 5 minutes. Remove from heat and discard the lemon and orange zest slices. Stir in 1 tbsp honey and set aside.
Unroll the puff pastry on a clean work surface. Spread olive mixture evenly over puff pastry. (A pastry brush works well to spread the oil). Evenly distribute brie over the pastry.
Roll the pastry up as tightly as possible. If there is any remaining oil brush the roll with it. Using a sharp knife, cut into 8 swirls (approximately 1 ¼ inch slices).
Place the rolls on the prepared baking sheet and bake 20-25 minutes or until the cheese is bubbly and the pastry golden (these are a little messy, don’t worry). Drizzle rolls with remaining honey and top with flakey sea salt and lemon zest. Serve immediately.