Preheat the oven to 350º and grease a 9-inch cake pan. Sprinkle the brown sugar and rosemary over the bottom of the pan and spread the plum slices over top. Set aside.
To make the batter, whisk the melted butter, buttermilk, eggs, and sugar together. In a seperate bowl, combine the flour, baking soda, and salt. Add the egg mixture to the flour mixture, mix until well combined, then spread the batter over the plums.
Bake until the top of the cake is golden brown and a toothpick comes out clean, 50-60 minutes. Let cool for 5 minutes, then run a knife around the edge of the pan. Put the serving plate on top of the cake pan and flip the pan. Serve warm with ice cream.