In fact this transition is right in the article: "...paddlefish caviar — a byproduct that for years local fishermen tossed back with fish guts — ..."
Lobster didn't go from inedible to highly desirable because of scarcity. It's because we learned that when you kill the lobster and let it sit around for days it very quickly decomposes and releases ammonia, which tends to taste like cat piss. If you cook it immediately after killing it it actually tastes good.
I believe we actually cook it before killing it (boil alive).
Frozen lobster is also not cooked. Sometimes it's death by freezing, if they're not killed another way first (for instance, frozen lobster tails are obviously killed before being cut). I'm quite certain boiling alive is one of the least common ways it's done, especially with the popularity of frozen lobster.
What's the point of any luxury item? People enjoy it. If you don't, don't worry, you can probably find something else to enjoy instead.
> ...paddlefish caviar — a byproduct that for years local fishermen tossed back with fish guts
Lots of luxury items were once byproducts - it's not unusual and it doesn't diminish the enjoyment people get out of them now. Lobster (as you said), oysters, alcohol, are all examples.