Think of it as first class seats giving the optionality to board at any time (not to mention those free drinks and snacks).
Part of the annoyance of boarding isn't just the total time it takes from when they open the gate door until when it's done. It's the amount of standing up, holding your luggage, shuffling forward a bit, and wondering when the person in front of you will move.
Use driving as an analogy. Suppose you get out of a doctor's appointment at 3pm and are meeting someone to see a movie at 6pm. During those 3 hours, the only two activities you can do are kill time on your phone (either lingering in the doctor's office lobby or sitting in the movie theater lobby) and drive. Do you leave at 3pm and avoid rush hour, or do you wait until the last minute and get in traffic? Either way, you're going to wait 3 hours, so you're not saving time, but one is less stressful.
Even though I'm not really a frequent flyer, I generally buy business class when they are on sale. E.g. Beijing to Bali on Hong Kong Express for $900 roundtrip (vs. $500 for economy), or Beijing to LA on ANA for $2000 last minute one way (vs. $1500 for economy, and since I was moving, it was tax deductible at the time). You can find some good deals, especially in Asia, and the experience can be worth it.