All you say is true. I suppose except if IPO has a big pop then obviously you can miss out on that but for a few days here and there.
In my own case I described above, yeah I was actually very envious of the pre-IPO crowd (almost destructively so) when I joined, but my envy eventually calmed down out of exhaustion -- after the Nth meeting with a billionaire or centi-millionaire, and after you meet the Nth person like yourself who missed out on the rocket-ship, and after you turn down the Nth supposed rocket-ship that flames out (bullet dodged!) you kinda lose the ability to care much about what-could-have-been.