You are vastly overstating the difficulty of pivoting X11 access to complete system compromise with the "somehow gets all the right details to work" phrase. It's trivial.
People have literally gotten killed because of this type of attack. Yes, hostile nation-states probably don't care enough about you personally, but there are people for whom this is a life-and-death matter.
The one thing that reduces the impact of this is that it's Firefox on Linux, which is a niche browser on a niche OS, but desktop Linux Firefox is a product that Mozilla officially ships, which means that it's Mozilla's responsibility to protect its users.