I also end up near the back of the plane all the time, because those are the available seats that don't have a higher price tag.
The delay is just a waste, but unfortunately the airlines aren't incentivized to make boarding or deplaning quick, they're trying to upcharge for more expensive seats. All the engineers are sitting at the gate for 15 minutes wondering why everyone's in a rush and paying extra to get "priority boarding" onto a cramped plane, then at the end of the flight, they're sitting in the back of the plane wondering why it takes 15 minutes for everyone to get off the plane, and how this wasteful process could be made more efficient.
But I also think they should put healthy 25 year olds in the emergency exit row for… emergencies, but they’d rather sell those seats to someone that’s often the poorest candidate to execute in an emergency.