In case of security fixes, you will stay unpatched for a bit less time. ESR is intended for places that really don't like any form of change like rigid enterprises and banks and such. End users should probably steer clear of that unless they have a good reason not to; which they generally don't. Except for a false sense of security.

I personally don't see a good reason to opt out of security changes for any longer than strictly necessary and that's exactly what you do when using ESR. First they get backported by Mozilla. That's after normal users receive them. This takes time. Then testing takes place because they don't want to push out a hasty fix for ESR. This takes more time. And then third party packagers need to pick up the changes and repackage (which is mostly pointless and adds more time). And then eventually it rolls out days/weeks/months after normal users have long received the patches. I don't seen any good reason for such lengthy delays for normal users. In some big companies where they manually review updates for workstations it's a compromise between the extra work and stability. But the tradeoff is timely access to security fixes; which ESR simply doesn't provide.