There's whole classes of highly paid engineers whose job is to do this. But they work for old fashioned, boring, companies.
Another part is hiring a breed of test engineers who like breaking stuff and have a knack for it.
It's one of the reasons why I don't trust myself to test things fully, we write the software with all sorts of assumptions in our heads and subconsciously steer away from doing silly things - in that context it's really difficult to aim at a point a zero-knowledge user would hit.
On one of my teams, we had a hard rule where all features must be tested by 2 other team members (and if multiple people worked on a feature, none of them can be the testers). Something like every other case found something questionable, if not an outright broken edge case, that the developer(s) completely missed.
Moreso than any of the security tests I've written, that fuzzing has broken every enterprise API our customers have thrown at it.
I thought this was so dumb it was brilliant QA.
He's the sort to dig as far into the internals of things as he can and then start messing with it (he implemented partial function application for C, for example [1]).
My poor prototypes never stood a chance.