I've actually spent quite a long time writing code where, for various legal reasons, you are never going to get direct access to the production data. (And indeed some cases where accessing the data in question would be a serious criminal offence).
Makes you really keen on defensive programming and comprehensive logging - although you do then have to be careful about what you log.
I actally found it kind of fun to try and debug complex systems based only on a log file and QA with production operations folks.
[NB Of course, none of this would be applicable to Facebook]
It just works.
In the last 5 years, we've never hooked a debugger to production instance or given access to a developer.
In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 billion page hits (most of our stuff is backend non OLTP).
That's how it's done.
Now that I think about it, I doubt Facebook is that careful, considering how often they break things.
My whole mantra is don't manage complexity: avoid it.
Design is the key word here. It requires thought, it requires intelligence and it requires multiple people's input.
There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.
Of course, I'm still filled with rage when I find a cargo-cult suite of unit tests. But it's so much easier to convince those people to write smarter tests than it is to deal with the team that has no time to write tests because they're spending all their time debugging.
(Poe's Law says hi.)
Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports).
Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L.
Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops.
Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.
Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume.
Madness, sure, but common madness.
There's your problem...
Makes me wonder what dynamic language you're used to. An uncaught exception in both python and ruby aborts the program, not just the tiny inner loop that happens to be running.
PHP would not throw an exception if the mail function got a bad email address. well, using some mailing libraries might, but the core mailing function (basically just a wrapper for sendmail/postfix CLI stuff) wouldn't.