Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.
Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.
Or when the bug is only triggered in specific user profiles.
Or when all the devs went on a retreat in the mountains with no cell service.
Or when a dev makes a mistake (which we know never happens to even the best devs)
Or when the only developer that knows which one of the 1000 changes that were pushed could be the one breaking, turned his phone off.
Or when a flaw is discovered in the process for the first time (which we know never happens because everyone's process is perfect, until it isn't)
Or how change management's requirement that the fix be tested and verified by all affected teams might have people staying a few hours after 5pm on a Friday when they just want to get their weekend started.
Or how 10 different people from 10 different teams might need to be called and kept to work until 2am because the change can't be pulled because the database was already modified and the old client data is already expired from cache and a refresh would destroy the frontend servers.
Or another reason.
It drives me nuts when people tell me off for saying 'yeah yeah, no, automating our entire infrastructure of 5 servers isn't really worth it right now', like I'm some unprofessional bozo.
I pretty much have experience with all but one or two of your suggested scenarios, and by now I have no patience for annoying software developers who think that using chef or puppet somehow sufficiently embiggens them to run ops on their own (of course dev ops is almost a political assault on existsing ops guys, not merely a nice new solution to existing problems).
Sigh. This is why I don't work on teams these days (if I can help it).
EDIT: Though I also agree with the sub-parent, that deploying on 5pm is fine in certain teams and certain projects, the most important thing is are the guys pushing to do the deploy going to own the deployment? Are they going to hang around for another 60 minutes to check everything is OK? Are they going to be available at 10pm or on Saturday if something goes wrong and are they going to own it? If the answer is no, then nope, don't do it.
Even if it only happens once in your career, once you've had a dev push out code at 5pm friday night, jet out the door and hit the bar, meanwhile you (the sysadmin/ops on call) get woken up at 1am by site down alert, and have to debug/rollback the changes while the dev who pushed them is unreachable, you learn to really avoid friday evening pushes. Fool me once...
Its sad that most places dont have a proper technical copy (with a full copy of live data) to do full tests on TDD is all very well but you need to test the entire system.
No matter how great your processes and your code are, no test can catch everything that can go wrong in a live environment, and doubly not if your system interfaces with anything third party.
But the real reason we deploy weekday mornings is so everyone is on deck and we can get outside help if required. When I was doing system integration, the problem was never in my code, it was the vendor's. Testing can only get you so close to the real world.