The fundamental reason why there's such a difference in efficiency between private efforts and government ones is this:
Private companies that are horrible at managing efficiency go out of business. People in private companies who are horrible at managing efficiency lose their jobs.
Government agencies that are horrible at managing efficiency get bigger budgets. Government employees who are horrible at managing efficiency rarely lose their jobs. The GSA scandal is the only one in recent history that has received any kind of real attention; and that's only because the idiots at the GSA made videos the went viral.
If you work for a company like Microsoft or IBM or Bank of America, you work in a bureaucracy at least as dysfunctional and Byzantine as an average US state. The Federal government is a whole other beast unto itself, but there are probably companies as screwy as they are too.
The fact that the private entity purges some folks doesn't make them better.
I assume it's the same outside of oil as well, but I know the most about that sector. Someone elsewhere in this discussion mentions that the NYTimes paywall cost $40m to implement!