it was a west coast company that moved most operations to the east coast. however, the manager was still on the west coast, and wasn't seeing the day to day issues. In one major instance, I remember seeing that the initial 'customer info' screen was taking a LONG time to load. Like...30-40 seconds. In some cases, it was only a few seconds, but for important clients, it was taking a long time. They could vamp and chat for a bit, but it was getting worse.
The agents were hitting a webserver in california, and pulling HTML back, across a private network, and... it was a LONG time for older clients. It was pulling up the entire client history. For important clients - who did a lot of business - it was pulling back megs and megs of data.
When I looked under the hood, the data was being repeated - an entire block of data was being duplicated inside HTML comment tags - someone had put those in for debugging stuff and never got rid of it. So we were pulling back, say, 35 meg when maybe only 18 was needed. I simply deleted the commented info - it had not been touched in 6 months - commented the reason in the svn commit log, requested a new push go out, and the next day things were hugely faster.
I got my ass kicked. I "went behind someone's back" (untrue) and "jeopardized production systems" (untrue). Other people had seen the code and it was pushed out via normal process. It was simply the original developer didn't like that I'd highlighted something he'd forgotten to do. :(
I then angled for actually using gzip compression but was shot down at the time (no, "there's a bug in IE6 with gzip compression and printing" or something like that), even though the gzip would have brought this down to under 1 meg, and every call would have been 'up' in under 4 seconds.
We were further shielded from ever talking to CSRs, because they "might overload us with requests". It was (and still is) a bizarre rationale. :/
It's stupid behavior as far as the company is concerned, but from whoever you made look bad, well, they stopped that from happening again, didn't they?
They exist to solve real world problems for people, using software as a tool. I don't know how anyone can possibly do that even if they don't actively engage their user base and proactively solve workflow issues the end-users may not even know are technically possible.
Very frustrating to watch entire teams spend years on internal corporate business logic apps and never even once spend time with the teams consuming them. The results are always entirely predictable.
It's really, really frustrating when developers don't realize that. The most valuable thing I ever did at any company, saving just under a million dollars a year, was literally just me talking to three people, realizing that none of them were on the same page, and just talking to them to solve the problem.
More valuable than any code I've ever written. We're problem solvers, not just programmers.