Ask HN: How to Handle Impossible Situation
The situation is this; we host the applications on our infrastructure. The company has an infrastructure group that manages the servers, network, databases, etc. We basically throw a .war over a wall and they just take care of it (this is a java shop btw). Due to PII/PHI reasons, development doesn't have access to the prod or UAT environments.
The issue comes when there are problem with a release. We had one release recently that's been very unstable on our largest customers. Everythings fine for smaller customers but when it comes the customers with the really big DB's we have servers going unresponsive, require restarts etc. Needless to say, we don't do load testing, so this was not caught before release. The infrastructure team just rolls back to previous versions if customers complain too much.
It's a real disaster. The problem is the infrastructure team takes no responsibility in trouble shooting or diagnosing the JVM issues, and Dev has no access to do it. If it was me, I'd connect visualvm to a server/jvm and watch it constantly. But I can't, I don't have access and can't get access.
Does anyone have input on this type of situation? What course of action can the development take in this situation? I feel like I'm just whining and pointing fingers, I'd really like to have a solution for this!