Comment by lmm | Hacker News Reader

Comment by lmm | Hacker News Reader

One from my first job (not mine though).

Java app, deployed to four servers (by rsyncing a zip file and unzipping it). One of the four fails on startup, in some way that's hard to trace (I think it might have been a JNI crash)? But all four servers have the same OS, same version, same packages installed, same JVM, and it was the same zip file. Check the md5sum on the zip, it matches. In desperation one of my colleagues writes a script to recursively go through the unpacked version of the app and check the md5sums of all the files. Still matches perfectly, and the same files are present on all machines.

We get the dev team to try the app - there's a bit more variety in our devboxes than servers. Two of them can reproduce the failure, but there's no obvious correlation - one's java 1.5, one's java 1.6. One's Debian, one's Gentoo. For every combination there's another developer with a similar machine where it works fine.

Turns out that one server had been installed with a different filesystem from the other three (reiserfs?), which meant that the directory entries for the files were in a different order. The JVM just lists all the classes in the directory and then loads them in on-disc order, so classes were getting initialized in a different order, and it was that that was ultimately triggering the bug.

I recall averting what would have been a rather opaque security weakness by pointing out that occasionally someone would goof and vary the default locale of some JVM instances. (With a particular knock-on effect that I'll omit here.)

Paying attention to the system you'll land on is something that can go lost even to senior developers. And, therein, details, details, details...

Another reason to have a strong ops (or devops) team: Providing/enforcing proper, intended, and thoughtful context and runtime.

P.S. As I now recall, there was also the use case for locale to deliberately vary, although that use case / those instances should have remained orthogonal to those of other locales. Nonetheless, one more possible driver of a mistake that would enable this weakness to occur.

> Another reason to have a strong ops (or devops) team: Providing/enforcing proper, intended, and thoughtful context and runtime.

I have to disagree, actually. This kind of problem is exactly where you really benefit from having full-stack people who understand both sides of the system; it would have been very hard for an ops person who didn't know about Java's quirks or a pure dev who didn't know about the unix filesystem to diagnose.

I see your point. And... I had to escalate to a very senior person of that sort (full stack, or full picture -- in detail) in order to get that particular problem paid attention to and resolved.

(Such a perspective is also how I determined the problem in the first place -- which actually resulted from a botched fix to a prior problem that I'd identified.)

However, despite the variance -- or risk of same -- that I described, in general we had some very capable and dedicated devops people who put a lot of effort and care into our environments. I get rather uncomfortable considering how things would have been had that not been the case.

It's been a while, and maybe I'm mixing my stories a bit. But I left that role and product mix -- which had very significant security requirements and ramifications -- quite impressed with the role those devops folks played in keeping us safe.

Perhaps what I meant by "strong" goes somewhat in the direction of your description of "full stack". Our senior devops people tended to trend in that direction.

And we didn't have "turf wars". Instead, devops was a partner often throughout the development lifecycle. It helped make sure that the final destination was appropriate, safely, and consistently configured. (My "locale" situation aside; and in such an instance, the setting would subsequently receive heightened and sustained scrutiny, putting a curb on unintended variance of the setting as well as fixing the code that such variance would impact.)

Strong in knowledge and ability, as opposed to simply or foremost in an authority to dictate.

I think we agree that it's good to have these skills somewhere on your team, and to work together during development. I just don't think dividing people into "ops" and "dev" is helpful. There it was something like one guy who spent 80% of his time on traditionally-ops stuff and 20% development, another guy who was 70% development and 30% database admin, one who was basically full-time ops but happened to be the maintainer of an open-source library that we used in our products, one who had been hired as a developer but was starting to pick up ops tasks because he preferred that...

This is a very common problem for buggy Java programs that have different versions of the same class on the classpath. The JVM will take the first instance of a class it finds on the classpath, so some orders of the classpath will work and others won't.

The solution is to eliminate any duplicate classes from the classpath and to enforce this in your build process. We use this Maven plugin to do it: https://github.com/ning/maven-duplicate-finder-plugin

Or to properly modularize your application with something like OSGi [1]. No more flat classpath and name collisions will be apparent very early on.

[1] http://www.osgi.org/Technology/WhatIsOSGi

I knew Java was buggy but that's really a desaster.

Reply on news.ycombinator.com