Out of control Java processes when dealing with a leap second? | Hacker News Reader

Out of control Java processes when dealing with a leap second?

bugzilla.mozilla.org

87 pointswzm41 comments

Open articleView on HN

Leap second + Java + Linux fix (without reboot):

/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;

-- then restart java

This is what worked for us. Credit to https://bugzilla.mozilla.org/show_bug.cgi?id=769972#c5

A lot simpler:

date --set "`date`"

On a localized box, I had to run "unset LANG" first, because otherwise date doesn't seem to understand its own output.

Thank you! Was banging my head against the wall on RHEL6's tomcat. Went on Hacker News to stop thinking about it for a few minutes. Son of a bitch, the solution was right on the front page.

Where were you 11 hours ago when we rebooted all of our cassandra and mysql machines? :-)

Doesn't work for everyone (Didn't work with our java processes on RHEL 6.2)

fixed it for Twitpic too

you saved the orphanage!

Works wonders for me. Thanks!

I had a lovely morning (2am-4.30) rebooting about 50 boxes by hand. All wearing some shade of java.

Reddit is also down due to Java https://twitter.com/redditstatus/status/219244389044731904

We saw this on every Ubuntu Lucid machine we run. Restarting the JVM was insufficient - we ended up fully rebooting the machines.

Would love to read a root cause analysis of the relevant source code. Anyone have a link?

https://lkml.org/lkml/2012/6/30/122

I, too, am a little curious. Anyone seen thread dumps from any of these floating around?

I didn't really think about it at the time, but right about then (I'm in GMT+10) a Minecraft server (OpenJDK 64-bit, Ubuntu 11.10) I had running got rather upset ...

     2012-07-01 09:59:59 [WARNING] Time ran backwards! Did the system time change?
     2012-07-01 10:01:14 [WARNING] Can't keep up! Did the system time change, or is the server overloaded?

... and stuck on 100% CPU until I noticed it and restarted it this morning. Thankfully no other issues that I've noticed.

I saw the same across all of our Ubuntu servers. Some running Tomcat others running SmartFox Server (a java based game server). A reboot + app restart fixed it.

I was just wondering why everything stopped working at night. System.exit() didn't work anymore in any of the java processes, or just random hangs.

wow.

Even if you're not running java in your stack, it's worth a quick check on your servers. Our SoftLayer servers use an Adaptec RAID card and the monitoring software uses the JVM and will suck up all of your CPU. All of our MySQL/Cassandra/Redis/Kestrel servers needed to be bounced because they had RAID cards. Frustrating.

I also use Softlayer and have a pair of servers with Adaptec RAID cards. Checking up on them they were not affected.

Having looked in to it, it seems our server admin decided to install OpenNTP to replace the standard NTP daemon. I'm not sure why he did that, but it seems OpenNTP just ignores leap seconds and compensates after the fact just as if your clock had drifted a second out of sync in the usual way.

Even if you don't have java installed but you're running on shared infrastructure that might be running MySQL/Java elsewhere it's a crazy day. My inbox is full of performance-below-acceptable-levels/high-load alerts.

Same problem here. I am supposedly on vacation in Barcelona and was summoned to help :/ Gotta love your java.

Except that it's not a Java problem, it's a Linux futex problem.

At the risk of shooting myself, all our jvms were fine on windows and Ubuntu.

Fun that a bug for my team happens to be the thing linked in this article. :)

Hey Daniel,

Funny to see your name there!

its the futex() call that is bugged, so therefore things that are highly multi-threaded... for example mysql and java!

Many unix things are not as multithreaded as those examples.

I was able to restore my java app boxen (and all other boxen) by only setting the date to the current date. No restarts necessary.

What's the explanation of that wacky date format?

minute day hour month year . second

It's actually month day hour minute.second

Originally (and perhaps still) the month day is optional; allowing you to set the date with just hour and minute.

I experienced no problem today. Is this Java-specific?

No, it's a problem with futexes. If your production code doesn't use many (any) threads, you should be fine.

People are reporting the same issue with MySQLd, I had it with Firefox.

Yeah this is what is slaying linkedin at the moment.

MySQL is also apparently affected, according to the #mysql IRC channel.

Yes, this happened on my development MySQL instance and also one of my production instances of MySQL -- but not another one that I have. I'm not sure why it only affected one, maybe different versions of MySQL?

Has anyone seen this not on Ubuntu?

I saw this on my Fedora 17 laptop. Chrome started eating up 100% CPU and my laptop started getting really hot. I tried restarting Chrome, updating Chrome, etc. Nothing worked. Rebooting fixed the problem.

Surprisingly, none of my servers have had problems, but I don't run Java anywhere.

We saw it pop up on Centos6 as well. JRE 1.6 and 1.7.

opensuse (mysql supporting the kde desktop search).

FC15

Reply on news.ycombinator.com