Npm was down | Hacker News Reader

Npm was down

npmjs.org

30 pointsgingerrr24 comments

Our sincere apologies for tonight's downtime. We're back up now after 30 incredibly frustrating minutes, but we're making changes to ensure this incident can't be repeated.

The root cause was a network failure at our CDN, Fastly. The incident was limited to a single Point of Presence (POP) in San Jose, so if you were in Europe or Asia you didn't see anything wrong, but obviously at this time of day most traffic is from the west coast.

While our uptime over the last few months has been pretty great, in the last week we've had two non-trivial incidents. That's unacceptable to our users, and to us, and we're not just sitting around hoping it doesn't happen again, but will be making fundamental architectural changes to eliminate the sources of failure we've seen.

How fundamental are we talking here? Is npm still going to be a couchapp?

99.9% of requests to npm over the last 3 months have been hitting things other than couch. Binaries are being served directly from disk by nginx, and 99% of JSON GETs are served from our CDN's cache. Couch is still the source of truth, but we treat it much more like a database than an app these days, and it's been much more reliable that way. The only time you're really hitting couch directly these days is when you publish.

Almost none of our downtime since February (when we sorted out some bugs in our app that were affecting couch performance) is attributable to couch. Mostly it has been network-related problems at the caching layer, and the fixes we need to make involve making failover faster and more reliable, as failures are inevitable in any large distributed system.

I'm in Tokyo and it was definitely down for me and now it's down again.

I'm in Zurich and it was also down here.

Yes, we had another 20 minutes of downtime 4 hours later, caused by our CDN accidentally re-instating their broken datacenter. The second outage is documented here: http://status.npmjs.org/incidents/jc65gc8tzk5v

Mirrors, anyone? CPAN has like 50,000,000 mirrors. NPM could get at least three.

Has to be one of the more unreliable of the million package managers out there.

Good thing our devs all rely on this for successful app builds...

At least you know there's a whole bunch of venture money behind them....

Outage status pages:

NPM: http://status.npmjs.org/incidents/nz0kw54kdvfq

Fastly: http://status.fastly.com/incidents/381z6ydt7ddb

Thanks for the link thats very helpful. It would be nice if they had this in their error logs.

One reason where Go's lazy approach is wonderful. It doesn't need to depend to any centralized server.

Use this: npm --registry http://registry.npmjs.eu/

Is it possible to point to a folder containing the packages instead?

I noticed every package installed is cached locally and since I'm usually reinstalling the same packages it would be great if I could point to a folder and let npm install packages and their dependencies from there. Extra points if it could fallback to the official registry if it fails to find a package in the local folder.

I'd be more surprised if you had posted that NPM was up.

The reason for the comfort of Node is imho. NPM and the way it easily gives you fast access to a lot of functionality. That being said it is worrysome that NPM has the size of 80GB+ as far as I know and mirroring is not quite that appealing. I wonder what spoke against a repository-list instead URL of a single-repository URL with fallbacks when they first developed NPM.

Does anyone have links where I could read about what I can read about possible workarounds? Like mirrors may be or setting up private repositories?

I tried to use the eu mirror, but that seems down too.

It's easy to setup a private cache to keep your builds going while npm is down: http://eng.yammer.com/a-private-npm-cache/

There should be a general backup NPM service somewhere though. Just like there are with Linux distro package systems. Maybe it could be a few hours behind in terms of replication, but generally that is good enough for nearly everyone.

Just when I wanted to get some work done.

Guru Meditation?!?

"Guru Meditation" is part of a default error message in Varnish, a caching HTTP proxy. Fastly (NPM's CDN) uses Varnish.

The Guru Meditation error originated with Amiga: http://en.wikipedia.org/wiki/Guru_Meditation

npm_lazy to the rescue!

Reply on news.ycombinator.com