Hacker News Reader

Comment by npunt | Hacker News Reader

I get the impression Wayback Machine data is stored in powered down drives and they only spin them up when someone accesses the data. That would explain the several second delay and it'd make sense that an archive wouldn't need 95% of its data ready to go at a moments notice since that'd be a terrible waste of power.

View on HN

The disks are spinning all the time, and most disks are seeing fairly frequent reads to some content or another. A lot of content is very rarely accesses, but almost every disk has some content which gets accessed. If spinning disks had only frequently-accessed content, they would be unable to keep up with the read rate or read throughput, things balance out reasonably on average.

Wayback content is on the same disks as most other content, in the form of WARC files, with individual records fetched out of the middle of WARC files via HTTP range request.

Note: I work at IA but am not on core infrastructure team

Interesting thanks for the insights! Then would the few second delay be more a matter of time it takes to decompress the contents, or that files are stored on disks which are being accessed a lot, or something else? Always been curious about it.

And disk life.

perversely, wouldn't spinning drives up and down impact drive life?

Probably yes, but only if it was happening a lot; I don’t know what the cross-over point would be though that you’re better off just keeping the drives spinning...

Reply on news.ycombinator.com