Hacker News Reader

Comment by toomuchtodo | Hacker News Reader

> Your system clock must be set correctly, or at least it must not move before memcached has restarted. The only way to tell how much time has passed while the binary was dead, is to check the system clock. If the clock jumps forward or backwards it could impact items with a specific TTL.

Why not write the system timestamp to the memory state file as well? And then "evict" (ie just not load) anything exceeding TTL on state file load?

View on HN

That's exactly what it does, though since it's resuming a monotonic clock the actual code was a bit more complicated...

To find out how long it was down, it notes system time into the state file on shutdown. On start it checks the current system time and adds the delta to the monotonic timer and resumes. Objects exceeding TTL are removed appropriately.

I appreciate that you took the time to explain this.

I think they use the system clock to determine the time after the restart

I might’ve just misunderstand that paragraph. I’d assume with $snapshot_timestamp, current time, and TTLs, time wouldn’t be an issue (and of course, don’t futz with system time while memcached isn’t running).

Not really. If it's 4:00:00 and the system clock says 4:20:00, and then it exits... then 21 minutes goes by and the system clock is set correctly. It is indistinguishable from the clock not stepping and 1 minute going by.

> (and of course, don’t futz with system time while memcached isn’t running).

Well, yes, that's what the original thing you quoted said: "Your system clock must be set correctly, or at least it must not move before memcached has restarted."

I’ve never worked in environments where real time and system time had that much drift (due to ntp), but I acknowledge it probably happens out there in distributed systems. Accurate time is important!

And the more granular the time-based caching is within that system the more likely that mine time skew can kill cache.

I've had distributed systems perform unreliably in the < 30s range even with ntpupdate syncing in place.

Before memcached had a monotonic clock people would end up with immortal objects (underflowed TTL's) because ntp would start after memcached and make a huge adjustment due to the hardware clock being really off.

With the restart code, people could run a kernel upgrade and reboot while the daemon is down... so if this ends up causing a huge clock adjustment you're screwed.

Reply on news.ycombinator.com