Mystery Blips
mosquitocapital.substack.com
mosquitocapital.substack.com
Britain being filled with Brits, what would happen is that once the show was over, half the nation got up from the couch to turn on the kettle for a cup of tea. Those electric kettles are quite demanding, especially if millions of them turn on at the same time.
So every time there was a major event, dedicated people are monitoring the grid frequency, power plants on standby and foreign contracts at the ready when needed, just to increase capacity at the right time. You can't just schedule this stuff, because if a football match runs over its allotted time, you may suddenly add power to the grid without any load, causing all kinds of problems like the grid frequency increasing and making digital clocks run ahead. There is a temporary demand of gigawatts of power that rises within five minutes and lasts until the kettles are done.
The YouTube video provides an example of 600MW of power being requested from France... for the end of an episode of Eastenders.
I always knew the British like their tea and that there's some kind of planning going on for events like street lights turning on, but the combination of the two is a great example of complex behaviour that's easy to overlook.
Particularly British kettles, which pull 13 amps at 240v = over 3000 watts. A lot more than (standard) American outlets can deliver.
For a while there was a theory going around that this is why Americans typically do not have electric kettles, because they would boil much slower in the USA, but the real answer is that Americans just don't drink much tea.
I would drink tea and use an electric kettle if it would boil faster...
Zojirushi has been in business since 1918 and (in my understanding) is a household name in Japan. Their more expensive (to be blunt) lines are made entirely Japan and replacement parts (like rubber seals, which can wear out over time) are available for purchase through their site. I fully expect mine to last 15-20 years of continuous duty.
Not affiliated, just a thrilled customer.
[1] https://www.zojirushi.com/app/product/cvjac
Edit for additional info: the Zojirushi replaced my Sunbeam Hot Shot single cup boiler that had an unfortunate dry-run and became unusable. They were unavailable in late '21, but appear to be back on the market now. They boil a cup of water in under a minute. If the Zojirushi didn't exist (or I hadn't been willing to bite the bullet on the price) I would buy another in a heartbeat.
Value is what is left when price has been forgotten. Both of these utensils have been high-value for me, but the Sunbeam has a lower initial cost of ownership.
Don't be foolish like some and have 240V wired to a standard plug, though! NEMA 6-15 and 6-20 exist just so you don't mistake your beefed up socket for a normal American socket and fry your devices/zap yourself or others.
Today I made tea and I used a standalone electric kettle which is odd to most of us here in Canada and the US. Most times I would have used the microwave to boil the water. But I used the kettle 120V x 12.5A or 1500W and popped the breaker since the dishwasher was operating and on the same circuit.
In Canada we drink a lot of tea; Red Rose, King Cole and especially south-eastern Canada The Maritimes and Atlantic Canada.
In theory you could sell a 240 volt electric kettle but your kitchen would have to have a 240 volt outlet, probably a NEMA 6-15. They're not common.
Not sure why you'd think so, but no.
(ovens and dryers are not wall plugs)
It's technically a single phase with two hot legs and a center tap for neutral, also called "split phase". From the neutral to either leg is 120V, from leg to leg is 240V.
https://en.wikipedia.org/wiki/Split-phase_electric_power
It would technically be possible to have a reverse-Y extension cord that plugged into two outlets and gave you a 240V circuit. The two outlets would have to be on different legs, of course. This is basically what a 240V breaker is doing. I've contemplating wiring up one of these for home and buying a British tea kettle, but the aesthetics got my plan vetoed.
Something like: https://www.nooutage.com/images/adaptor-wye.jpg
Ahem: the ones driven by the grid frequency are analogue clocks.
This function was so important that if load changes caused the frequency to fluctuate the power companies will compensate as soon as they can to make sure the clocks can catch up. I said "was" because I have no idea (and does anyone?) how many such clocks are still in service. But it's a legacy function and, like POTS, such features are still maintained. Because you never know who'll need it.
https://hackaday.com/2018/03/29/ask-hackaday-is-your-clock-t...
It's been my experience that, in the long term, mains-referenced clocks are extremely accurate, while local quartz ones seem to drift significantly. Clock also tend to last long enough that these are relatively common around here: https://en.wikipedia.org/wiki/Telechron
I don't have any clocks that do this, so I bought one of these, which keeps accurate time without intervention from me or the power company: https://mitxela.com/projects/precision_clock_mk_iii
Alarm clocks, microwave clocks, all manner of time indicators without sync mechanism built in were affected because relying on Europe's near perfect 50 Hz grid frequency is pretty damn cheap.
Our monitoring system included a mercator projection on some big screens, with colored that showed where our gear was and whose color indicated whether it was down, being hammered, etc.
We had no equipment in UTC+14, so there was an hour of waiting between that zone reaching y2k and when we'd maybe see some action. The control room was mobbed since we also had some televisions running cable news there and everyone wanted to see it.
Despite the anxiety, the gear was looking fine everywhere, until about 5 minutes before midnight in UTC+13, when our locations in New Zealand started to turn red. A hush fell over the room as the operators attempted to open connections to those boxes. The connection attempts appeared to hang, and the party got somber. But after the eternity of a few seconds, we were in. The machines were alive but being hammered; the alerts just indicated the excessive traffic caused by mass numbers of users refreshing NZ based websites as the date change approached, to see whether they'd fallen off the net or not. Breathing resumed.
After 15 minutes or so had passed, the traffic fell off, and by the time y2k reached us in UTC+5 we'd been in full festive mode for hours. I still can't look at New Zealand on a map without seeing it covered in angry red circles, though.
Our director was very sensitive to traffic changes. Every week or two there was a big meeting at her office to explain what happened when some traffic went down.
That day she was really mad at us. There afternoon before there was 20% less traffic in the backbone and nobody knew where it went.
I was on call that week for the management systems and I was questioned about why the operators didn't have received any alarm.
Turns out Spain was playing soccer against some Latin American team. Spain was our biggest customer. I don't remember if the other team was Argentina, Chile, or Brazil... but it was our 2nd biggest market.
People just decided to watch TV instead of web browsing (that was before mobile phones had an affordable internet connection).
Funny enough eMule traffic spiked during the match. That was the reason I could justify there was no alert in the system for down interfaces.
If your bandwidth is limited enough such that background downloading would slow down your browsing significantly, then it would make perfect sense to let the downloads saturate the connection while you watched TV.
That is - we were mandated to implement all the SRE tooling on top of our apps, with like 1 guy owning the SRE infra, but with no one looking at the charts proactively.
The idea was sold to dev teams that it was something like a black box recorder for the on-call rotation to check first when you got called in the middle of the night/weekend.
In reality it turned into CTO reporting metrics of whether SLOs were being met, lol.
Constantly feel like other industries read like 1/4 of a book about how FAANG does stuff and then adopts the laziest worst implementation of the part they skimmed.
I think this is less wrong than you think it is. In the P&C insurance industry, I saw a handful of initiatives that seemed like they were started because a senior manager read about something that sounded cool and high-tech in an industry publication, and/or heard it in a sales pitch from a Microsoft rep, without actually checking to see if it was feasible or even useful.
The web farm was about two-dozen Sun Ultra 2s connected on a FDDI loop (we eventually upgraded to GigE), with content on NetApps (4GB drives). A couple Sun E450s. Apache[1] on the Ultra 2s. Apache + mod_perl on the E450s. Hosted at a Global Center DC in Sunnyvale (same DC that early Yahoo was hosted in).
Monitoring with MRTG[2].
It's 1998. The Ken Starr report drops. Now, we knew it was coming, and we did our best to be prepared, but this is the late 90s. There was only so much load testing we could do and didn't really know how much traffic it would drive. We kept the site up, but it meant, as I recall, a lot of fine tuning of the mod_perl box and disabling interactive parts[3] of the site.
I really wish I still had some of the traffic graphs.
Fun times.
[1] Receipt: https://github.com/apache/httpd/blob/1.3.x/src/CHANGES#L5754
[2] Receipt: https://github.com/oetiker/mrtg/blob/master/src/CHANGES#L310...
[3] The forums. OMG forum software was so terrible. I think we eventually wrote our own after nothing based on Netscape Application Server or that we found open source worked well at all.
Space Park 1?
Famously, Yahoo wasn't quite up to the task on 9/11 which led to considerable thought about planning for the unexpected 100x event.
Some of our metrics came in 5 minutes delayed which wasn't a problem for normal days. These metrics moved slow enough that when you got an alarm, there was still plenty of time to take corrective action.
But for HVEs this was an issue. During black Friday or prime day, sometime some metrics spiked so fast you had no time to respond (usually from people hitting page reload a few minutes before a sale kicked off.)
To get an idea for what was going on, I would go in twitter and search for things like "amazon failure" or "amazon 502."
We often got problem reports via Twitter before they showed up on our dashboards.
The database holding signatures for known music samples was sharded by artist. Not as crazy as you might think. You get a trial sample and you send it to all the shards which chunk on it in parallel then you just wait for the servers hosting each shard to respond.
But then Prince died.
The queue for the server that owned the shard Prince was in backed up.
Then our proxy that distributed trials to each shard backed up.
Then our regular reverse proxy backed up.
Then our anemic load balancer fell over.
What we learned for about the bazillioneth time is keep a backup of each shard handy so you can add it into the rotation and pay an intern to watch Facebook looking for news about recently deceased musicians.
> There's a deep beauty to the thought that untold millions
of people using the app randomly, but oh so slightly
habitually, aggregated together, makes such a predictable pattern
I also recall seeing someone push a global update to that system that was packaged wrong, and watching the graph gradually drop and flatline as 20,000 VM hosts across the planet stopped taking traffic. That had its own subtle beauty, in no way diminished by the fact I was just a bystander and couldn't get in trouble for it.
They kept complaining about "the whole desk being slow" and we spent weeks trying to figure out what was wrong with our software.
Eventually we figured out the whole firm was nuts about golf, and they'd all stream golf tournament to their workstations at the same time and saturate their ISDN or whatever they had.
Maybe it would be useful to have a local company radio streaming service, haha.
For us (at normal company scale) I would actually go look at recent activity, new customers, new workloads we are running, but I guess at a Facebook scale it’s a lot harder to do that.
in addition to being a good story about the site reliability, it's also a great lesson in the value of collaboration, mentorship, and having senior people around whom you can ask for help!
And I really agree with your second point too! Without that in-person transfer of tribal knowledge, it would have taken me ages to learn the ropes well enough to take such a high-pressure high-stakes oncall.
SRO is madness. SRO is admitting that you're enough underwater and there's enough unexpected critical events to dedicate a team to burn out on it. There was a lot of spend in overcapacity and failover automation testing that kept it that way, and explicit mandates by a C-Suite and VP to keep keeping it that way. More importantly, though, it was baked into the architecture.
Everyone's talking this week about how Twitter will go up in flames in the next few days; I don't think so. I think they're closer to Google in that respect. There's a lot of automation that will keep things alive and a lot of redundancy that will handle the immediate problems of having a lot of the team leave. I actually think the company can continue to run indefinitely if they just follow the operations manuals and replace failed drives and machines. But I think there will be a major outage when they try to launch a new feature and fail.
A lot of time it would end up being DNS or routing somewhere outside of our control.
Sometimes a disk would get close to filling up and a cron would clear it just in time so you would see decreasing performance then it would clear.
Other times we would have a customer that did something that we just did not expect with the system and it would cause SQL queries to slow down in ways we could not imagine. And it was transient so those would be hard to find or explain.
Or we would hit a limit in the load balancer (haproxy) in really odd ways. Too much traffic on the frontend or not enough capacity in the backend. And many other various ways of things not working. Haproxy was and still is amazing software. Really almost magical.
Ah, good old "it's always DNS"!
I bet every seasoned SRE has a few to add.
It was always noisy. If it's not the miscreants, its the normal stupidity (DNS is so amazingly survivable of misconfigurations, some of them become folk wisdom; like disabling TCP/53, but I digress).
Othodox Easter Sunday morning. Really really early in the morning, like I can't remember if it was light out yet (US Pacific coast time). The SOC calls me. Nothing is happening.
Nothing is happening. There is no noise. So they've been looking at this, and everything is working. So they called me. At dark-thirty on Sunday morning.
I confirm everything appears to be working correctly. I repeat diagnostics, do ad hoc stress tests, everything. About two hours later the normal crap starts coming back.
We have to conclude that the miscreants, as well as the sysadmins, had taken the morning off.
They knew not much work was going to get done either way.
If you graphed the visit rate to our shopping cart page it would follow a curve that was sort of like a sine wave (well, a sine wave plus a constant because you can't have negative cart traffic) with a 24 hour period. I'm just going to call it a sine wave.
The amplitude varied seasonally, and was different on weekends and weekdays, but generally any given day was a sine wave.
If you graphed order rate it also followed a sine curve too, in phase with the visit curve and generally with a fixed ratio between them. So if the cart was getting N visits per minute, we'd be getting fN order per minute for some number 0 < f < 1, and f was pretty much constant.
This continued as the business grew, so the overall amplitude of the visit curve increased. But at some point the order rate seemed to hit a wall.
What we would then see is that as the visit rate was heading toward its peak for the day when it got to a particular fraction of that peak, say 80% of the peak, the order rate would stop tracking the visit rate and would become flat. It would stay flat as the visits peaked and declined until the visit rate was back down to 80% of its peak and then orders would resume tracking visits.
So far nothing too weird about this. What was weird is that when we'd have a day of especially high cart traffic such as when something in the news caused a big increase in demand for a few days for the kinds of things we sold, and we'd get 3 or 4 (or sometimes even 20 or 30) times the normal cart visit rate throughout the day--the plateau on the order rate would still start when the visit rate hit 80% of the visit rate peak for that day and go away when the visit rate fell back below 80% of peak.
The order processing was handled by a different server than the shopping cart. They did use the same database for product and customer information, but we could not find any signs that the database was a bottleneck.
This just made no sense. Suppose we'd been seeing on normal days that visits would peak at 1000 per minute, and orders were 10% of visits. Then we'd see visits grow from 0 to 800 and orders grow from 0 to 80. Then orders plateaued at 80 while visits continued on to 1000 and back to 800. Then as orders fell from 800 to 0, orders fell in step from 80 to 0.
Now suppose the next day we had 10 times the activity due to some news item. So the next day visits rise from 0 to 10000. The order system would now not plateau until the visit rate got to 8000 and the order rate got to 800.
And then when the increased demand was over in a few days and we were back to 1000 visits peak the order system would again plateau at 80.
So whatever was causing the plateau was somehow tied to the visit rate, making the plateau depend on what the peak visits would be for that day.
But how on a given day when the visits would first rise to 800 and the orders to 80...how did it "know" if the day was going to be a 1000 visit peak day or a 10000 visit day so it could "know" if it was time to plateau?
We never figured this out. Reasoning about our systems and their dependencies I could not figure out any way in theory that this could be happening nor could I find anything no matter how much logging and tracking I added all over the place.
Everything I could think of that could even remotely cause a plateau would have caused a fixed plateau, not some weird plateau that's level gets set daily based on what the future peak visit rate is going to be for that day.