snewman

This was the premise of "The Cool War" by Frederik Pohl. It's been a long time so I can't remember whether it was a great read, but definitely an interesting idea.

Well: in many cases, compactions can make more files eligible to be garbage collected. So if you want to accelerate the freeing up of space, running a compaction may serve that goal. It will temporarily increase usage, but once the garbage collector catches up the net effect is to reduce disk usage. Unless I'm mis-remembering how things work. (Caveat: I left Google almost 9 years ago.) And so "run a compaction to free up space" is a reasonable shorthand most of the time... unless you're so close to the edge that you don't have room for the temporary increase.

Yup, many an outage has something like this at its core. At my company, to address that, we've built an internal library for enforcing rate limits and size limits that is (a) configurable on the fly and (b) generates logs, so that we can trigger an alert whenever any limit reaches 70% of capacity. And thus, hopefully, head these things off before they reach the tipping point.

https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/

If you're seriously interested in building this and would be open to employment for that purpose – please drop me a line (contact info in my profile). This absolutely deserves to exist and it would be a natural extension of what we're building at Scalyr. I'd love to talk.

Scalyr CEO here. I'd love to hear more about how you're thinking about attack surfaces – drop me a line? My email address is in my profile.

Scalyr | DevOps, Backend Engineers | San Mateo, CA | ONSITE

I've built half a dozen startups, including Writely (aka Google Docs), and I can honestly say that Scalyr is my favorite by far. We are the blazing fast log management platform built for the DevOps front line. We built our technology stack from the ground up, optimizing it for speed and scalability. With over 180 customers, real revenue and only 32 employees, the time to join is now!

DevOps Engineer: Join the core engineering team to help us scale our growing infrastructure, ingesting tens of TBs of logs every day.

Backend Engineer: We've built a NoSQL data engine from scratch that searches test at 1TB/second. How'd you like to help us scale and optimize to multiple TB/second?

Sound interesting? apply now: https://www.scalyr.com/jobs

As an engineer working on Docs and related products from inception through 2010: I would be astonished if this were the case. Had such behavior become known within the engineering team, there would likely have been an uprising. There was no backdoor mechanism for looking documents, at least on the Docs (word processing) side, and in any case this sort of thing Was Not Done. My guess is that things are locked down even more tightly now.

I have no idea. :)

You might be right about the OT idea, or even some code, having been borrowed from Wave. I wouldn't necessarily have known about it (or I might simply have forgotten). But I'm pretty certain that Kix as an overall project was much more closely connected to Trix than to Wave.

Good question. I don't know how much of that was done in this case. I can say that at the outset of the Kix project, they did ask me for advice about pitfalls. (I was skeptical that it could be done at all using the browser technology of that time; they proved me wrong. One of my concerns was proper handling of complex international character sets, both for input and rendering.)

Kix is a drastically different architecture than Writely, so many lessons would not have been applicable. Writely was built on a contentEditable DIV, so the data model was literally whatever junky HTML the browser's contentEditable implementation would create, and synchronization was implemented by performing three-way diffs on the HTML code. It was very much a dancing bear, in the sense of "the amazing thing is not how well it dances, but that it dances at all". ...but I digress. My point is that the mechanisms and challenges of that approach are entirely different than the Kix approach. I think the important lessons would have been product-level rather than code-level.

Bear in mind that this was almost a decade ago, and I wasn't directly involved. With that caveat: the core mechanisms for client / server coordination in Kix were a direct descendant from Trix (short for "Matrix"), which was the original code name for Google Sheets. And Kix was developed by a team in New York, an offshoot of the Trix team which was also in New York. Writely (post-acquisition) was in Mountain View, and Wave was in Australia (I forget which city), and the picture that might give you of how closely the various teams communicated is probably accurate.

I don't know whether Trix originally used OT specifically, or the Kix team borrowed that idea from Wave. But the basic idea of clients performing small, tidy logical operations and then synchronizing them in realtime through a server was part of both Trix and Kix. Writely used a very different (and more primitive) approach.

I don't know anything about the origin of the comments feature, but my guess is that there was no connection to Wave, except (again) possibly a bit of inspiration. The Wave team was very separate, both geographically and organizationally, and the tech stack they built was very disconnected from anything that was going on with Trix, Kix, or Writely.

I'm pretty sure there is no lineage from Wave to Kix (which, yes, was the code name for the modern incarnation of Docs). Kix is more of a descendant of the architecture of Google Sheets.

Source: I am one of the original authors of Writely, did not work on Kix but I was still at Google when it was being written, had a minor role in reviewing the design.

Exactly, you use a fixed RNG seed so that the behavior is reproducible.

For complex multithreaded code, it can become difficult to achieve consistent execution, but a fixed seed is at least a good start.

> the tests never try to mix PSYNC2 with replication of Lua scripts

This sort of testing gap is so hard to avoid. Any complex system has an inexhaustible number of potential feature interactions. It's very difficult at best to guess which combinations need to be tested, and often difficult to even write a test which exercises a particular combination.

What techniques do people use to ensure test coverage of complex cross-feature interactions like this? Two things I emphasize:

* Emphasize integration tests, and try to use multiple features in each test. This can help drive out interaction problems; the downside is that these tests are harder to write and (especially) maintain. It can also be difficult to exercise specific code paths.

* Write randomized tests, that exercise as many features as possible. The challenge here is identifying failures. "The program didn't assert or crash" is often a good start. A gold standard is to run a simple reference implementation alongside your real implementation, and compare the output... but that's not always possible.

Oops -- thanks for catching this copy/paste glitch. Here are corrected links:

https://www.scalyr.com/jobs/senior-software-engineer-fronten...

https://www.scalyr.com/jobs/software-engineer-backend?gh_jid...

Scalyr | Frontend, Backend Engineers | San Mateo, CA | ONSITE

I've built half a dozen startups, including Writely (aka Google Docs), and I can honestly say that Scalyr is my favorite so far. We're building an amazing team, users rave about our product, and things are taking off – we're closing seven-figure contracts and revenue grew 5x last year. And we're doing it on a sane, 40-hour, daily-team-hike, Fridays-at-home schedule. We're pre-series-A, but in many ways playing like a series B company, so this is a chance to get the best of both worlds -- early-stage equity and impact, with later-stage compensation and stability; plus a very strong team for peering or mentorship.

Backend Engineer: We've built a NoSQL data engine from scratch that searches text at 750 GB/second. How'd you like to help us scale and optimize to multiple TB/second?

Frontend Engineer: "The fastest blog in the world" (https://jacquesmattheij.com/the-fastest-blog-in-the-world) loads in under 100 ms. Help us push our data visualization tools toward that goal.

https://www.scalyr.com/jobs/senior-software-engineer-fronten....

https://www.scalyr.com/jobs/software-engineer-backend?gh_jid....

I think this is comparing Amish people with the mutation vs. Amish people without the mutation. Both groups are Amish eating Amish food in Amish communities with other Amish.

Scalyr | Frontend, Backend Engineers | San Mateo, CA | ONSITE

I've built half a dozen startups, including Writely (aka Google Docs), and I can honestly say that Scalyr is my favorite so far. We're building an amazing team, users rave about our product, and things are taking off – we're closing seven-figure contracts and revenue grew 5x last year. And we're doing it on a sane, 40-hour, daily-team-hike, Fridays-at-home schedule. We're pre-series-A, but in many ways playing like a series B company, so this is a chance to get the best of both worlds -- early-stage equity and impact, with later-stage compensation and stability; plus a very strong team for peering or mentorship.

Backend Engineer: We've built a NoSQL data engine from scratch that searches text at 750 GB/second. How'd you like to help us scale and optimize to multiple TB/second?

Frontend Engineer: "The fastest blog in the world" (https://jacquesmattheij.com/the-fastest-blog-in-the-world) loads in under 100 ms. Help us push our data visualization tools toward that goal.

https://www.scalyr.com/jobs/senior-software-engineer-fronten...

https://www.scalyr.com/jobs/software-engineer-backend?gh_jid...

Google Docs does use OT. Or at least it did about six years ago, and I doubt that has changed. Source: I cowrote the original implementation (which was not OT, it was a three way merge of complete HTML files), and I was around when a new team rewrote the entire app using OT.

Historical note: live three way merging of HTML files, being produced by content-editable div implementations in different browsers, which like to rewrite each other's coding decisions in arbitrary ways (suddenly all your BRs become Ps, whee!) is "fun" in a way that I don't really ever need to revisit.

How will you know the person emailing you is actually the owner of that username?

Scalyr | Frontend, Backend Engineers | San Mateo, CA | ONSITE

I've built half a dozen startups, including Writely (aka Google Docs), and I can honestly say that Scalyr is my favorite so far. We're building an amazing team, users rave about our product, and things are taking off – we're closing seven-figure contracts and revenue grew 5x last year. And we're doing it on a sane, 40-hour, daily-team-hike, Fridays-at-home schedule. We're pre-series-A, but in many ways playing like a series B company, so this is a chance to get the best of both worlds -- early-stage equity and impact, with later-stage compensation and stability; plus a very strong team for peering or mentorship.

Backend Engineer: We've built a NoSQL data engine from scratch that searches text at 750 GB/second. How'd you like to help us scale and optimize to multiple TB/second?

Frontend Engineer: "The fastest blog in the world" (https://jacquesmattheij.com/the-fastest-blog-in-the-world) loads in under 100 ms. Help us push our data visualization tools toward that goal.

https://www.scalyr.com/jobs/software-engineer-frontend?gh_ji...

https://www.scalyr.com/jobs/software-engineer-backend?gh_jid...

Scalyr | Backend, Frontend, Customer Success Engineers | San Mateo, CA | ONSITE

I've built half a dozen startups, including Writely (aka Google Docs), and I can honestly say that Scalyr is my favorite so far. We're building an amazing team, users rave about our product, and things are taking off – we're closing seven-figure contracts and revenue grew 5x last year. And we're doing it on a sane, 40-hour, daily-team-hike, Fridays-at-home schedule. We're pre-series-A, but in many ways playing like a series B company, so this is a chance to get the best of both worlds -- early-stage equity and impact, with later-stage compensation and stability; plus a very strong team for peering or mentorship.

Backend Engineer: We've built a NoSQL data engine from scratch that searches text at 750 GB/second. How'd you like to help us scale and optimize to multiple TB/second?

Frontend Engineer: "The fastest blog in the world" (https://jacquesmattheij.com/the-fastest-blog-in-the-world) loads in under 100 ms. Help us push our data visualization tools toward that goal.

Customer Success Engineer: our customers are engineers. We're looking for someone with lightweight ops experience and good communication skills to help them get the most out of Scalyr. A great opportunity to join a high-powered team and work your way up the engineering ladder.

https://www.scalyr.com/jobs/software-engineer-frontend?gh_ji...

https://www.scalyr.com/jobs/software-engineer-backend?gh_jid...

https://www.scalyr.com/jobs/customer-success-engineer?gh_jid...

"C", but that's meaningless because AWS scrambles the zone names for each account. (Presumably to prevent everyone from putting all their servers in "A".)

There was definitely an issue. Around 25% of our servers in one availability zone of us-east-1 fell off the network for 15 minutes or so, starting around 13:47 GMT. They're back now.

During this time period, we were also unable to access the console (500 errors).

I've built half a dozen startups, including Writely (aka Google Docs), and I can honestly say that Scalyr is my favorite so far. We're building an amazing team, users rave about our product, and things are taking off – we're closing seven-figure contracts and revenue grew 5x last year. And we're doing it on a sane, 40-hour, daily-team-hike, Fridays-at-home schedule. We're pre-series-A, but in many ways playing like a series B company, so this is a chance to get the best of both worlds -- early-stage equity and impact, with later-stage compensation and stability; plus a very strong team for peering or mentorship.

Backend Engineer: We've built a NoSQL data engine from scratch that searches text at 750 GB/second. How'd you like to help us scale and optimize to multiple TB/second?

Agent Engineer: you'll own our lean, mean data collection agent. If you enjoy constantly getting to play with new tools, come help us connect to... everything. You'll get to play with packages from Apache to Zookeeper, APIs from AWS to, er, Azure, while tackling challenges such as monitoring 100,000+ simultaneous log files using minimal CPU.

Customer Success Engineer: our customers are engineers. We're looking for someone with lightweight ops experience and good communication skills to help them get the most out of Scalyr. A great opportunity to join the team and work your way up the engineering ladder.

https://www.scalyr.com/company/jobs/back-end-engineer

https://www.scalyr.com/company/jobs/agent-engineer

https://www.scalyr.com/company/jobs/customer-success-enginee...

[edit: fixed link – thanks vnchr]

Scalyr | Backend, Agent, Customer Success Engineers | San Mateo, CA | ONSITE

I've built half a dozen startups, including Writely (aka Google Docs), and I can honestly say that Scalyr is my favorite so far. We're building an amazing team, users rave about our product, and things are taking off – we're closing seven-figure contracts and revenue grew 5x last year. And we're doing it on a sane, 40-hour, daily-team-hike, Fridays-at-home schedule. We're pre-series-A, but in many ways playing like a series B company, so this is a chance to get the best of both worlds -- early-stage equity and impact, with later-stage compensation and stability; plus a very strong team for peering or mentorship.

Backend Engineer: We've built a NoSQL data engine from scratch that searches text at 750 GB/second. How'd you like to help us scale and optimize to multiple TB/second?

Agent Engineer: you'll own our lean, mean data collection agent. If you enjoy constantly getting to play with new tools, come help us connect to... everything. You'll get to play with packages from Apache to Zookeeper, APIs from AWS to, er, Azure, while tackling challenges such as monitoring 100,000+ simultaneous log files using minimal CPU.

Customer Success Engineer: our customers are engineers. We're looking for someone with lightweight ops experience and good communication skills to help them get the most out of Scalyr. A great opportunity to join the team and work your way up the engineering ladder.

https://www.scalyr.com/company/jobs/back-end-engineer

https://www.scalyr.com/company/jobs/agent-engineer

https://www.scalyr.com/company/jobs/customer-success-enginee...

Scalyr | Backend, Frontend, Devops, Customer Success Engineers | San Mateo, CA | ONSITE

I've built half a dozen startups, including Writely (aka Google Docs), and I can honestly say that Scalyr is my favorite so far. We're building an amazing team, users rave about our product, and things are taking off – we're closing seven-figure contracts and revenue grew 5x last year. And we're doing it on a sane, 40-hour, daily-team-hike, Fridays-at-home schedule. We're pre-series-A, but in many ways playing like a series B company, so this is a chance to get the best of both worlds -- early-stage equity and impact, with later-stage compensation and stability; plus a very strong team for peering or mentorship.

Backend Engineer: We've built a NoSQL data engine from scratch that searches text at 750 GB/second. How'd you like to help us scale and optimize to multiple TB/second?

Frontend Engineer: "The fastest blog in the world" (https://jacquesmattheij.com/the-fastest-blog-in-the-world) loads in under 100 ms. Help us push our data visualization tools toward that goal.

Devops Engineer: We run hundreds of servers, process 750 GB/second, maintain 99.99% uptime, and almost never get paged (http://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/). But we're still not satisfied. How'd you like to take our operations to the next level as we go through our next 10x of growth?

Customer Success Engineer: our customers are engineers. We're looking for someone with lightweight ops experience and good communication skills to help them get the most out of Scalyr. A great opportunity to join the team and work your way up the engineering ladder.

https://www.scalyr.com/company/jobs/back-end-engineer

https://www.scalyr.com/company/jobs/front-end-engineer

https://www.scalyr.com/company/jobs/devops-engineer

https://www.scalyr.com/company/jobs/customer-success-enginee...

I don't think this is a fair use of the word "normally".

The service may have been working within SLA. Based on the data in this article, it was not working the way it works most of the time. I think most people would interpret "normally" to refer to the latter.

Or, let's drop the word "normally" and take a step back. A good dashboard should be designed to inform. Amazon's dashboard was not providing the information that Ably or their customers needed. And this isn't an isolated complaint; if I had a nickel for every time someone had complained about the green mockery of the AWS health dashboard, I'd have... well, several dollars, anyway.

They also probably have one or more test regions where they could perform a test like this. But it's presumably not at nearly the same scale as us-east-1, the region affected by this incident. And to a considerable extent the problem was one of scale. The writeup makes the recovery sound fairly straightforward; but due to the sheer size of S3 in this region, it took hours for the system to come back up, which was apparently unexpected.

(Nit: this incident affected a region, not a zone. us-east-1 is a region, which is divided into zones us-east-1a, us-east-1b, etc. S3 operates on regions.)

> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well.

Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enough for full-scale tests to be tractable, but certainly not cheap, and how often do you run it? Also, hindsight is 20/20, but before this incident it might have been hard to identify "full-scale restart of the index subsystem" as rising to the top of the list of things to test.

One approach is to try to extrapolate from smaller-scale tests. It would be interesting to know what kinds of disaster testing Amazon does do, and at what scale, and whether a careful reading could have predicted this outcome.