When Amazon Switched from Sun to Linux

Let's take Uber. Their core product is a social media app with mapping, tracking and billing features. It has burnt through an incredible amount of money trying to kill competitors, employing 20,000 or so people directly and claiming that a self-driving electric fleet will someday make them insanely profitable.

Market cap is up almost 80% in the last year.

Who's the idiot? The investor who believes that Uber will be insanely profitable in N years, or the investor who doesn't want any more exposure to Uber than as part of a whole-market index?

All processors (or at least all processors I'm aware of) will make a core observe its own writes in the order they appear in the machine code. That is, a core by itself will be unable to determine if the processor performs out-of-order memory operations. If the machine code says Core A makes writes A0 and then A1, it will always appear to Core A that A0 happened before A1. As far as I know, all processors also ensure that all processors will agree to a single globally consistent observed ordering of all atomic reads and atomic writes. (I can't imagine what atomic reads and writes would even mean if they didn't at least mean this.)

On top of the basic minimum guarantees, x86 and x86_64 (as well as some SPARC implementations, etc.) have a Total Store Ordering memory model: if Core A makes write A0 followed by A1, and Core B makes write B0 followed by B1, the two cores may disagree about whether A0 or B0 happened first, but whey will always agree that A0 happened before A1 and B0 before B1, even if none of the writes are atomic.

In a more relaxed memory model like the SPARC specification or Aarch64 specification (and I think RISC-V), if the machine code says Core A makes write A0 before A1, Core B might see A1, but not yet see A0, unless A0 was an atomic write. If Core B can see a given atomic write from Core A, it's also guaranteed that Core B can see all writes (atomic and non-atomic) that Core A thinks it made before that atomic write.

With the DEC Alpha, the hardware designers left themselves almost the maximum amount of flexibility that made any semantic sense: if Core B makes an atomic read, then that read (and any reads coming after it in machine code order) is guaranteed to see the latest atomic write from Core A, and all writes that came before that atomic write in machine code order. On the Alpha, you can think of it as all of the cores having unordered output buffers and unordered input buffers, where atomic writes flush the core's output buffer and atomic reads flush the input buffer. All other guarantees are off. (Note that even under this very lax memory model, as long as a mutex acquisition involves an atomic read and an atomic write, and a mutex release involves an atomic write, you'll still get correct behavior if you protect all reads and writes of shared mutable state with mutexes. A reader's atomic read in mutex acquisition guarantees that all reads while holding the mutex will see all writes made before another thread released the mutex.) This might be slightly wrong, but it's roughly what I remember of the Alpha memory model.

The thing that confused some programmers with the Alpha is that with most memory models, if one thread makes a ton of atomic writes, and another thread makes a ton of non-atomic reads, the reading thread will still never see the writes in a different order than what the writer thought it wrote. There's no such guarantee on Alpha.

On a side note, the Alpha team was also pretty brutal about only allowing instructions that were easy for compilers to generate and showed a performance improvement in simulations on some meaningful benchmark. The first generation of the Alphas didn't even have single-byte loads or stores and relied on compilers to perform single-byte operations by bit manipulation on 32-bit and 64-bit loads and stores.

Many of the Alpha design people went on to the AMD K6 III (first AMD chips to give Intel a run for their money in the desktop gaming market), the PASemi PWRFicient (acqui-hired by Apple to start their A-series / Apple Silicon team), AMD Ryzen, etc.)

When I bought my first computer in the fall of 1997, the fastest Intel desktop processors were 300 MHz PIIs. DEC Alphas at the time were running at 500 MHz, and had more instructions per clock, particularly in the floating point unit. The Cray T3E supercomputer used DEC Alphas for good reason.

I seem to have confused which of the 3 memory ordering states was the default and supported on all models. I'm well aware of the Alpha.

> e.g. if you load a pointer then read the value at that pointer you may get stale data

For anyone thinking that sounds horrifying, a single core doing the writes and read won't observe anything out of the ordinary.

The designers of the Alpha thought "Okay, we can simplify and speed up the cache coherency logic if the "happens before" temporal dependency introduced by a memory fence between two writes won't make any guarantees about two reads on another core, unless there's also a "happens before" temporal dependency introduced by a memory fence between the two reads". This sounds totally reasonable, particularly at a time when lockfree data structures weren't as popular. Reads and writes done while holding a mutex work just like any other processor: the memory fence involved in acquiring a mutex guarantees that any reads made while holding the mutex will see all writes made before the previous release operation on that mutex.

The difficulty with the Alpha memory model comes mostly in writing lockfree data structures, where the safety comes from modifying some data structure while no other threads can see it, then atomically changing a pointer (usually a load-locked/store-conditional or compare-and-swap) to make that pointer available to other threads. On most architectures, you only need a memory fence right before (or a part of) the write operation that changes the pointer. On most architectures, this happens-before guarantee will be seen by all normal read operations. On Alpha, you also need a memory fence between reading that pointer and chasing it. In C, the loading the pointer and chasing it happen in a single expression *p, so you need to load the global pointer into a local variable, execute a memory fence, and then deference the pointer (or else use inline assembly, C11 atomics came out way way after the DEC Alpha AXP).

Lockfree data structures become more popular well after the Alpha was released, so it's understandable why the Alpha designers were willing to make these optimizations that don't affect code that correctly uses mutexes to protect global mutable state.

PAE might not have met Amazon's needs for holding their catalog or whatever. If they were trying to directly address a data structure that was larger than 4GB, they would have needed some sort of trickery as PAE was usually implemented kernel side to provide separate 4GB address spaces to individual processes with more than 4GB total physical backing.

After the crash, you had to basically give that stuff away. I knew a guy who had 10 Sun boxes (all less than 5 years old) at home because a local company wrote them off. I had a couple of low end Ultras (an Ultra 2 and Ultra 10) but sold one of them off. The Ultra 2 I got used for pennies on the dollar at an auction. I think it was like $200 for what was a $15K machine years earlier.

Sun was so far ahead of it time. Solaris was a solid OS, but once Linux became "good enough", somewhere in the '99 to '03 timeframe, depending on industry, it was over for them.

My fingers still habitually run `sync` when they're idling because of my innumerable experiences with filesystem corruption and data loss on Linux during the 1990s. There were just too many bugs that caused corruption (memory or disk) or crashes under heavy load or niche cases, and your best bet at preserving your data was to minimize the window when the disk could be in a stale or, especially, inconsistent state. ext3, which implemented a journal and (modulo bugs) guaranteed constant consistent disk state, didn't come until 2001. XFS was ported to Linux also in 2001, though it was extremely unreliable (on Linux, at least) for several more years.

Of course, if you were mostly only serving read-only data via HTTP or FTP, or otherwise running typical 90s websites (Perl CGI, PHP, etc, with intrinsically resilient write patterns[1]), then Linux rocked. Mostly because of ergonomics and accessibility (cost, complexity); and the toolchain and development environment (GNU userland, distribution binary packages, etc) were the bigger reasons for that. Travails with commercial corporate software weren't very common because it was uncommon for vendors to port products to Linux and uncommon for people to bother running them, especially in scenarios where traditional Unix systems were used.

[1] Using something like GDBM was begging for unrecoverable corruption. Ironically, MySQL was fairly stable given the nature of usage patterns back then and their interaction with Linux' weak spots.

I sank about $10k into that system in ebay parts. at the time there just wasn't any way to get such bandwidth out of x86, but used gear was cheap. if you could afford the time, sweat and blood to keep it running.