bigmutant

Pretty common attitude from folks who have never worked in one of the BigTech companies where Java rules (Amazon being a prime example). Since they never encounter Java in the "SF-style Startup" world, they assume that it must be dead. Meanwhile hundreds-of-thousands of Engineers deal with hundreds-of-millions (billions?) of lines of Java every day

DynamoDB is used *everywhere* in AMZN Retail, this is absolutely not surprising. Plus the vast majority of internal Services are using EC2 in the form of Apollo/ECS. So OP probably hit some parts of the site that are hosted in us-west-2. For all I know they started routing all requests for us-east-1 traffic to other DCs, figuring latency is a fine trade-off for availability

To clarify, most of CDO (Consumer Devices Other) does run on AWS in the sense that NAWS is the target state, MAWS is legacy and actively (slowly) being migrated off of. CDO (including Alexa) has been using DynamoDB/Lambda/Kinesis/SQS etc forever, its just the compute and kind-of network layers that are still MAWS. Even then, a large part of CDO has moved from Apollo to ECS/FarGate/whatever unholy Hex or DataPath thing they're pushing these days

Source: Ex-AMZN

Sure, I did the same, BS/MS with a focus on Compilers/Programming Languages. It's been personally gratifying to understand programming "end-to-end" and to solve some tricky problems, but 99% of folks aren't going to hit those problems. There are tons of people interacting with Cloud Services every day that aren't aware of the basic issues like:

- Consistency models (can I really count on data being there? What do I have to do to make sure that stale reads/write conflicts don't occur?)

- Transactions (this has really fallen off, especially in larger companies outside of BI/Analytics)

- Causality (how can I handle write conflicts at the App Layer? Are there Data Structures ie CDTs that can help in certain cases?)

Even basic things like "use system time/monotonic clocks to measure elapsed time instead of wall-clock time" aren't well known, I've personally corrected dozens of CRs for this. Yes this can be built in to libs, AI agents etc but it never seems to actually be, and I see the same issues repeated over-and-over. So something is missing at the education layer

Def agree. Most people will never touch an Abstract Syntax Tree or even Expression Trees. Almost everyone working in back-end will use Cloud Services, will make mistakes based on assumptions of what they provide

The fundamental problems are communication lag and lack of information about why issues occur (encapsulated by the Byzantine Generals problem). I like to imagine trying to build a fault-tolerant, reliable system for the Solar System. Would the techniques we use today (retries, timeouts, etc) really be adequate given that lag is upwards of hours instead of milliseconds? But that's the crux of these systems, coordination (mostly) works because systems are close together (same board, at most same DC)

Good resources for understanding Distributed Systems:

- MIT course with Robert Morris (of Morris Worm fame): https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39...

- Martin Kleppmann (author of DDIA): https://www.youtube.com/watch?v=UEAMfLPZZhE&list=PLeKd45zvjc...

If you can work through the above (and DDIA), you'll have a solid understanding of the issues in Distributed System, like Consensus, Causality, Split Brain, etc. You'll also gain a critical eye of Cloud Services and be able to articulate their drawbacks (ex: did you know that replication to DynamoDB Secondary Indexes is eventually consistent? What effects can that have on your applications?)

As others have said, this is a solved problem in a lot of companies. Basic answers are: 1. Queuing 2. Asynchronous APIs (don't wait for the 'real' response, just submit the transaction) 3. Call-backs to the Client

A good async setup can easily handle 100k+ TPS

If you want to go the synchronous route, it's more complicated but amounts to partitioning and creating separate swim-lanes (copies of the system, both at the compute and data layers)

Absolutely not true in my experience. MySQL has its share of issues (all DBs do) but it is rock-solid when using the correct engine (InnoDB for most cases, RocksDB for high-throughput writes, Memory for caching). MySQL is very hard to beat for very high-volume OLTP workloads, both reads and writes. Its replication systems were years ahead of other systems (SQL Server, Postgres, SQLite doesn't have replication). DuckDB AFAIK is OLAP and they don't compete in the same space. Every DB system has "the things its good at" and MySQL really shines at very high-volume OLTP spread across partitions.

That all depends on the setup. The "standard" setup (not specific to MySQL) is:

- Single Write Leader per partition

- Backup Write Leader that is setup with synchronous replication (so WL -> WLB and waits for commit)

- Read Followers all connected asynchronously using either binlog replication (not recommended anymore) or GTID-based row replication (recommended)

In the above scenario, the odds of loss are pretty small since the Write Leader has a direct backup, and any of the Read Followers can be promoted to a Write Leader/Backup. DDIA calls the above semi-synchronous replication, although MySQL now supports a similar-but-slightly different version out of the box: https://dev.mysql.com/doc/refman/8.4/en/replication-semisync...

Agree, Data Quality in-the-wild is a huge concern. I've led efforts to establish Lineage/Quality in large orgs and doing this after-the-fact is a massive undertaking. Having this "up-front" before all the data pipelines (origination, transformation, pre-processing) calcify saves a lot of headache down the road.

Never really understood the folks pushing for RPC-over-HTTP. RPC is for systems that are close together (ie intra-DC). These simple rules work well: 1. JSON-over-HTTP for over the web 2. RPC (pick your flavor) for internal service-to-service

I will say that Amazon's flavor (Coral-RPC) works well and doesn't come with a ton of headache, its mostly "add ${ServiceName}Client to build" and incorporate into the code. Never mind its really odd config files

Related note, I've never understood why Avro didn't take off over GRPC, I've used Avro for one project and it seems much easier to use (no weird id/enumerations required for fields) while maintaining all the code-gen/byte-shaving

Northrop Grumman had a lot of folks from Crenshaw/Hawthorne/Carson when I was there, due to a partnership program with the local Cal State (Long Beach). All of the security staff was from that area too. Good folks, would 100% work with them again.

On the other hand, I've seen exactly 1 guy at the FANG I work at. What's the difference? I think it's companies like Northrop realizing that folks from under-represented communities have great value and prioritize that instead of whatever the current HackerRank-based interview process selects for

The Tech-Priests of Mars are calling

Sort of. The main issue is that AMZ/AWS doesn't differentiate (by title) between someone recently promoted to L6 (generally ~8-10 YOE) and someone deep in to L6 (~12-15 YOE). Other companies solve this with the Staff Engineer title, but AMZ doesn't have that. If you're an L6 who reports to an L7 Sr. Manager then you're functionally a Staff Engineer and supposed to be on the Principal track. Mileage varies across orgs/teams.

Gist: "junior" L6 SDE is more like Google L5, "senior" L6 SDE tracks mostly with Google L6 (or even L7 in rare cases).

Before you accept any offer from FANG/MANG/hot startup, *make sure* you understand the on-call expectations. Example: at Amazon, *even data scientists* do on-call. Practically all SWEs before Principal do. In AWS, first-line managers do on-call (they typically don't in Devices/Consumer). In places like Alexa, you can expect to have shitty 5-6 week on-call rotations and extra helpings for holidays. AWS at least advertises this in their job postings, other orgs (cough Alexa) do not. So buyer beware.

FAANGs/large companies in general have a ton of accidental complexity. A lot of the work devolves into "make this work in our complicated environment". Its just more pronounced at FAANG due to the scale.

Amazon's vest structure is 5/15/40/40. Every vest after the first two years is bi-yearly ie every six months (including additional grants). Amazon makes up the difference with cash in the first two years. The Target Compensation limit is another story entirely...