n_u

Hi, thank you for your work on this and being willing to answer questions on it.

"We guarantee durability via replication". I've starting noticing this pattern more where distributed systems provide durability by replicating data rather than writing it to disk and achieving the best of both worlds. I'm curious

1. Is there a name for this technique?

2. How do you calculate your availability? This blog post[1] has some rough details but I'd love to see the math.

3. I'm guessing a key part of this is putting the replicas in different AZs and assuming failures aren't correlated so you can multiply the probabilities directly. How do you validate that failures across AZs are statistically independent?

Thanks!

[1] https://planetscale.com/blog/planetscale-metal-theres-no-rep...

A couple dumb questions:

1. You say "ephemeral", but my understanding is that NVMe is non-volatile so upon crash and restart we should be able to recover the state of the memory. Is is ephemeral because of how EC2 works where you might not get that same physical box and memory addresses back?

2. Can you explain what "Semi-synchronous replication" is? Your docs say "This ensures every write has reached stable storage in two availability zones before it’s acknowledged to the client." but I would call that synchronous since the write is blocked until it is replicated.

Thanks!

From my position

MySQL pros:

The MySQL docs on how the default storage engine InnoDB locks rows to support transaction isolation levels is fantastic. [1] This can help you better architect your system to avoid lock contention or understand why existing queries may be contending for locks. As far as I know Postgres does not have docs like that.

MySQL uses direct I/O so it disables the OS page cache and uses its own buffer pool instead[2]. Whereas Postgres doesn't use direct I/O so the OS page cache will duplicate pages (called the "double buffering" problem). So it is harder to estimate how large of a dataset you can keep in memory in Postgres. They are working on it though [3]

If you delete a row in MySQL and then insert another row, MySQL will look through the page for empty slots and insert there. This keeps your pages more compact. Postgres will always insert at the bottom of the page. If you have a workload that deletes often, Postgres will not be using the memory as efficiently because the pages are fragmented. You will have to run the VACUUM command to compact pages. [4]

Vitess supports MySQL[5] and not Postgres. Vitess is a system for sharding MySQL that as I understand is much more mature than the sharding options for Postgres. Obviously this GA announcement may change that.

Uber switched from MySQL to Postgres only to switch back. It's a bit old but it's worth a read. [6]

Postgres pros:

Postgres supports 3rd party extensions which allow you to add features like columnar storage, geo-spatial data types, vector database search, proxies etc.[7]

You are more likely to find developers who have worked with Postgres.[8]

Many modern distributed database offerings target Postgres compatibility rather than MySQL compatibility (YugabyteDB[9], AWS Aurora DSQL[10], pgfdb[11]).

My take:

I would highly recommend you read the docs on InnoDB locking then pick Postgres.

[1] https://dev.mysql.com/doc/refman/8.4/en/innodb-locking.html

[2] https://dev.mysql.com/doc/refman/8.4/en/memory-use.html

[3] https://pganalyze.com/blog/postgres-18-async-io

[4] https://www.percona.com/blog/postgresql-vacuuming-to-optimiz...

[5] https://vitess.io/

[6] https://www.uber.com/blog/postgres-to-mysql-migration/

[7] https://www.tigerdata.com/blog/top-8-postgresql-extensions

[8] https://survey.stackoverflow.co/2024/technology#1-databases

[9] https://www.yugabyte.com/

[10] https://aws.amazon.com/rds/aurora/dsql/

[11] https://github.com/fabianlindfors/pgfdb

Regarding cross-shard consistency, there's a doc here[1] on the options. In short you can just send the writes to all replicas, you can disallow cross-replica transactions (read and write), use two-phase commit to perform a distributed transaction or use their own hybrid approach that they set as the default.

[1] https://vitess.io/docs/20.0/user-guides/configuration-advanc...

Thank you.

From the doc

> Automerge uses a combination of LWW (last writer wins) and multi-value register. By default, if you read from doc.foo you will get the LWW semantics, but you can also see the conflicts by calling Automerge.getConflicts(doc, 'foo') which has multi-value semantics.

> Note that "last writer wins" here is based on the internal ID of the opeartion [sic], not a wall clock time. The internal ID is a unique operation ID that is the combination of a counter and the actorId that generated it. Conflicts are ordered based on the counter first (using the actorId only to break ties when operations have the same counter value).

Seems like they use LWW with Lamport clocks to order operations and a unique ID for each client as a tie-breaker.

Is there info anywhere on the structure of the semi-lattice they are using for their CRDT?

Is the map based on a multi-value register or a last-writer-wins register?

I agree with both of you. I think it’s sort of a hybrid and a spectrum of how much you do of each first.

When you test part of the circuit with the scope, you are using prior knowledge to determine which tool to use and where to test. You don’t just take measurements blindly. You could test a totally different part of the system because there might be some crazy coupling but you don’t. In this system it seems like taking the measurement is really cheap and a quick analysis about what to measure is likely to give relevant results.

In a different system it could be that measurements are expensive and it’s easy to measure something irrelevant. So there it’s worth doing more analysis before measurements.

I think both cases fight what I’ve heard called intellectual laziness. It’s sometimes hard to make yourself be intellectually honest and do the proper unbiased analysis and measuring for RCA. It’s also really easy to sit around and conjecture compared to taking the time to measure. It’s really easy for your brain to say “oh it’s always caused by this thing cuz it’s junk” and move on because you want to be done with it. Is this really the cause? Could there be nothing else causing it? Would you investigate this more if other people’s lives depended on this?

I learned about this model of viewing RCA from people who work on safety critical systems. It takes a lot of energy and time to be thorough and your brain will use shortcuts and confirmation bias. I ask myself if I’m being lazy because I want a certain answer. Can I be more thorough? Is there a measurement I know will be annoying so I’m avoiding it?

Ah I see. I thought by query you meant "user search query". I'm guessing by query here you mean "read".

>It isn't the target of very many queries (relatively speaking)

Wow why is that? Do you use a vector index primarily?

Ah interesting. Is your keyword-document map (aka term dict) too big to keep in memory permanently? My understanding is that at Google they just keep it in memory on every replica.

Edit: I should specify they shard the corpus by document so there isn't a replica with the entire term dict on it.

Which part of the index are you putting in the buffer pool here? The postings list, the doc store or the terms dict?

Is it being cached for future queries or are you just talking about putting it in memory to perform the computation for a query?

It’s more about whether or not the company has taxable profits for that year (importantly these are not the same as real profits). I would read this article to understand more about how being forced to amortize tax deductions for expenses affects a business’s taxes.

https://news.ycombinator.com/item?id=44180533

more info here too

https://news.ycombinator.com/item?id=44226145

It also classifies software development as R&D which together with immediate expensing for R&D undoes the Section 174 changes as far as I understand.

“For purposes of this section, any amount paid or incurred in connection with the development of any software shall be treated as a research or experimental expenditure“

Page 303 of bill here https://www.congress.gov/119/bills/hr1/BILLS-119hr1eas.pdf

Original article about Section 174 tax code causing layoffs

https://news.ycombinator.com/item?id=44180533

Post from @dang with more info about Section 174

https://news.ycombinator.com/item?id=44226145

Relevant quote from article “Three business tax deductions would be made permanent. That includes the ability to use depreciation and amortization as the basis for interest expensing, the research and development write-off”

from the bill itself on page 795

“SOFTWARE DEVELOPMENT- For purposes of this section, any amount paid or incurred in connection with the development of any software shall be treated as a research or experimental expenditure.”

https://www.congress.gov/119/bills/hr1/BILLS-119hr1eh.pdf

Relevant articles

https://news.ycombinator.com/item?id=44180533

@dang if there’s a more relevant article feel free to change the link

Signed and called my representative and senators.

I ask simply "If I have $1m of revenue and $1m of expenses that is entirely software dev salaries, what do you think my profit is for that year? How much should I be taxed on that?"

https://www.house.gov/representatives/find-your-representati...

https://www.senate.gov/senators/senators-contact.htm

Some OTs do, some don't. OTs with the TP2 property do not require a central authority to order edits I believe.

In my experience if you are persisting your edits or document state, you have something that creates an ordering anyways. That thing is commonly an OLTP database. OLTPs are optimized for this kind of write-heavy workload and there's a lot of existing work on how to optimize them further.

But now even S3 has PUT-IF, so you could use that to create an ordering. https://docs.aws.amazon.com/AmazonS3/latest/userguide/condit...

What I’ve always been curious about is if you can help the S3 query optimizer* in any way to use specialized optimizations. For example if you indicate the data is immutable[1] does the lack of a write path allow further optimization under the hood? Replicas could in theory serve requests without coordination.

*I’m using “query optimizer” rather broadly here. I know S3 isn’t a DBMS.

[1] https://aws.amazon.com/blogs/storage/protecting-data-with-am...

It’s not addressed directly but I do think the article implies you hope your request latencies are not correlated. It provides a strategy for helping to achieve that

> Try different endpoints. Depending on your setup, you may be able to hit different servers serving the same data. The less infrastructure they share with each other, the more likely it is that their latency won’t correlate.

I found a broadcast[1] of it and it seems the humans and robots run in separate lanes. It also looks like there are humans walking/running with the robots to hold them up if they start to fall over. Some of them need it and some of them seem fine running on their own.

[1] https://www.youtube.com/watch?v=lEYVbq7OF3w start at 1:05:00 for video of running robots

Thanks for the example! I'll play around with it.

> On my machine, the inflection point is around 10^14 goroutines - after which the mutex version becomes drastically slower;

How often are you reaching 10^14 goroutines accessing a shared resource on a single process in production? We mostly use short-lived small AWS spot instances so I never see anything like that.

> Why? Designing and implementing concurrent runtimes has not made its way onto my CV yet; hopefully a lurking Go contributor can comment. > If I had to guess, the channel mutex may be specialised since it protects only enqueuing or dequeuing onto a simple buffer. A sync.Mutex is a general construct that can protect any kind of critical region.

Haha fair enough, I also know little about mutex implementation details. Optimized specialized tool vs generic tool feels like a reasonable first guess.

Though I wonder if you are able to use channels for more generic mutex purposes is it less efficient in those cases? I guess I'll have to do some benchmarking myself.

> If the communication pattern is passing streams of work where goroutines have an acyclic communication dependency graph, then it's a no brainer: channels will be performant and a deadlock will be hard to introduce.

I agree with your rules, I used to always use channels for single processt thread-safe queues (similar to your Kafka rule) but recently I ran into a cyclic communication pattern with a queue and eventually relented to using a Mutex. I wonder if there are other painful channel concurrency patterns lurking for me to waste time on.

Do you have any benchmarks for the pattern you described where channels are more efficient?

> sync.Mutex, if left to wait long enough will enter a slow code path and if memory serves, will call out to a kernel futex. The channel will not do this because the mutex that a channel is built with is exists in the go runtime

Do you have any more details about this? Why isn’t sync.Mutex implemented with that same mutex channels use?

> [we] use channels extensively in a message passing style. We do not use channels to increment a shared number

What is the rule of thumb your Go shop uses for when to use channels vs mutexes?

Strong yes for 3 reasons

1. Reducing dev friction.

When I had managers who coded they were ruthless about removing friction in the dev and deployment pipeline because they had to deal with it too. If build times went up, deployment infrastructure broke or someone’s PR broke dev they would roll it back immediately. If someone consistently blocks PRs the manager noticed the trend and would address it.

2. You get a much better sense of IC’s contributions by writing code.

There are ICs who play politics very well and sell themselves but that set is not the same as the ICs who deliver. If you are writing code you start to notice which ICs have written key features, built critical APIs or worked on hard problems because of comments and Git blame.

3. Understanding your codebase.

I hope most managers have solid CS and engineering fundamentals but that is a necessary but not sufficient condition to grasping the full picture. There’s a reason it takes time to ramp up to full productivity on a new codebase. If you work in the codebase and have had to use that one annoying but critical library or dealt with that tech debt from 2 years ago then you know what is hard and what isn’t. I’ve found when a codebase has a quirk that makes developing certain features hard all of the non-technical people keep forgetting why we can’t do that thing and all the technical people have it burned into their brains.

> It is a curly-brace language

What does this mean? Not Python?

1. > However, customers still get a lot of value out keeping downstream systems synced with their warehouse tables. For instance, you can define which people you want to receive different campaigns and make sure that's consistent across all your ad platforms.

Ah thank you that's a very helpful example for understanding your product.

2. > Some people have warehouse tables with many billions of rows.

I don't doubt customers have multi billion row tables in their CDWs but I guess I'm not seeing why you would need to send billions of rows to FB (or any other downstream system that isn't an OLAP database) rather than some much smaller payload distilled from that data via an ML model or SQL query. I admittedly have never run a FB ad campaign but Meta has 3.35 billion daily active users [1] across all of their products. If they are sending billions of rows to FB, do your customers have individualized ad campaigns for every single FB user? Perhaps I'm just not familiar enough with the state of modern digital advertising.

3. > If you have a billion rows, you need to hit a very high rows per second number in order to run a sync in a feasible amount of time.

I wonder why do you have to send billions of rows to FB every time? Surely you send it once for initial setup and then incrementally sync smaller deltas? And presumably your customer is OK with the initial setup being slower. Unless your customer is doing billions of writes to their CDW in between syncs?

Thanks for taking the time to explain!

[1] https://s21.q4cdn.com/399680738/files/doc_financials/2024/q4... page 10

Did you test the "stop the world" approach? I wonder how the write downtime compares. It seems the 1 second of replication lag is unavoidable. The arbitrary 2.5 seconds of waiting for txns to finish could be removed by just killing all running txns, which your new approach does for txns longer than 2.5 seconds already.

> ;; 2. Give existing transactions 2.5 seconds to complete.

> (Thread/sleep 2500)

> ;; Cancel the rest

> (sql/cancel-in-progress sql/default-statement-tracker)

Then you have 2.5 seconds less downtime and I think you can avoid the problem of holding all connections on one big machine.

> Our switching algorithm hinges on being able to control all active connections. If you have tons of machines, how could you control all active connections?

> Well, since our throughput was still modest, we could temporarily scale our sync servers down to just one giant machine

> In December we were able to scale down to one big machine. We’re approaching the limits to one big machine today. [15] We’re going to try to evolve this into a kind of two-phase-commit, where each machine reports their stage, and a coordinator progresses when all machines hit the same stage.

I guess it depends on what your SLO is. With your approach only clients with txns longer than 2.5 seconds started before the upgrade see them fail, whereas with the "stop the world" approach there would be a period lower-bounded by the replication lag time where all txns fail.

Cool work thanks for sharing!

Edit: I feel like a relevant question regarding the SLO I'm not considering is how txns make their way from your customers to your DB? Do your customers make requests to your API and your application servers send txns to your Postgres instance? I think then you could set up a reasonable retry policy in your application code and use the "stop the world" approach and once your DB is available again the retries succeed. Then your customers never see any txns fail (even the long-running ones) and just a slight increase in latency. If you are worried about retrying in cases that are not related to this upgrade you could change the configuration of your retry policy shortly before/after the upgrade. Or return an error code specific to this scenario so your retry code knows.

Then you get the best of both worlds: no downtime perceivable to customers, no waiting for 2.5 seconds, and you don't have to write a two-phase-commit approach for it to scale.

If your customer sends txns to your Postgres instance directly, this wouldn't work I think.

Looks like you are also hiring a distributed systems engineer https://news.ycombinator.com/item?id=42920292

From that listing "Sync Speed: Customers want to sync a lot of data to important destinations like Facebook and Snapchat, which requires us to analyze every part of our syncing process and find where we can optimize to sync data more quickly"

I'm curious about this. What workflow requires syncing high volumes of data from a CDW to Facebook & Snapchat at low latency? It's my understanding that businesses mostly use those platforms for advertising. I'm struggling to think of a use case where you want to adjust your advertising with low latency and lots of data? I could understand feeding lots of data from your CDW into a ML model that updates your ads through the FB Ads API but I can't see why

1. it needs to go straight from CDW to FB ?

2. it needs to be a lot of data?

3. it needs to be fast?

Perhaps there is some other use-case besides adjusting ads.

4. Also why do you use the word "syncing" rather than "send"? I tend to think of syncing involving multiple programs that can edit data (e.g. Google Docs, distributed consensus etc.). Are Facebook and Snapchat actually updating the data you send and you have to sync the other direction? Or is just one-way?

> If we went with the ‘stop the world approach’, we’d have about the same kind of downtime as blue-green deployments: a minute or so.

> After about a 3.5 second pause [13], the failover function completed smoothly! We had a new Postgres instance serving requests

> [13] About 2.5 seconds to let active queries complete, and about 1 second for the replica to catch up

Why is the latter approach faster though? It seems in the "stop the world" approach wouldn't it still take only 1 second for the replica to catch up? Where do the other ~59 seconds of write downtime come from?

They say the "stop the world" approach that causes more downtime is

  Turn off all writes.
  Wait for 16 to catch up
  Enable writes again — this time they all go to 16

and instead they used a better algorithm:

  Pause all writes.
  Wait for 16 to catch up. 
  Resume writes on 16.

These seem pretty similar.

1. What is the difference in the algorithm? Is it just that in the "stop the world" approach the client sees their txns fail until "wait for 16 to catch up" is done? Whereas in the latter approach the client never sees their txns fail, they just have a bit more latency?

2. Why does the second approach result in less downtime?

To add onto this I feel like one of the hard things about TF is that there is like at least 3 ways to do everything because they have supported multiple APIs and migrated to eager. So if you find an example or an open source project it might not be for the flavor of tensorflow that your codebase is in.