Comment by endisneigh | Hacker News Reader

Comment by endisneigh | Hacker News Reader

It’s amazing that this isn’t a solved problem, but we have all of this crazy language model stuff.

Unfortunately Spanner isn’t open source. Yugabyte and Citus are close but have annoying issues. Cockroach isn’t 100% compatible (and has its own issues) and things like FoundationDB which are truly HA and comparable to Spanner in terms of consistency and fault tolerance are not easily plugged into Postgres as the underlying storage engine since sadly it’s only a key value store.

edit: when I say close, I'm talking strictly about HA, not general functionality.

lately I've been thinking of using FoundationDB, which is closest to Spanner in terms of ACID and serializability and mvsqlite.

Then, I was thinking, since SQLite doesn't have online schema changes (nor does mvsqlite) to have a schema such as:

   [UUID, Data, Version, CreatedAt, UpdatedAt]

Where Data is a JSON or Proto and Version is an integer. You then could mimic an online schema change by in your application code supporting two adjacent "versions", and then in an eventually consistent manner run [small] transactions to update the Data to the new Version as necessary. You would index Version, and UpdatedAt as necessary to find the rows in the table that are not "migrated."

In SQLite you can also create indexes on expressions so technically all of your JSON or Proto could also have indexes.

Citus is not close to Spanner. No global secondary indexes, no cross-shard ACID (the commit status is eventually consistent). No auto-resharding. Yugabyte and Cockroach have a spanner-like architecture, but different open-source model and postgres-compatibility. What are those annoying issues?

This is true, but the article is talking about HA and Citus is HA, no?

by itself, Citus is just sharding to multiple databases. Then each of this database can be protected like any database with a standby

Honestly, just use Spanner. It's the perfect database*. If you need to trade freshness for savings, put a cache in front of it.

If I were building a new startup in 2023, I would need a mountain of evidence against using Spanner. It's ugly that it locks you into GCP but hey an iPhone locks you into Apple's ecosystem, that's just the price you pay to get good things.

* unless you need timeseries, columnar, FTS, geospatial, graph or something special like that

I've built my own little pile of evidence against Spanner using it for several years, after previously being on Postgres. It has good potential but isn't there yet. And if you're making a startup, you're probably a long way from the scale where sharded DBs become necessary.

I hate this tired argument. If I'm building a startup I'm building it with the intention to hit scale. If I can use cockroach or yugabyte and get 90% of the benefits of postgres without significant additional cost I'm going to use it. I've been at multiple startups that have had trouble scaling their database and the cost to switch database technologies when you hit scale is massive. And worst of all, the limitations of the technology always impact product development. I don't care what cool features postgres has if I can't do online schema changes.

So yeah, I'd rather spend a few more hours picking an actual scalable technology now than spend what could be 7 figures (it can be a lot more, I've heard of this migration uber had to do that was costing them half a billion a year) and massively impact product development moving to scalable technology..

Different teams have different abilities. You can't really make general rules around any of this. Should you use spanner as a default selection for your company's database needs? Maybe.

both good points, including the devs having different abilities and using a ready-made scalable solution. We're living in the world of having something working that you can test in a few days shrug

Some general rules make sense for common use cases. Like, I'll say that a relational DBMS is generally what you want rather than a graph one. For a single-node DB, I'll always use Postgres if I have the choice. For a webserver, can't really go wrong with NodeJS, though you might have reasons to use something else (e.g. cost to run).

But sharding is too use-case specific to prescribe something broadly like Spanner.

> If I can use cockroach or yugabyte and get 90% of the benefits of postgres without significant additional cost I'm going to use it

Here's the problem, it's not 90%. Idk about those two, but you sacrifice a lot using Spanner, maybe enough to delay your launch. Any slightly advanced query becomes hard to optimize, for one. Not to dis Spanner, it's just a very different beast.

CEO won't want to hear "launch is delayed, but it'll scale better later," and it's also not really true. In the initial phases, you can't tell what your actual needs will be at scale, so you'll probably have to rework things anyway. Just using Spanner doesn't solve your scaling problems. Maybe you'll need to totally redo your Spanner schema to actually scale, maybe you find you're better off sharding at the application layer than at the DB.

Spanner is ridiculously expensive. Though I get your point, it’s not like an iPhone. An iPhone is not actually that expensive compared to top end android phones.

Spanner is very expensive even compared to hosted cockroach and RDS (Aurora). That being said I’m sure if you're enterprise it's substantially cheaper.

Spanner generally is about twice the cost as cockroach which seems cheap but the gap in nominal cost only grows with usage.

The other issue is that since there’s no source to self host you also have to pay, unlike cockroach.

Generally with cockroach I’d expect one to self host all instances except a staging/pre-pod and prod. With Spanner all your instances necessarily will be hosted, which means $$.

That being all said, Spanner is worth it if money isn’t an issue

The thing is, there is nothing else like Spanner, unless you want to trust a startup with your database, which opens up Cockroach, Yuga, and TiDB. Databases like RDS don't compare because they don't offer the same guarantees. Nothing even in AWS' and Azure's portfolios is like Spanner.

Back of napkin math I did a couple of months ago: Spanner costs about $1/month for 2 writes/second and 10 reads/second. The minimum is $88 which is 176wps and 880rps. If your startup gets more traffic than that, you have good problems. The simplified guarantee model (fully linearizable and serializable everything; automagic sharding, replication and failover) could save you costs elsewhere.

Low-value high-volume data like logs and search queries can be sent to a cheaper self-hosted timeseries/fts database.

You forgot storage costs and especially bandwidth costs and lockin costs and probably range-queries costs.

Spanner is great if you have unlimited money

> perfect

Doesn't spanner introduce new (very unlikely) failure modes that other databases are not impacted by? The reliance on an external consistency model feels to me like a complete outsourcing of liability that warrants thorough investigation.

Hypothetically, if GPS went down for a prolonged period and/or a bug was found in the TrueTime system, what would happen to the consistency model around Spanner?

I feel like some applications and customers would much rather wait for a synchronous acknowledgement from the actual, live system. An extra 150ms when you are confirming a 6-figure wire transfer could easily be framed as a good thing in most circles.

Spanner offers both linearizability and serializability. They use the non-standard term "external consistency" to describe the union of those two properties.

>Doesn't spanner introduce new (very unlikely) failure modes that other databases are not impacted by?

Curious if you could go into that.

Doesn't Spanner depend on access to globally-synchronized custom atomic clock hardware? i.e. even if it were open source, it wouldn't really be valuable unless you're a data center operator.

As I understand it, it's not a solved problem because there is no silver bullet, but rather trade-offs in every direction, and which solution works for you (including Spanner) is heavily dependent on your use case.

the clocks help but are not strictly required. FoundationDB also achieves external consistency without the clocks, but has limitations, such as a 5 second window for a transaction, due to optimistic concurrency.

the clocks remove such limitations.

There are some fundamentally really hard problems with multi-node DBs, and the usable solutions depend a lot on your use case, so I'm not surprised at the current state.

Out of curiosity, what's wrong with cockroach?

Reply on news.ycombinator.com