How we built a serverless SQL database | Hacker News Reader

How we built a serverless SQL database

cockroachlabs.com

265 pointsTheresaBraccio117 comments

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic.

The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres.

I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even funny anymore.

Alternatively, people really do need to put a lot of thought into scaling, but only because they did something like write some core web service in an interpreted language framework that maxes out at 100 requests per second.

What happens when that database fails? Are you OK losing some data, or do you want the data to be synchronously replicated off the machine and be available somewhere else after failure? Distribution isn't only about scale, it's also about availability.

What happens when that database loses some data? Do you want an up-to-the second backup, or point-in-time recovery? Or are you OK restoring last night's backup? Distribution isn't only about scale, it's also about durability.

What happens when you need to run an expensive business process ad-hoc? Do you want it to be easy to scale out reads, or to export that data to an analytics system? Or are you OK building something else to handle that case? Distribution isn't only about scale, it's also about flexibility.

What happens when you want to serve customers in one market, and make sure that their data stays local for regulatory compliance reasons or latency? Are you OK with having separate databases? Distribution isn't only about scale, it's also about locality.

Fair points. I would argue that for most people a simple master-slave setup with manual failover will produce far fewer headaches than a "serverless" architecture.

When you are big enough to worry about the other issues, you surely are big enough to handle the requirements in-house. I see the dependence on some specific companies as the bigger threat to reliability.

The setup you describe is very much not simple. I worked at a place with very good DBAs and our replication setup caused us more downtime than anything else. Cockroach and Spanner exist because many programmers observed that what you describe is hard.

As a counter-anecdote: multiple startup projects I've worked on with separate MySQL setups where each had just a single master + two slaves (one warm for fast failover in case of hardware failure or upgrades, one cold for slow analytics-style queries) did just fine with millions (to tens of millions) of users. No downtime at all for years on end.

MySQL and Postgres are massively more widely-used than Cockroach and Spanner, broadly very successfully. It's entirely feasible to run it with high uptime.

Very few deployments experience actual failures. Could be some fridge-door/light situation going on.

> fridge-door/light situation going on

what does it mean ?

I think that is meant to be parsed as: Just like you can't check if the fridge light is on without opening the door (which of course turns it on), it's hard to know if a system is resilient to failure without having one. It just might be that there hasn't been a situation that would cause a failure.

This is probably one of the best motivations for a distributed database that I've read.

I find that it's not often that people grasp that distribution is about availability. It's obvious when you say it, but for a long time my own intuition was that distribution is about mostly durability or consensus protocols to provide total order across multiple machines. Yet these build together into availability.

In fact, I first noticed this distinction when reading Brian M. Oki's seminal 1988 paper on Viewstamped Replication, the work that would pioneer the field of consensus—a year before Paxos but with an intuitive protocol essentially identical to Raft. The surprising thing is that today many of us might have titled the paper something about "consensus" or "total order" (which it practically invented, and which was the major breakthrough, at least how to do this in the presence of network partitions) but that he titled it "Viewstamped Replication: A New Primary Copy Method to Support Highly-Available Distributed Systems".

I did a short intro talk to Viewstamped Replication (and particularly why FTP or nightly backups or manual failover are not a solution): https://www.youtube.com/watch?v=_Jlikdtm4OA

The talk is followed by interviews with Brian M. Oki and James Cowling (authors of the 1988 and 2012 papers respectively).

The question isn’t whether you have 10k req/s now, but whether you expect to in the future. If you are designing a blog, then yeah, you probably don’t need to worry about it. If you are starting a social network or SAAS business application, then you probably do.

The future will be different.

A lot of successful businesses start with things that are not scalable, and it is a strength, not a weakness. If you start a social network for instance, you can't beat Facebook at its own game. You have to do something that Facebook can't do because it is too big. Scalability problems will be tackled as you grow.

Among the many things Facebook can't do is running its service on a single database. It makes things much harder on them, thankfully, you are much smaller than Facebook and you can. Take that advantage.

Unfortunately, easy to hit that with say GraphQL where each client request can resolve to dozens of db selects vs a single hand written/tuned SQL select.

Maybe that's a good reason to avoid GraphQL then?

That's not the reason, there are many but this a nice feature where a client can send one request and it will resolve internally (so less client side requests), you can request only the fields you need and it's easier when hitting multiple micro services.

Now you get into the issues of tracing, request failures and retries, payload size (only json? really..) etc.

But I do agree that a super optimized sql for a a specific purpose has it's performance benefit

In theory the graphql server could be smart enough to assemble a single SQL select to satisfy the query. Does anyone know of an implementation that does this?

If you're using GraphQL with SQL -- postgres specifically -- I would say use Hasura, but support between Hasura & CockroachDB seems to have stalled due to missing triggers [0] [1]. CRDB supports a feature called "changefeeds" [2] which is claimed might cover some of Hasura's use-cases, but that's a proprietary extension not present in base PostgreSQL.

[0]: https://github.com/hasura/graphql-engine/issues/678

[1]: https://github.com/cockroachdb/cockroach/issues/28296

[2]: https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-...

Any views on which to choose between Prisma, Hasura or postgraphile for GraphQL and postgres?

(robot voice) Does /dev/null support sharding? Sharding is the secret ingredient in the web scale sauce.

it's impressive how well that video has aged with modern web dev.

This is a good argument against running a k8s cluster when u don't need one, but not a good argument against this new serverless Cockroach product.

Serverless is not just about auto scaling up from 1 to n, it's about autoscaling down from 1 to 0.

If Cockroach provides a robust SQL DB at a marginal cost of ~$0/mo for small request volumes, that is a real value add over running your own pg server.

Not having to deal with administration or backups is another big value add.

This offering looks like it compares very nicely to say running an rds instance with auto backups enabled.

Tradeoffs are tradeoffs.

In your k8s example, by running a k8s cluster when you don't need one, a cost you pay is the overhead.

In the Cockroach serverless case, costs that come to mind include vendor lock-in when you evolve a pattern of production traffic that is hard to migrate to other solutions, and security and compliance challenges due to the virtualized instances running on shared clusters. In many cases these tradeoffs may be worthwhile. My point is that looking at it only in the dimension of scaling up and down, doesn't tell the whole story. The OP doesn't talk about tradeoffs, so in the comment section we must.

In the case you mention, you've made a tradeoff to defer developing an in-house solution for supporting a potential future pattern of production traffic, and that can be a huge accelerator.

Further, once you're experiencing the types of success that demands a superior solution the priority to invest in such a solution is clear.

It isn't "lock-in" - there's enough experience and capability in the market to solve those problems _once you have to_. Solutions like this let you decide whether the right time is at start-up or scale-up.

k8s lets you put multiple deployments on virtualized hardware in a cloud agnostic/consistent way. You can do all that on one machine. Even if you're running a single instance deployment you still probably want at least two environments. K8s isn't without value in that respect.

The answer is unfortunately less clear cut. Particularly if you assume that whoever is tasked with scaling this hypothetical DB doesn't know what they are doing a-priori.

The following questions are likely to come up

1) My t3.xl DB is down, how much bigger can I make it?

2) My r3.24xl DB can only handle 100 TPS and now my site is down, what can I do?

3) My 2x r3.24xl DB cluster costs a lot of money, Are other solutions cheaper?

4) My latency is high, are other solutions faster?

For someone who hasn't dealt with these questions before, these will become long and painful lessons with massive material impacts to the business.

It's appealing to use Dynamo as it takes the magic out of scaling. It's appealing to use serverless RDBMS as you don't have to think about it anymore unless it has high costs/latency.

Why would you assume that the person responsible for a thing doesn't know what they're doing?

> The answer is unfortunately less clear cut. Particularly if you assume that whoever is tasked with scaling this hypothetical DB doesn't know what they are doing a-priori.

The answer is very clear-cut:

Work with professionals.

The amount of data being collected in the world is growing much faster than the number of database engineers is.

The number of professionals who know and care about scaling who also want to work on small scale applications is relatively small. Hiring them will pose a challenge.

I disagree. Even the simplest master-slave setup bound to cause an outage once or twice.

Now, if you setup your DB using public cloud's flavor of PostgreSQL... That's a different story.

But wouldn't you agree that the initial architecture is really important; It should be at least designed with scaling in mind? Since it can get difficult changing things afterwards.

Sure if "in mind" includes realistic capacity planning.

During a technical interview many years ago, I commented that the system architecture didn't scale.

The interviewer responded "When that matters, I won't even be managing the person whose problem that is."

Isn't that the right mindset?

This will be somebody else's problem? Ehm, no it's not.

- When the bridge collapses I'll be managing someone else.

- When they find out this airplane has critical design flaws I'll be managing someone else.

- When this software I'm working on is hacked I'll be managing someone else.

All bridges have a capacity limit.

"Doesn't scale" doesn't imply "inadequate."

Overbuilding is waste.

Software isn't bridges though. If you run into scaling issues if you're lucky it's just a case of swapping out a database for a bigger one. More likely it isn't just that though and being able to do things like scale up distributed workers coherently with a change of a config file requires upfront thought and design that YAGNI would say isn't necessary. People just breezily saying "we can optimise it later" for a scale up of orders of magnitude are nearly always wrong. I'd argue Reddit is a perfect example of a site that's clearly running into scaling issues but are locked into an architecture with few escape hatches built in

When a bridge fails it'll just fail. When software runs into a scaling limit it'll degrade and fall on its arse constantly and be absolutely terrible as long as it takes for the software team to completely rework their entire architecture often having to learn completely new technologies.

No, software isn't bridges. When bridges fail, people die. When reddit crashes, people take a piss and pet the cat.

I don't know the relevant details about reddit but the assumption that the early reddit people could have easily built something more scalable yet there are tech reasons why the later people, with far more resources, can't.

As to the assumption that one knows the important bottlenecks 2-3 orders of magnitude in advance, that's just wrong.

"late answers are wrong answers" isn't just for real-time. That applies to products as well as signals.

Technical debt is not necessarily a bad thing.

It's not a matter of who's problem it is. It's just that a scale-able architecture in most cases is a premature optimization. When building a product, scalability is only one aspect. And in case of most startups and companies, amongst the smaller ones.

I personally interview a lot of people and if they start proposing microservices or k8s or anything trendy like that (before having context), I consider it a negative point.

When hiring, I want someone to take a look at a busines problem, break it down into a smaller pieces. Most of the times, the most important engineering work is coming up with the right data models/data structures.

So yeah, maybe the architecture I have now wont scale up. But at least it'll get the business going. Years later when there are more resources, the software could be rewritten or whatever.

Also, you'r example mentioned "software being hacked". I never said undermine security. Security should always be taken seriously. Security is not a premature optimization. Scalability is.

> Security is not a premature optimization

There are plenty of situations where decent security is a premature optimization too. For example prototypes or proofs-of-concept that are only intended to demonstrate or benchmark a capability. The sort of thing where, even if you did insist on making it 'secure' the username/password would be admin/admin.

I wasn't commenting on whether scalability is important or not. I was commenting on that mindset: "When that matters, I won't even be managing the person whose problem that is."

What you said all sounds reasonable. Sure, don't build something scalable if you've figured out you want something quick to prove your market and you're ok with tossing it all away or if you'll never need the scale. These are not the same thing. But this will be somebody else's problem? I don't want that guy on my team... Something someone told me a long time ago, write code like the guy that takes it over from you is a psychopath and has your address.

Your examples aren't equivalent. You're comparing the failure of a prototype or proof-of-concept under stress to the failure of production models under expected conditions.

The real risk you're incurring by deploying a prototype to production until traction is demonstrated and scaling becomes an issue is that deferring the design of the 'real' system risks succumbing to second-system syndrome and prematurely trying to make it a 'platform'.

Roughly, the first version just has to work, the second has to scale, and the third (assuming you get there that soon) is the one that starts needing to be refactored/generalized into various reusable subsystems.

The second doesn't (necessarily) have to scale.

It has to handle the projected load, with some safety margin.

"projected until when?" you ask?

Well, your projections will be wrong. And, your needs will change, which means that you're going to be changing the system anyway.

That's where judgment comes in.

Suppose that each customer is worth $10/month and that the "not scalable" version can handle 1M customers. That's at least $5M MRR when the system is at 50%.

If the scalable version takes significantly longer to develop, you might choose to put it off until you have more money/resources, especially since there will probably be other changes. (And, you don't know where the actual bottlenecks are.)

Side note: this guy is Dominic Szablewski, author of the amazing ImpactJS, JSMpeg, QuakeVR. Probably a 10x developer. He rivals Fabrice Bellard of ffmpeg fame. Check out his site: https://phoboslab.org/

And the rational that validates this was written back around 2006[0]

[0] https://web.archive.org/web/20090306191715/http://www.my-idc...

I feel like you forgot SQLite, and for really small scale you could just use whatever storage/serialization solution comes built in to your favorite language.

It seems like you’re defining “scaling” as growth of a workload to the point that it cannot be handled by a single-server DB.

But with any service without a constant workload (I’d wager almost all services besides prototypes that get no users) you’re going to have to literally scale that one machine, by replacing it with a bigger machine. When you have 50 users you’re not going to be paying for some yy.24xlarge. You’ll start with something much more affordable. When the service grows to 50,000 users, you certainly won’t be at “Facebook scale”, but that t3.small isn’t going to cut it. Should your service ever decline, it’d be nice to scale that machine down to save on costs.

At a previous job, we spent many human hours continually ratcheting up the size of our Postgres machine a few times a year. Not only did this take non-trivial engineering hours and mind-space, it also caused maintenance downtime due to the limitations of traditional DBMSs.

Self-managed CockroachDB eliminates the downtime needed to scale. To handle a more intense workload, add machines. If you want to vertically scale each machine, that can be done without downtime too.

CockroachDB Serverless takes this a step further by scaling up and down to suit the demands of a highly dynamic workload, while minimizing costs.

Maybe what looks like an mega-scale obsession to you is actually a bunch of people trying to avoid the common headaches of managing a moderately sized, dynamic service.

To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.

What's your elevator pitch for why my organization should use CockroachDB Serverless vs. something like AWS Aurora Serverless, particularly if we're already relatively invested in the AWS ecosystem?

Not CDB employee but CDB scales beyond what Aurora can support.

Oh boy, I'm an engineer, but I'll do my best to pretend I'm on the sales or marketing team for a minute...

First of all, CockroachDB Serverless is available on AWS, and should integrate quite well with that ecosystem, including with Serverless functions offered by AWS Lambda.

Here are a few advantages of CockroachDB Serverless that Aurora will struggle to match (note that we're still working on Serverless multi-region support):

1. Free-forever tier. We offer a generous "free forever" tier that doesn't end after a month or a year. As the blog post outlines, our architecture is custom-built to make this economical.

2. No ceiling on write scalability. Even non-Serverless Aurora runs into increasing trouble as the number of writes / second increases past what a single machine can handle. CockroachDB just keeps going. We've had multiple high-scale customers who hit Aurora limits and had to move over to Cockroach to support business growth.

3. True multi-region support. Aurora only allows read-only, stale replicas in other regions, while CRDB allows full ACID SQL transactions. If you want to move into other regions of the world and have latency concerns or GDPR concerns, CRDB is custom-built to make the full SQL experience possible.

4. No Cloud lock-in. Perhaps this is not a concern for you company, but many companies don't like getting completely locked in to a single Cloud provider. CockroachDB works on multiple cloud providers and doesn't have a monetary interest in locking you in to just one.

5. Online schema changes. CockroachDB supports operations like adding/removing columns, renaming tables, and adding constraints without any downtime. You can perform arbitrary schema changes without disturbing your running application workloads. SQL DDL "just works".

6. Cold start in an instant. CockroachDB clusters automatically "scale to zero" when they're not in use. When traffic arrives, they resume in a fraction of a second. Compare that to Aurora, where you need to either have a minimum compute reservation, or you need to endure multi-second cold starts.

7. Great support. We've got a friendly Slack room where you can get free support and rub shoulders with fellow CockroachDB users, as well as CockroachDB folks like myself. We also have 24/7 paid support for deeper problems you might encounter.

Taken altogether, CockroachDB can go wherever your business needs it to go, without all the constraints that traditional SQL databases usually have. Do you want thousands of clusters for testing/development/tiny apps at a reasonable cost? Could your business take off and need the scale that CRDB offers? Could your business need to expand into multiple geographic regions? Are some of your workloads erratic or periodic, but still should start up instantly when needed? It's not just about what you need now, but what you may need in the future. It makes sense to plan ahead and go with a database that has "got you covered" wherever you need to go.

That’s a nice pitch, and certainly catched my curiosity! Going to check out CRDB.

I'm sold. What's your migration story from decades old MySQL Database over to CockroachDB?

I'm only half joking there.

not sure if this is working yet for Serverless but... https://www.cockroachlabs.com/docs/v21.1/migrate-from-mysql....

You will do great with Sales role as well. Thanks for this write up!

8. Online upgrades.

Haven’t yet got time to read the whole thing so sorry if it’s already answered but is it possible to run this sql pod/storage pod separation setup yourself with crdb community/enterprise? We run enterprise crdb but it’s all in one process (with replicas)

It's not currently possible, partly because it complicates the deployment model quite a bit. Dynamically bringing SQL pods up and down requires a technology like Kubernetes. It takes some serious operational know-how to keep it running smoothly, which is why we thought it would be perfect for a managed Cloud service.

What would be your company's reasons for wanting this available in self-hosted CRDB? What kinds of use cases would it address for you?

I’m actually thinking for side proj (I’m very familiar with running production crdb in k8s via my current job) - the usecase is to bring isolated “virtual” crdb cluster per customer to save on operational overdhead of managing many crdb clusters in kubernetes (which is doable but requires extra automation). That’s not how we handle customer mutitenancy at my current job but I’m sure this could be common usecase. Now I could use your cloud but I sure am interested in diy option as well

It's understandly not something a lot of vendors consider, especially early stage products, but classified systems don't have Internet access and can't use managed cloud services (unless offered by AWS via C2S, and possibly in the future by whoever wins JEDI if they resurrect that), leaving no choice but to self-host anything. That's the extremest example, but lots of enterprises airgap part of their network for other reasons, and private companies that don't handle officially classified data may still be restricted in where they can host data based on PII/PHI legal regulations, and may not be able to use your data center.

In addition to security/legal stuff there are also concerns around slo/business continuity and costs

Sure, those are all good points. We'll definitely keep an eye on customer demand in these areas so we can make good prioritization decisions. It can be frustrating when there are so many things that we'd like to do, but we can only pick a small subset to actually do.

I'm quite confused by Request Units, and trying to predict how many would be used by queries/operations.

I launched a test cluster, and the RUs are continuously increasing without me having connected to the cluster yet. At this rate of RU climb, the cluster would use over 8mil of the available 10mil RUs in a month without me touching it.

Coming from AWS, one of the most difficult aspects of using Aurora is guessing how many I/Os will be used for different workloads. It would be a shame to introduce this complexity for CockroachDB Serverless, especially if the RUs are impacted by internal cluster operations that aren't initiated by the user.

I haven't connected to my cluster but my RUs keep going up. Extrapolating I'll be at 20M RUs over 30 days without using it.

You've run into a "rough edge" of the beta release that will be fixed soon. When you keep the Cluster Overview page open, it runs queries against your cluster so that it can display information like "# databases in the cluster". Unfortunately, those queries run every 10 seconds in the background, and are consuming RUs, which is why you see RU usage without having connected to the cluster yet. But never fear, we'll get that fixed.

One thing that may not be clear - you get 10M RUs for free, up front, but you also get a constant accumulation of 100 RU/s for free throughout the month. That adds up to >250M free RUs per month. This ensures that your cluster is always accessible, and that you never truly "run out" of RUs - at most you get throttled to 100 RU/s.

I hear you on the difficulty of understanding how your queries map to RUs. SQL queries can be enormously complex and differ by multiple orders of magnitude from one another in terms of their compute cost. That's why we built a near real-time dashboard that shows you how quickly you're consuming RUs. You can run your workload for a few minutes and then check back on the dashboard to see how many RUs that workload consumed.

Do you have a getting started guide on CockroachDB Serverless? I couldn’t find one in your docs [1]. This looks very interesting.

[1] https://www.cockroachlabs.com/docs/v21.1/example-apps.html

The Quickstart [1] is what you're looking for. Examples for NodeJS, Python, Go, and Java.

[1]https://www.cockroachlabs.com/docs/cockroachcloud/quickstart...

Can I connect using an SSH tunnel (without the SSL cert)?

At this point, only Postgres SSL connections are supported.

I only skimmed the article, but... is Cockroach serverless multi region?

The reply below says yes:

https://news.ycombinator.com/item?id=29005597

How does this serverless offering compare to Fauna?

I would assume the main point is that it’s actual PostgreSQL, not a new query language.

Can this CRDB Serverless offering handle the burst connections of a serverless function based app? Are pooling or query queueing features built in?

Or would users face connection limits at some upper bound until the old function connections get spun down?

> Each node runs in its own K8s pod, which is not much more than a Docker container with a virtualized network and a bounded CPU and memory capacity. Dig down deeper, and you’ll discover a Linux cgroup that can reliably limit the CPU and memory consumption for the processes. This allows us to easily meter and limit SQL resource consumption on a per-tenant basis.

Nice use of K8s here and overall a great post! This is not related to CockroachDB but related to kube and cgroup. I am wondering if you guys have faced this infamous CPU throttling issue[0] when you guys were doing the metering and limiting.

[0] : https://github.com/kubernetes/kubernetes/issues/67577

We haven't run into that so far, but thank you for pointing it out as something to watch out for.

How will you handle PrivateLink and VPC Peering connections into customer VPCs/Vnets with the multitenant architecture?

We're still discussing options there, so I don't have an answer on how we might handle them. We do recognize that many business customers will have such requirements.

How low can CocroachDB go with resource usage?

Is the open source version viable for small hobby projects?

> And you’ll never be surprised by a bill, because you can set a guaranteed monthly spend limit.

It’s amazing that this a killer and not standard feature, but here we are!

Does CockroachDB Serverless expose an HTTP api? This sounds like a great fit for use with Cloudflare Workers, but that requires an http api.

Great question. We recognize how important this is and are actively working on it.

Is it possible to use PostgREST in front of CRDB to solve this use case?: https://postgrest.org/

We're working on enabling PostgREST. There is some small-ish functionality that we need do in CockroachDB for it to fully work.

Why do you prefer an HTTP API versus a DB connection? Isn't the former going to inherently have the overhead cost of creating the connection + TLS handshakes?

My question is similar, which is, is CockroachDB going to have an equivalent RDS proxy so that apps can handle traffic spikes and not have to deal with problems with DB connection pools

I think we won't be getting socket connections in some of the wasm powered JS runtime engines soon. Using http solves that, and a bunch of caching issues.

there are some great projects that present an http api for postgres that you could use with CRDB I think. I'm thinking of something like https://github.com/pramsey/pgsql-http

Now it would be awesome if we could use Prisma against an open standard like this.

I kind of assumed this came with an HTTP api, woops :) I've been using Airtable with Workers for tiny side projects. This seemed like a nice alternative to Airtable's lack of support for vanilla SQL.

Prisma will launch a data proxy, as well as support for cockroachdb and Cloudflare workers early next year.

I think Supabase does, 500mb storage + 2gb outbound data transfer for free.

This offering seems a lot more compelling than the serverless databases offered by AWS directly.

At least if cost and scaling to zero is important to you. Dynamodb addresses the cost issue, but it's so painful to use, and is unsuitable for a lot of use-cases. Aurora Serverless takes 30 seconds to cold-start, so it's also usually a non-starter, other than for batch type workloads.

So great work, but I have a couple of questions:

Any plans to support AWS privatelink? I played around with this a bit, and it seems you need to connect to it over the internet which isn't always ideal.

Will there be a limit on how many free databases you can create? I think there are some valid use-cases where one might want to create a bunch of them, but I would be scared to do this at the risk of being kicked off your platform for abuse.

Our product roadmap will be heavily influenced by customer asks. So if there are things about CockroachDB Serverless that prevent you from using it (like requiring ingress through a public IP), we definitely want to hear about it.

Regarding a cluster limit, we currently allow up to 5 clusters per customer account. I'd like to hear what kind of use-cases you have in mind for having a lot more clusters. One I've thought about is CI runs, where you'd want dozens or hundreds of temporary clusters running at once, in order to run your testing in parallel.

I guess two main use-cases come to mind, providing isolated multi-tenant saas services, also easily creating test/dev environments.

In a similar vein to how you have made cockroach multi-tenant, not too long ago I worked on building a multi-tenant sass version of a business intelligence app. The app uses a relational DB, initially we used separate schemas on the same db cluster, but we had problems with noisy neighbors, as well as concerns about it's security.

We later opted to run dedicated database clusters for each tenant, however it greatly increases the marginal cost, and makes it difficult to provide a free tier of service, which is a valuable way to gain new customers.

I think you're right on target with the multi-tenant SAAS services. They often suffer from the "long-tail" problem, where the tiny customers without much usage cost too much to host. And of course, that's one of the big problems the CockroachDB Serverless architecture solves.

There's no technical reason why CockroachDB Serverless couldn't support that scenario - it's more of a product/business/sales question. I think we'd be very open to talking with companies that have that kind of problem to solve.

The main thing that's keeping me from moving to cockroachDB is that it doesn't support triggers. Once triggers are supported, I'll jump out of aws in a millisecond.

I really see the greatness of the serverless option. Congratulations!

What I can't really understand is why would someone use the dedicated cluster in AWS at that price.

pricing

- Pricing

- storage and db access is still mostly through hoops

- long running processes

- pain to develop, test and debug

I think some of the value of Cockroach Serverless may depend on what the RUs (request units) map to. Looking at some competitors...

Google says that Cloud Spanner should get you around 7,000 read queries per second and 1,800 write queries per second for $650/mo. If a simple indexed read is 1 RU, $650 on Cockroach Serverless would get you around 2,500 reads/second. Of course, I think it's completely reasonable for a Serverless option to cost a bit more given that you'd need to over-provision Cloud Spanner (even if you smartly increased/decreased the amount of compute allocated based on demand).

Planet Scale charges $15 per 100M rows read and $15 per 10M rows written. If an RU is a row read, then Cockroach Serverless would be $10 per 100M rows read. If a write takes 10 RUs, Cockroach would cost $10 per 10M rows written. Both of those would be less than Planet Scale's cost - but it's possible that a row read will cost more than 1 RU. Let's say that an indexed lookup of a row costs 5 RUs. Then Cockroach Serverless starts costing 3.3x more than Planet Scale.

AWS DynamoDB charges $1.25 per million write request units and $0.25 per million read request units. If I can get 10M reads from Cockroach Serverless for $1 and 4M reads from DynamoDB for $1, Cockroach's pricing looks pretty good. Of course, if I need 5 RUs to do an indexed read, the pricing doesn't look as good anymore.

I do respect Cockroach Labs somewhat ambiguous description here. Planet Scale's $15 per 10M rows written feels like something that could become bad. What if I define hundreds of indexes on the table? What if I'm inserting very large blob/text columns that are 50MB in size? Likewise, what if I index no columns and end up forcing a full table scan, but only 10 rows are returned? Do they consider that I "read" 10 rows or "read" all the rows in the table? If it's the former, I'm putting a lot of strain on their system without paying for it. If it's the latter, I'm going to just define indexes that might not be worth it if I were paying for the IO needed to do all that writing.

Still, it would be nice if Cockroach Labs offered some indication of what could be accomplished with 1 RU. "An indexed read of 1 row or an indexed read of a few rows in sequence; for example, 'SELECT * FROM people WHERE age > 18 ORDER BY age, name LIMIT 10' where there exists an index on (age, name)." That would let me know what to expect. "A write of a row under 4KB in size with no secondary indexes will cost 3 RUs; expect secondary indexes to increase the cost by 1 RU each" would give me an idea of what's going on.

I think there are definitely cases that one can't easily enumerate. For example, "UPDATE people SET age = 18 WHERE EXISTS (SELECT * FROM legacy_info WHERE people.id = legacy_info.people_id AND legacy_info.is_adult = true)". That's potentially going to require lots of stuff that's harder to predict. However, at this point I don't know if an indexed read of a row costs 1 RU or 10 RU. If an indexed read of a single row costs 1 RU, if I read 10 rows sequentially will that mean 10 RUs or will 1 RU have enough IO to cover that since they're sequential (or will the billing just over-charge since there's tangibly more rows and that's easy to explain)?

I think a decent amount of the value depends on the pricing and it's hard to judge that right now.

One thing I will note is that the storage seems expensive. It's slightly cheaper than Planet Scale, but a lot more than the $0.30/GB of Cloud Spanner or the $0.23/GB of FaunaDB or $0.25/GB of DynamoDB. I've been wondering a bit about the storage pricing of Planet Scale since $1.25/GB seems expensive. Cockroach Serverless is coming in at $1/GB which also seems expensive compared compared to alternatives. If Cloud Spanner is basically offering a third the price, is the "Serverless" flexibility worth it given that Spanner can be scaled up/down pretty easily in very granular increments.

Actually, one thing that could be useful might be noting how many request units per month one of the dedicated instances would have. A 2 vCPU CockroachDB instance costs $350-400. Would that be 4 billion request units per month (assuming you were fully utilizing the box)? Would it be more like 15 billion request units per month since you're presumably paying a premium for the Serverless flexibility?

Really great post, thanks for sharing. I spent a lot of time a couple months ago researching 'DBaaS' offerings (for fun, not business) and found it difficult to find any posts outlining the architecture of a DBaaS. Really cool to see CRDB putting this out in the open.

It's been something we've done since the start and plan to continue doing. If you read back over our engineering blog, you'll find a surprisingly thorough description of the entire CockroachDB stack, from the lowest storage layer to distributed transactions, Raft consensus, SQL => key-value mapping, online schema changes, cost-based SQL optimizer, Kubernetes usage, and so on. In fact, when we onboard new engineers, we typically point them to a stack of external blog entries to read in order to get up-to-speed on how CockroachDB works.

Being open on how we solve hard problems is the way to build our collective knowledge as a developer community. Certainly CockroachDB itself has benefited enormously from all that has gone before and been published in the open.

One of the first apps to migrate from CockroachDB Dedicated to Serverless is live now if someone wants to try it out. https://web.flightchop.com/dashboard - posting for a friend.

This sounds great! I’ve wanted to create an open data portal for a while that lets you spin up a (ephemeral, read-only) Postgres database of a dataset and run queries on it, maybe with a notebook. Sounds like this might be perfect!

You might like what we’re building at Splitgraph: https://www.splitgraph.com/connect

Oh wow, very relevant indeed! I guess I thought ephemeral DBs would be better so that a user’s expensive query wouldn’t bog down the db for other users. And rather than just limiting them, enabling them to do whatever queries they could with pg running locally

This is something we are currently building at Klarrio on top of YugabyteDB.

We have opted for YugabyteDB table spaces to solve the isolation problem. We allocate a number of dedicated tablet servers to a tenant so that their db objects are close to each other. YugabyteDB executes queries only on those tablet servers where the data relevant to the query is. A resource heavy tenant doesn’t have an impact on other tenants. It’s not really “serverless” but does seem to solve the problem at a cost.

Tenants receive complete databases with grant permissions so they can do as they please.

Because YugabyteDB uses Postgres 11 under the hood, we have an additional sanity layer in form of a Postgres process utility hook which prevents the tenant from escaping the sandbox.

The query layer is composed of ephemeral Postgres instances that get created on-the-fly. These instances either proxy to live data via FDW, or they lazily download the columnar fragments necessary to resolve a query. This is meant to perform as a horizontally scalable caching layer for shared queries. For more optimized/predictable use-cases you can “check out” a version of a data image ahead-of time. In this case you use the `sgr` command line tool to load the objects of an image into a Postgres database that you can then scale vertically as normal.

Our site is a bit out of date — we’ve got a new marketing site coming shortly, and we’ve gathered an all-star engineering team to take this to the next level. Stay tuned :)

(Or better yet if you want a unified data stack for your team, please get in touch… we can load data from 100+ sources into versioned Splitgraph images, or we can index and proxy to existing data sources with federated querying).

[Disclaimer: I work for TileDB, Inc]

We have developed just this[1] except using a MariaDB storage engine (MyTile) we've written. You can serverless run queries against TileDB arrays without spinning up a MariaDB instance. You can run any type of query MariaDB supports (joins, aggregates, CTE, etc). I've linked the basic documentation and an example notebook below[2]. You can run SQL queries from python/R or even JS or curl. We support a number of data return formats, i.e. arrow and JSON to facilitate use cases.

I'll also mention that we have a number of public example dataset[3] in TileDB cloud, such as the NYC taxi data used in this notebook[4], which you can explore!

[1] https://docs.tiledb.com/cloud/api-reference/serverless-sql

[2] https://cloud.tiledb.com/notebooks/details/TileDB-Inc/Quicks...

[3] https://cloud.tiledb.com/explore/arrays

[4] https://cloud.tiledb.com/notebooks/details/TileDB-Inc/tutori...

The lack of a serverless option was the only reason that I did not use Cockroach on a recent project. I am excited to see the serverless offering now, and the architectural details in the post are awesome. Nice work!

Any plans to expand the region availability for AWS beyond us-west-2 in the US? I am interested in us-east-2.

Yes, definitely, we'll be expanding to more regions soon.

How does this compare to Snowflake?

Snowflake is OLAP and billed based on usage time + storage. This looks like it’s a regular OLTP SQL DB and pay by request.

I see. Looking at the docs, the pricing for storing data is measured in Gigs (I think $1 per gig per month) - not Tb. This makes me think it is intended for smallish data.

I have a database now with 40 Tb - this would cost $40k a month just to store!

Yikes! I think this is really positioned for side projects and microservices. There is a clear path to migrating to a full instance or Postgres cluster over certain usage levels, but for many cases being able to scale to zero is still economically and operationally valuable.

40Tb is a ton-o-data! Snowflake sounds like a good fit.

Yeah, they get style points for "scaling to zero". I've seen plenty of vendors who "are scalable" - but with a floor of $70k/yr.

I like their product offering - I'd be interested in using them for hobby projects, but not for work atm.

Reply on news.ycombinator.com