Hacker News Reader

Comment by jgrahamc | Hacker News Reader

This no need to compile SQLite into your Cloudflare Worker. We provide it native on our platform as D1. And it gives you replication. https://blog.cloudflare.com/d1-turning-it-up-to-11/

Also, I think the idea of “edge” doesn’t make a ton of sense. What we really need is code and data that move around as needed for the best performance. See: https://blog.cloudflare.com/announcing-workers-smart-placeme...

What people call “edge” is a single optimization bringing code near the user. I think we should go far beyond that. Sure, use something like Cloudflare Workers for your “edge” needs (ie bringing your React app close to the end user/doing server side rendering). But don’t stop there because where data resides, what APIs you call are all going to matter.

That's the vision of something I called the "Supercloud": https://blog.cloudflare.com/welcome-to-the-supercloud-and-de...

View on HN

How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.

The maximum size will increase to at least 1GB in the near future, enabled by our recent rewrite of the underlying storage layer. Perhaps we can push it further, even, we'll see.

But yes, I think the next step is then some sort of sharding. Sharding by user would be an obvious approach for many apps. I think we should build a framework to help manage this, so apps would only need to provide some callbacks e.g. to compute shard key for a particular query.

Alternatively, apps that want full control will be able to use Durable Objects directly. We'll soon (in a few months, probably?) enable the new storage engine for all Durable Objects which means every object will have a private SQLite database.

(I'm the tech lead for Workers in general, and currently focused on this project in particular.)

do you think you will have any API driven access to the data held in these services? IE, I don't per se want to have to use a worker to arbitrage between the Cloudflare platform and HTTP access to the data within Cloudflare systems.

Much like R2 has an S3 compat API as well as a way to retrieve files outside of workers (IIRC) will this be true of all forms of storage on the platform?

Sometimes it'd be nice to prime the cache with KV for example, without having to invoke a worker

> do you think you will have any API driven access to the data held in these services? IE, I don't per se want to have to use a worker to arbitrage between the Cloudflare platform and HTTP access to the data within Cloudflare systems.

For D1, yes — on the roadmap is a native HTTP API. KV has a HTTP API as well, so you can write directly.

Awesome, thanks for the info! We are building more and more features on Cloudflare and absolutely love the whole experience, faultless.

afaik, that's how it is envisioned to be used. Many databases rather one big monolith. The era of microdbs is here. Who's not going to bite?

>Should I create one database per user, for example?

You can't. D1 won't let you create databases at runtime. At least that was the case last time I tried it, and it's what killed it for me.

Not really sure how you distinguish between runtime and something else. It seems to me that you could provision a "shard" as simply as executing in a worker:

    wrangler d1 create customer28 --experimental-backend
    wrangler d1 execute customer28 --file=schema.sql

What I mean is that workers cannot create databases. You have to create each database using wrangler (or the UI) and then edit the configuration file to add a binding. The binding becomes a variable in the JavaScript environment, which means you have to change the source code every time a new database is created.

Therefore, creating a database per user as part of the user signup flow is not possible.

Ah, I see. I thought it would be possible to dynamically set a secret which would effectively be a database uri. But upon further investigation, it does indeed need a redeploy.

I suppose you could make something to write the bindings to the toml and trigger a redeploy, but that's definitely not pretty.

The linked https://blog.cloudflare.com/d1-turning-it-up-to-11/ page shows pricing for databases over 5GB

the docs say the Alpha is limited to 100Mb, and, given the size of SQLite DBs i have seen, 100Mb is quite large, is it not?

SQLite is readily capable of vastly bigger databases than that. Whatever limitation there is here isn't from SQLite.

The SQLite website recommends using a client/server DBMS "when the size of the content looks like it might creep into the terabyte range"

https://sqlite.org/whentouse.html

Thank for sharing this, that makes lot of sense to me. The focus on running everything on edge has been confusing to me for a while due to the increased db latency. That makes so much sense to dynamically move processing to the best location in the current network instead of the binary choice “close to db”, “close to users”.

We've finally come full-circle with shared hosting.

For our product, (SQL your cloud & saas resources as a service) we want to consider D1, but for security we would want database per tenant. Would that model work for Cloudflare? 1000 databases for 1000 tenants. Or should we use something else with shared tenancy?

Not quite sure understanding SQL your cloud (using their AWS account?) but from what I guess by the consideration of D1: I'd wager you want something like Neon but self-hosted. I am building a similar product where I give each customer a full-blown Postgres (without individual extensions) and that is the solution I landed on with the most flexibility and security after testing out SQLite vm per user (via Fly.io), SQLite on the edge (Worker + S3, before D1 existed) and Postgres cluster per user (via Stackgres)

The nice part is the decoupling. User not using their DB for 5 minutes? Time to idle them and use that compute for another customer.

And it's also really nice that storage becomes way way way easier. Storage is the biggest hurdle when your database count scales with your user count so you can instead build out the compute system once, fix a few issues along the way and from then on just babysit a single big storage cluster.

And if that becomes too big at some point, replicate it, shard the tenants by region or whatever and you "only" have to manage storage a dozen times or something. It's obviously always annoying, but at least the setup is similar enough that you can park on FTE per Storage cluster and serve enough customers to make it worth it

Edge does make a lot of sense, but only if you’re actually talking about physical computers that need to exist at physical locations to do no-network/embedded systems/rtos without the cloud, but with a cloud-metrics or aggregation layer (or other apps) which explicitly do not need to be on a nearby computer to help the users, and often serve as a control plane for a set of edge resources. Think Tesla wanting to do BI on every car vs. having the car actually work for the user when they decide to drive through the network-disconnected wilderness.

Server side rendering or “single optimization” as the “edge” is kind of an abuse of terms.

Does D1 support FTS?

`wrangler d1 execute abcdtest --command 'CREATE VIRTUAL TABLE people USING fts5(firstName, lastName);'` succeeds with no errors, so I'd say it does.

Reply on news.ycombinator.com