Hacker News Reader

Comment by haolez | Hacker News Reader

How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.

View on HN

The maximum size will increase to at least 1GB in the near future, enabled by our recent rewrite of the underlying storage layer. Perhaps we can push it further, even, we'll see.

But yes, I think the next step is then some sort of sharding. Sharding by user would be an obvious approach for many apps. I think we should build a framework to help manage this, so apps would only need to provide some callbacks e.g. to compute shard key for a particular query.

Alternatively, apps that want full control will be able to use Durable Objects directly. We'll soon (in a few months, probably?) enable the new storage engine for all Durable Objects which means every object will have a private SQLite database.

(I'm the tech lead for Workers in general, and currently focused on this project in particular.)

do you think you will have any API driven access to the data held in these services? IE, I don't per se want to have to use a worker to arbitrage between the Cloudflare platform and HTTP access to the data within Cloudflare systems.

Much like R2 has an S3 compat API as well as a way to retrieve files outside of workers (IIRC) will this be true of all forms of storage on the platform?

Sometimes it'd be nice to prime the cache with KV for example, without having to invoke a worker

> do you think you will have any API driven access to the data held in these services? IE, I don't per se want to have to use a worker to arbitrage between the Cloudflare platform and HTTP access to the data within Cloudflare systems.

For D1, yes — on the roadmap is a native HTTP API. KV has a HTTP API as well, so you can write directly.

Awesome, thanks for the info! We are building more and more features on Cloudflare and absolutely love the whole experience, faultless.

afaik, that's how it is envisioned to be used. Many databases rather one big monolith. The era of microdbs is here. Who's not going to bite?

>Should I create one database per user, for example?

You can't. D1 won't let you create databases at runtime. At least that was the case last time I tried it, and it's what killed it for me.

Not really sure how you distinguish between runtime and something else. It seems to me that you could provision a "shard" as simply as executing in a worker:

    wrangler d1 create customer28 --experimental-backend
    wrangler d1 execute customer28 --file=schema.sql

What I mean is that workers cannot create databases. You have to create each database using wrangler (or the UI) and then edit the configuration file to add a binding. The binding becomes a variable in the JavaScript environment, which means you have to change the source code every time a new database is created.

Therefore, creating a database per user as part of the user signup flow is not possible.

Ah, I see. I thought it would be possible to dynamically set a secret which would effectively be a database uri. But upon further investigation, it does indeed need a redeploy.

I suppose you could make something to write the bindings to the toml and trigger a redeploy, but that's definitely not pretty.

The linked https://blog.cloudflare.com/d1-turning-it-up-to-11/ page shows pricing for databases over 5GB

the docs say the Alpha is limited to 100Mb, and, given the size of SQLite DBs i have seen, 100Mb is quite large, is it not?

SQLite is readily capable of vastly bigger databases than that. Whatever limitation there is here isn't from SQLite.

The SQLite website recommends using a client/server DBMS "when the size of the content looks like it might creep into the terabyte range"

https://sqlite.org/whentouse.html

Reply on news.ycombinator.com