Hacker News Reader

Comment by billllll | Hacker News Reader

Building a KV store into the language is kind of nuts. I love how it abstracts away the local SQLite and deployed FoundationDB behind one interface. Testing would be super easy, as there is no question of "do I spin up an entire db instance or mock the db interface?" as SQLite is relatively lightweight and the burden for keeping both cases consistent falls to the language instead.

That being said, was wondering whether you can take advantage of this without using deno deploy, or if you'd be locked in by using deno kv. Also, wondering how many bugs will only show up when deployed due to the different backbends.

View on HN

I dunno.

Is having a database that happens to be a product you sell as part of your runtime good, or are you creating some mixed incentives here?

Are you a database vendor now?

If not, don’t build a database.

If this is a mature product that someone else is looking after, and it’s good and free and we’ll maintained like redis or SQLite, sure.

…but it seems like building cloud databases is not the core competency of a group building a javascript runtime.

Once money is involved, it’ll either be a distraction from their core mission, or become their core mission.

A KV db you can use for quick hacks? Sure. Awesome!

A scalable production database you’re selling to people? O_o why are you doing that?

Deno Inc. has two core products: The free Deno runtime and the for-profit Deno Deploy, a Deno hosting service with 35 locations around the world. The question that often popped up was where to store data. Deno Inc. provided several guides to connect to different cloud services. But they want the friction reduced to a simple `await Deno.openKv()`.

Deno Inc. has enough expertise in running a global service that two other companies rely on their work to offer edge functions to their customers (Netlify and Supabase). Adding a database to the service makes sense. And to be clear, they don’t develop a brand new database. They build atop of SQLite and FoundationDB.

> The question that often popped up was where to store data. Deno Inc. provided several guides to connect to different cloud services.

Sure.

> But they want the friction reduced to a simple `await Deno.openKv()`.

Do “they”?

If so, who’s using it to solve that problem? …because it seems the big uses of deno deploy are not using it, fine with that and it’s pretty unclear who the “they” is in this circumstance.

Still, if it’s a thin layer over foundation db or some other established database product and this is just part of the lock-in for their cloud offering, fair enough.

It’s not like others (eg firebase) don’t do the same thing.

The messaging “we’re building a database” and “we’re offering a hosted database service based on existing mature reliable technology” are different things though.

The latter all cloud vendors do.

The former is ridiculous, and it really really wasn’t clear that wasn’t what was happening.

"They" in OP's post is referring to Deno Land Inc.

I think this kind of "batteries included" approach is interesting. My feeling is that in many cases we spend way too much time when building our platforms from various bits and pieces. Some apps and businesses would get huge productivity boost by using more opinionated platform.

Not saying this is for everything and everyone, but there's so much competition on this space that I can understand why somebody would like to try something different.

They need this because Deno Deploy (their Edge platform) is not a normal/traditional deployment target.

If Deno were nothing but "Deno CLI" (the equivalent of the 'node' executable, that runs as a typical server process on a Unix box) then you would be right. But there are two separate Deno runtimes (CLI and Deploy).

It's a lock-in strategy. There is zero reason for a language runtime to be opinionated about the database. All the good programming advice will tell you such a coupling is a bad idea. What if you want to move parts of your application to a different language in the future, do you need to redo the entire database? That is just nuts!

Database providers, cloud application platforms should be agnostic of the language their users are using. There are many such decisions deno is making where they want to be the end-to-end service. That is not a good sign. It creates perverse incentives for the company to devise lock-in mechanisms at each stage and discourage compatibility with the rest of the programming ecosystem, perhaps even undermine the rest of the ecosystem. I would stay away for this reason.

> There is zero reason for a language runtime to be opinionated about the database.

The API is extremely simple. It's little more than a hash map. That's hardly opinionated.

> All the good programming advice will tell you such a coupling is a bad idea.

That advice is outdated. Virtually any modern application will want a database, and this API can serve as a foundation for it – a foundation that conveniently already has many of the features (esp. global replication and consistency) that you will want and that are incredibly difficult to get right if you build them yourself. Think of this as the application's "file system". PaaS today usually doesn't provide direct disk access, so this is the low-level abstraction for persistent storage that is available. An abstraction that, in many cases, will in fact be all your application ever needs.

I'm not familiar with the offering but find your comment confusing. It seems to suggest there isn't lock in because it's just a hashmap, but also that it is globally replicated and consistent.

If you don't deploy to deno cloud it's just SQLite and not distributed.

If you do deploy to deno cloud it's trivial to migrate because there are many competing offerings of kv stores.

If no claim of compatibility with other backends is provided I would be skeptical that it will be trivial by necessity.

I'm not that familiar with either technology, but as an example, deno kv claims strong consistency as the default; while dynamodb requires you to explicitly ask for strong consistency in queries and does not support strongly consistent reads on secondary indexes.

When not in deploy, you'd be stuck with a non distributed SQLite database for each instance your application is hosted on. There currently isn't a way to swap the backend for the deno kv API, so you would still need to update your code to use the new kv store that would have a different API. If you use redis instead, you'd be able to easily be able to switch providers by changing configuration options instead of having to update your code wherever it is used.

> The API is extremely simple. It's little more than a hash map. That's hardly opinionated.

In that case, why does Deno have to provide this? Can't users simply create their own hashmap?

> There is zero reason for a language runtime to be opinionated about the database

* The Erlang runtime has Mnesia and ETS built in. Batteries included. It's awesome.

* Python has a built-in persistent key-value store, dbm.

* Ruby has one as well, pstore.

Now, you can always decide to use something else! 99% of the time m not going to use dbm, or pstore, but they are nice to haves. (Mnesia and ETS rocks though, and I use those all the time.)

Uh oh better tell python to get rid of pickle and _sqlite3_ modules from its standard lib. Seems too opinionated...

This KV thing is not meant to replace your main database. I see it more for catching at the edge.

Yeah maybe you'd prefer to do that with an agnostic solution like Redis (which afaik is possible in Deno) but everything has pros and cons. An e2e solution is coupled but otoh it has many objective advantages. And it's not like Deno will force you to use their cloud service after all.

> What if you want to move parts of your application to a different language in the future, do you need to redo the entire database? That is just nuts!

How often does that happen?

Either way, it's not like you can't export your data and shove it into something else like redis.

iOS apps have an opinion about which Db to use and why it should be core data.

There is a good little overview of Deno KV here (by @simonw): https://til.simonwillison.net/deno/deno-kv

So it's backed by SQLite on the OS local version.

I'm intrigued if there will be a way to swap out the backend for other cloud providers.

I think what's nuts is that databases data structures are not built into the language/stdlibs. I can make an array but if I want to add an index so I can access it fast then I have to jump through some inordinate hoops. I can make an in memory hashmap, but if I want to persist it and keep accessing it fast directly from persistent storage suddenly things get unnecessarily complicated.

I think fast searchable file backed data structures should be primitives of any computer language and runtime.

To me this is quite similar to the Erlang VM's ETS in-memory store. There's a lot of usefulness in ETS itself, especially how it distributes inside clusters. Having an easy to use, efficient, and optimized store as part of the language can be a good enabler. Only time will tell if it is/isn't.

It's not really built into the language though? It's just a function on the global Deno object, similar to how there's a global 'document' object in browsers (and that's not 'built into the language' either).

Backed into the runtime is probably what your parent meant.

> Building a KV store into the language is kind of nuts.

Isn't it part of the standard library rather than the language? Anyways, Erlang has mnesia [0] and ETS [1]. ETS is practically a KV store, which can also be persisted to disk.

[0] https://www.erlang.org/doc/man/mnesia.html [1] https://www.erlang.org/doc/man/ets.html

I think almost all languages should have a database built in. Almost all programs need a way of storing and querying data but the drama required to hook up to an external db is excessive for little programs. Basic dB functionality should be as available in language standard libraries as file access.

What do you mean by drama to hook up external db? It’s almost always just a one liner: let conn = db.newConn(host, port, …).

One line _after_ you figure out the myriad of database libraries available, half of which are abandoned, a quarter which are little toys, and if you're lucky one that has a core dev who is being paid real money to support and maintain the library.

Ah yes, that's fair. Finding the good library to rely on can be really time consuming.

Deno.dev (deploy) uses a different, proprietary runtime than Deno. I believe Ry mentioned at a recent talk that on deno.dev, the kv will be synced across all instances. Obviously this can’t be the case for stock Deno.

KV storage is SQLite, and SQLite replication and syncing using an out-of-process daemon is becoming a solved problem.

For that application at least no changes to the runtime would be necessary

Building a KV store into the runtime, that has more features when using deno deploy is just that: vendor lock-in. It's just one more way to force you into paying for the service when you have locked yourself in and you can't use an alternative store.

Deno is not and has never been an "alternative runtime" to node.

I am confused. In my book vendor lock-in would refer to a situation where a user or customer would be unable to switch vendors or only with prohibitivly large cost and often under the notion that they were unknowingly lead into that situation.

So first of all, _you_ can use whatever ORM or database solution you like in general so _you_ specifically choose which technology you want to invest in. If you want to hand craft your kubernetes cluster for your database nodes and put them on GCP or Azure, then go for it. You want redis? Go for it. If you want to use supabase, firebase, planetscale or whatever, use that. If you want to choose a KV store that's built into your runtime then that's great. Believe it or not but people use lowdb in production and their database is literally a json file. I would argue whichever you choose, switching from one hosting provider, database solution, managed or unmanaged platform to another will always incur cost and how high that is will depend on your depth of integration and familiarity with it. I am most likely going to use this feature and I am happy it exists. Also let's not ignore that there is a large breadth of different needs that developers have. Would I build a massive modular corporate CRM system with BI features with it? Probably not. Am I going to build my next D&D game tool, habit tracker or Tinder for sneaker trading for it? Maybe!

I sincerely wonder what specifically you loose from this feature simply existing.

If they provide a compelling product offering, and the people understand what's available where, and choose to use it, then I guess that can be called vendor lock-in. But it's hardly nefarious.

It is nefarious when you build up your entire product for years, spamming it everywhere as "node but better", yell "it's so open source too!" on every roof, only to sneakily slide in lock-in features, it's worse than being nefarious. You most likely had this plan in mind all along, and actively lied to get there.

Watch as the same happens with Bun.

What's stopping you sticking with Deno CLI, using it for free forever, and completely ignoring anything related to Deno Deploy?

In this relationship of 3 years and counting, they are the positive contributor because they give you something useful for free and you presumably haven't given them anything.

In this specific case I think it's super reasonable.

The KV store has two backbends: local sqlite and distributed deno cloud.

If you choose the deno cloud backend you're signing up for a closed source cloud service from the start. There's no trickery.

Why not just serialize objects to disk? I don't see how this is different from a javascript object, other than being disc-based. Unless I'm missing a detail?

Also, local KV obviously won't work for compute lambdas, as the KV isn't shared across instances.

The Mumps language did this in the 60s.

Reply on news.ycombinator.com