Deno KV internals: building a database for the modern web
deno.com
deno.com
I used it a couple of times locally with Sqlite for CLI apps, if you want to do some data manipulation stuff with TS from the CLI and need a db, don't look further.
I also used it in Production with FoundationDB on Deno Deploy.
It does not replace your postgres/mysql database but a different beast entirely for many reasons. One is pricing. You pay per read and write.
An issue I had is that it's hard to migrate from using KV with Deno Deploy. You can migrate to a local SQLite backed instance but will need to develop your own solutions on how you migrate and it will cost more the larger your database gets because you pay for reads.
I do think it's great, but I would recommend using Deno Deploy only if your reads and writes produce value that offset the costs, else you can find yourself in the issue of needing to migrate.
For example, use it for features you offer to authenticated users, but don't use it for things available in the open, else you open up yourself to high fees from DDOS.
Are any of the these tools open source? Would love to look at what you're doing
Stuff like parse some XML or JSON and output Go structs and functions and Typescript types and functions, HTML,React Components, SQL Tables, Stored functions, Pl/PGSQL.
Mostly to avoid writing boilerplate when I use the same data structures in the database, middleware, client. For simple CRUD apps, it works well. I use local KV to track changes so I don't have to rerun things I don't need.
But Deno is great for CLI reporting tools or Scheduled tasks, fetch and aggregate data.
I think of Deno as a little swiss army knife. It's a tool that got everything built in.
I use the Repl a lot, just for a specific task, get it done then move on.
1) You are starting with a blank canvas
2) You will be dealing with low-level primitives
3) You have complete freedom in what you build
4) It's fast if you use it right
For each one of these items you could look at it and say "that's awesome" or "yikes." You know which group you are in!
I hope more people in the 'awesome' group realize that using FoundationDB can transform the development of a new distributed data store from a multi-year project into a few weeks of hacking. This is because FoundationDB solves the scalability, fault tolerance, and ACID parts for free--the least creative and most time-consuming aspects.
The best resource after the tutorials is: https://github.com/FoundationDB/awesome-foundationdb
Unfortunately, I think a lot of the advanced 'tricks of the trade' (alternatives to use cases for long-running transactions, when exactly to dispatch to a cloud object store, how to migrate whatever schemas you end up creating, etc.) that all the big serious users of FDB are doing are not as well covered.
The same game plan every. single. time. Get into the JS developers' zeitgeist, wrap an existing cloud service with a new frontend, sell it at a markup.
Bonus points if the services can sell for each other.
React has RSC because Vercel wanted to fight Shopify. NextAuth was practically bought out by Clerk to serve as sales funnel. <img> tags are marked as "not good" by a leading React framework because the hosting provider behind it wants you to pay them for image optimization.
What Rauch is doing is the developer equivalent of private equity squeezing, and what's insane is how well it's working.
No one is saying they're not useful at all, the problem is Vercel (or really Rauch's entire cartel of companies) strong arm different technologies and topics to fit a narrative that's disproportionately in favor of "use our thing".
RSCs are not amazing if you're not Vercel and don't have access to their closed build format (especially after they killed the serverless output format)
I use image optimizers, I'm not about to go around telling people that img tags are bad.
> Finally, there are lots of alternatives to what you're stating, no one forces you to use NextJS over, say, Remix or Vite with SSR.
Remix had to reject RSCs to start because as one might expect, having one (1) design partner doesn't make for the most fully baked proposition.
Also the "there's other choices" refrain ignores the fact that no one is developing in a vacuum. Vercel is pumping enough money into the ecosystem that the mindshare is shifting towards Next regardless of technical merit. So it's not enough to say "who cares if Vercel distorts what Next is, just use something else"
As I've worked in PE companies, I know all too well how they operate. It's just a shame that developers are naive and clueless about it.
Dan Abramov (react core team) already said that the React team wanted to do RSC, Vercel were the followers that were eager to integrate and productionize what the React team wanted to do.
NextAuth is a competitor to Clerk auth. How is it bought out? Because Vercel pays an OSS developer to further develop NextAuth, and Guillermo also invested in Clerk? Someone using NextAuth means they're not using Clerk.
The implication is that you can't do anything that relies on node APIs that edge doesn't support which can be quite limiting.
There's rumour that they're walking back this decision, but it always struck me as an arbitrary way to "encourage" codebases to make themselves compatible with edge which in turn would make deploying using Vercel easier.
(In general I'm reasonably happy using NextJS, though there are many architectural decisions that I find frustrating)
One problem I saw with Vercel, and a reason why I steered people away, was that they were very slow to react to some of the challenges of serverless workflows, like building their own in-house services to reduce latency or allowing different kinds of serverless tiers for longer-lived workers. You could hit growing pains almost instantly.
I don't want to write negatives about it, it's a well thought out product. But it 's not free. It is a paid service at the end and that is fine as long as you know what you are getting into.
Is it a "replaceable part"?
Why do you think that?
> So, we set out to build two versions of the Deno KV system, with the same user-facing API. A version of Deno KV for local development and testing, built on SQLite, and a distributed systems version for use in production (especially on Deno Deploy). This post is about the implementation of the production, distributed systems version of Deno KV. If you’re interested in the SQLite implementation of Deno KV, you can read the source code yourself as it is open source.
Intentionally crippled open source software for the sake of selling a cloud subscription really isn't a great feature Deno should ship.
I think this situation is very different than Supabase. Supabase really publish the whole thing as Git repositories, and you can run it on your own servers if you wish.
You can also use localStorage in Deno but it won't synchronize the data between edge servers on Deploy, so you have to figure out another strategy.
and of course KV works without Deploy, so while you don't get the data distribution benefit, you can still use KV on other deployment platforms.
The alternative, of course, is to use another database. You don't have to use KV in your Deno apps.
Maybe what people want here is the ability to configure their own data synchronization mirrors? Is that already implemented?
Anyone know were Deno’s “Edge Servers” are? Is it anymore than EC2 instances because it doesn’t seem to follow the Akamai, Cloudflare, Fastly type approach of actually deploying their own hardware PoPs at the edge
I think it speaks to how out of control many sre groups are that people willingly spend so much on tools to avoid them. This has real echoes of how the cloud got traction as a way to avoid internal IT.
We found it to be super easy to set up and scale and it. Setting up FDB is literally install a package and make sure a file gets to all nodes in a cluster and you’re good. It also integrated really well with our function runtime.
You get a lot of distributed goodies for free in FDB with little effort and they “stack” their primitives very well too. As an example, there’s a built in multi-tenancy layer that is just using key prefixes under the hood, but it’s built in natively and can be accessed by the higher level apis.
It’s interesting that Deno went with a full separate transaction layer per region on top of a global cluster instead of doing regional clusters and making one region the primary writer and then doing request coalescing.
I knew Deno KV was built on FoundationDB, expected this would be a neat but simple systems architecture rundown. But... Turns out Deni really went super deep in building a KV, by adding atomics!
> To maximize performance and concurrency for atomic operations, we built the Deno KV Transaction Layer that manages the global order of atomic operations. For each transaction that is received by this Transaction Layer:
I thought it was particularly creative how a .get returns not just the value, but some reference to what the get was. So when the atomic change comes, they can check the get. This was a neat way to let some sets use previous data safely, I thought.
Does anyone else have any other examples of thicker kv APIs? It feels like we're nearing a cap'n'proto level of promise pipelining, as we extend kv this way; I though Denos dodge was extremely expertly picked to limit how complex references to existing operations would need to be.
The entire industry has been twisting itself in knots trying to solve ops problems that Erlang/OTP solved in software long ago.
I'm trying Deno Deploy though, because it seems like an attempt to combine those benefits with the JS ecosystem. That has advantages in: Language usability, frontend/isomorphic, libraries, and serverless.
So far it feels like the future. Something like this will be, though I'm almost expecting a new language for it.
I wish CF had a premium tier where both Workers and KV would persist the the edge.
D1 will replicate soon, but doesn't do so yet. My understanding is that it will replicate to ~5 locations and keep these locations fresh.
deno deploy just showed a completely useless error message and stopped working.
I installed caddy on a vps and got it to work in 5 minutes.
I ripped out the deno compatibility and switchded to bun sqlite instead of turso and the website got even faster.
That is because the server has the data in the sqlite db. No need to roundtrip to turso.
btw. the framework is open source now: https://github.com/spirobel/mininext
It is like a non bloated version of nextjs. My intention is to displace php and wordpress. It is possible to get started with just html and css knowledge, but you have the whole javascript ecosystem at your fingertips.
It has no external dependencies, just 3 modestly sized files you can understand in an afternoon and it will reload your browser and rebuild when you save your code.
Difference between deno and bun is night and day. They really shouldnt be put in the same bucket.
The big difference was I used the “record layer”, however, naively believing this meant only needing to understand the record layer . . . No. In foundationdb it is vital to understand all the layers up to and including the one you are using, partly so you can make sense of the documentation, but also because the problems you run into are so tied to the semantics of the lower layers.
That said, the great thing about fdb is deploying it is so easy. It is like a secret weapon hiding in plain sight.
The game https://luduxia.com/showdown/
The funny thing is I modeled the service for leaderboards on a commercial product an old boss of mine wrote v1 of in PHP/MySQL over 20 years ago. https://web.archive.org/web/20040806114005/http://www.macros...
Games people end up with things like massively sharded mysql and replication fun. One of the nice potential things with fdb is keeping transactions within user scope, and not having to deal with the sharding/replication yourself, you just have to arrange the keyspace correctly instead. I have worked on games where people sorely underestimated the complexities of db replication and literally burned tens of millions in recovering from the mistake.
One of the main things about it is you don't want to be updating the service for every new game, so you defer as many decisions as possible to the game, and be ready for it to make changes over time. The nasty part is this significantly complicates extracting the data for the leaderboards as scores are added, but you do that once and it's done. On some level it's like mongodb with a special index type.
A trend I see with too many efforts is to be way too overly specific, classically things like claiming something shuffling byte buffers around needs to know about the data format. You get surprising mileage and long term velocity out of embracing the lowest levels first, and allowing them to be built on arbitrarily later.
Maybe I didn't read it right though
A read-modify-write retry loop causes a high number of commit conflicts, e.g. for atomic increments on integers. Getting higher than 1/RTT per-key throughput requires the "backend" to understand the semantics of the operations - apply a function to the current value, instead of just checking whether timestamp(value) < timestamp(txn.start) and aborting commit if not.
For those that haven't seen it, SQLite on top of FoundationDB: https://github.com/losfair/mvsqlite
So this is a NoSQL argument.
> Atomic transactions: a database that provides atomic operations to ensure data integrity and enable complex application logic
Ok, so let's see if they mention "ACID". Nope. They do mention atomicity and durability, but they don't mention consistency and isolation.
So this is about NoSQL and NoACID, but it's not really stated this way.
K/V stores are great for many things. And not having a query language is great!! right up until you need one, then you get to be sad (I should know since I've had to write a query engine due to use of a K/V store).
[No]ACID is where NoSQL gets interesting, IMO. But NoACID doesn't mean you can't have a query language.
Anyways, if you're going to implement yet another K/V store you really should justify it in relation to other K/V stores that exist, not as "using RDBMSes is ETOOHARD".
I remember on launch, Deno specifically did not add backwards compatibility for installing Node.js packages from npm, by design. It was supposed to be a "greenfield" approach to serverside javascript/typescript... buuut then they folded and added npm dependency mgmt to their tooling.
Some of the decisions in Deno feel like the "grow and find ways to monetize" strategy of your average vc-funded tech startup, but instead of being a SaaS it's your whole runtime.
Since most companies go with VS Professional, zero adoption.
Same applies to unit testing code coverage on VS.
Oh. Deno KV uses SQLite3 under the covers. That's... funny.
Besides, if you're going to layer a network service on top of SQLite3 then things like rqlite and similar start to look very good, though, admittedly a K/V store is much simpler.
by actively seeking to meter DX, they've actually driven dollars to their competitors.
(Several clicks in it looks like https://github.com/denoland/denokv is the repo and it's an MIT license.)