Deno Queues
deno.com
deno.com
The most interesting detail is probably the schema they're using for that:
CREATE TABLE queue (
ts integer not null,
id text not null,
data blob not null,
backoff_schedule text not null,
keys_if_undelivered blob not null,
primary key (ts, id)
);
CREATE TABLE queue_running(
deadline integer not null,
id text not null,
data blob not null,
backoff_schedule text not null,
keys_if_undelivered blob not null,
primary key (deadline, id)
);
CREATE INDEX kv_expiration_ms_idx on kv (expiration_ms);Isn't that exactly what indexes were designed for?
I'm not as familiar with sqlite, but it might have a similar problem in this case since the primary clustered index is on rowid and not a value that correlates with whether the queue item is running.
Indexes are just the tip of the iceberg (but sometimes just using partial indexes may do wonders), it’s ability to tweak stuff like fillfactor for a frequently updated small portion of the table (e.g. active sessions vs archived sessions) is that makes a lot of difference.
That’s quite novel, but I also find it a bit unnerving. They gotta commercialize of course, but they didn’t have to go fully closed. They could’ve just used a different license (BSL, PolyForm..) for the scaling layer.
I'm using Go for the first time this year and some aspects of the language are very C like. Of the list of things that isn't C like (e.g. GC allowing one to return what look like stack allocated pointers from functions) there is the obvious inclusion of things like `map[string]string`. I bring this up because it struck me that inventing a language in 2023+, it would seem almost insane not to have built in syntax for the language to handle map types.
And so it seems logical that a web-server focused eco-system would start to garner libraries and even language syntax (maybe one day) for the kind of primitives we frequently use on web-servers. I mean, I can't even recall the last time I worked on a distributed web server that didn't have a KV store as a cache, or a locking mechanism, or even an ad-hoc queue. A distributed system without a KV store feels vaguely isomorphic to a computer language without a map type. It is such a Swiss army knife kind of technology.
One potential issue is that Deno is going alone down this path. Currently, I don't feel confident that the new features they are adding will be available on competing platforms. Even if they open source the API (it is on a `Deno` namespace), I'm not sure it will just work on AWS if I wanted to switch out FoundationDB for e.g. Redis.
For that reason, I feel I want to avoid Deno. Even if the syntax and exposed features are really attractive, I'm worried about becoming locked in and then having to do a lot of surgery to code to make it deployable on multiple cloud infrastructures. That is a requirement from a lot of clients. E.g. maybe I want to sell to Oracle or Salesforce one day but they mandate I have to run my systems on-prem. I'm now on the hook trying to figure out how to adapt whatever available KV store they have to the features I'm using from the `Deno` package.
It is a double edged sword. Maybe they will succeed in pushing forward this vision to a broader audience. For now I'll probably remain cautious.
I kicked the tires on this with a pure TS implementation of the protocol called kv-connect-kit that gives you the KV client api in any Javascript runtime (including Cloudflare workers, which does not have anything Deno namespace related)
- github: https://github.com/skymethod/kv-connect-kit
- npm: https://www.npmjs.com/package/kv-connect-kit
- deno/x: https://deno.land/x/kv_connect_kit
- demo: https://keyspace.deno.dev/
protocol seems to works as described on the tin, and it would be pretty straightforward to write another backend
What is really the difference between Deno.KV being shipped as part of the runtime vs adding an `import KV` statement at the top of the file (which could come from Deno or wherever else you want)?
And what are the chances that the Deno team is going to ship bindings for any language besides their own?
If Google started adding Google Cloud specific primitives natively to Go would you call that forward thinking as well?
This is not the case. The Deno runtime itself is not tied to the Deno Deploy hosting service. The KV feature in the Deno runtime can be used without the hosting service.
You can read the details about how Deno KV works in the Deno runtime here: https://til.simonwillison.net/deno/deno-kv (as has been posted in other comments)
But one writes to foundation, and the other writes to a sqlite file. You wouldn’t be able to self host an app written for Deno Deploy and have it work out of the box.
Are there any plans to open source the KV backend so that people could host their own KV databases? Now that you can connect to remote Kv databases, I suppose someone could implement their own?
As someone mentioned elsewhere, they have documented the protocol, so yes you could reimplement your own remote KV store.
The 1st party implementation is closed source: 3rd parties start on the back foot trying to implement alternatives and have to keep up with a 1st party that can move in lockstep.
And sure enough, like every other time I see this kind of behavior: Deno was invested in by the CEO of Vercel.
"Javascript is taken over by venture capital" wasn't on my 2023 bingo.
Right now the JS community has whipped themselves into a frenzy into building on VC backed technology.
- They refuse to acknowledge that the loudest voices in the room are openly sponsored and invested in by the same VCs who own the companies behind said tech
- They see no issue with a lack of diversity in implementations, instead settling for "it's a standard". Of course, defining a standard without a healthy variety of implementations means you end up with standards that don't benefit from a wide range of voices until well after they land (see RSC)
At the end of the day, those two alone are a pretty harsh combo: A VC-backed network effect machine built across multiple brands, and high technical costs to building something that meets the collection of standards.
I don't think anyone but FAANG can really compete with that without also getting VC dollars, thus reinforcing the loop.
Next.js and Vercel heavily push serverless deployment: 13 reworked the built-in API support to leverage Web Standards, which discarded interop with the a much larger server ecosystem in order to enable better edge support.
Serverless deploys require providers to support the Next.js Build API: https://nextjs.org/docs/pages/building-your-application/depl...
There is no open implementation of this API (unlike Remix for example)
This means projects like Open Next start from 0: https://open-next.js.org/
The end result is significantly fractured support for a headline feature of the framework and a lot of unnecessary pain (https://betterprogramming.pub/beware-of-next-js-on-aws-ampli...) trying to leverage it on any non-Vercel platform.
The same difference between `#include "myStringMap.h"` and `map[string]string` in C vs.g Go. There is some advantage to everyone in an ecosystem using the same primitives. That kind of language-level standardization goes a long way.
For example, it might increase the ecosystem of available distributed system libraries that only need KV stores. If I can compose several libraries, all of which use the same underlying KV interface and implementation - I can imagine that might result in some interesting use-cases.
As for the rest of your comment, I would humbly ask if you read the entirety of my comment? We seem to agree that as long as Deno KV is locked into their cloud that people should be very wary of lock in. And I myself will likely avoid it for the time being until the dust settles. There is some chance the rest of the community will just come along and fill in the gaps on popular cloud platforms. Or maybe they won't. Time will tell.
Except one is a language feature with custom syntax and the other is a library that could be implmented as a library just the same.
Not everyone in the ecosystem is going to use this because its not forced on them through syntax and espically because its inside deno and not node so barely anyone at all will use it.
Go actually ships with a quite forward thinking SQL interface. It's an abstract interface over a DB, and you just import the "driver" that powers it. The driver conforms to a standard interface, so all of them behave roughly the same.
I think this is what everyone wants from Deno/etc - why can't there also be a KV interface that's universal, or a Queue interface that's universal?
People attempted this w/ go [1], where it attempts to use the same nice experience of the SQL logic, but it never seemed to gain traction.
SQL is low hanging fruit in this regard, because you just need to standardize the lowest common denominator flavors of SQL types for deserialization and then it's just juggling SQL queries around.
JDBC for Java does the same as database/sql and it's from 1997. ODBC is from 1992.
Deno is basically trying to be an infrastructure framework for JavaScript.
Yeah, I'm hella confused.
Isn't Deno the Node.js replacement?
But now it's a database as well?
It's jumped the shark for sure.
For example erlang ships a persistent KV store, queue, relational database and much more as part of its standard library. I've never heard anyone complain about this or wish it were otherwise.
I contend that this is not actually forward thinking, but monetising Deno as a SaaS, which every man and his dog is doing in the JS world lately with their 'cloud' offerings (reselling AWS with a framework). That's why there is a pricing page attached to this.
If Deno's KV depends on FoundationDB then they're hardly going to build adapters over other databases - switching DB tech is always a massive ordeal because they all have different use-cases and performance characteristics.
I think this is due to having escape analysis and SSA rather than having a GC.
I don't buy this line of reasoning. At the end of the day we are building infrastructure that needs to be reliable. Spending 30 minutes to set up an SQS queue (proven technology) doesn't sound as bad as putting all my trust in a toy queue that Deno built "on top of SQLite / FoundationDB". Is the 30 minute setup cost and added "developer experience" really worth the risk? How often are you setting up queues anyway.
As a hobbyist programmer, I don't use the big providers like AWS and Google because they seem rather complex? Maybe it's not so bad if you're used to them.
I also like to make each project independent, as its own repo on GitHub. Ideally, a web app would be easy for anyone else to launch, as their their own independent web app, using a separate domain name, because I don't want to be responsible for their data.
(This is sort of the Sandstorm use case.)
"Try it out locally and get a Deno Deploy account if you want to use it for real" seem like reasonable install instructions?
As for making project's independent, so do I, but that's simple in other clouds too with infra as code or one of the various deployment frameworks (fly lets you do this, AWS copilot does this, terraform, serverless)
I looked at Fly and they have nice docs and interesting ideas, but they’re also clear that they don’t do “fully managed” databases and I don’t want to be a DBA.
I know of AWS as a huge pile of complexity that I’m not sure I want to get into? Using it directly seems too low-level for me
(I’ve used Digital Ocean and it seems more my kind of thing, but it doesn’t scale down to zero for a website that gets no traffic.)
Looking at Terraform, it’s not obviously about solving any problem I care about, and wasn’t there a huge controversy about them?
Serverless is a buzzword. Isn’t Deno sort of serverless too?
I think of Deno Deploy as a step up from Netlify, which is fine for static websites. Or maybe a reincarnation of App Engine, which I quite liked back when it launched, but it seems to have lost its way.
but fair enough.
I do think the simplicity of deno will be a trap though. The benefit of a mature framework is you don't have to get lost in the weeds on obscure bugs or functionality with slightly niche use cases.
Any other systems a "20 something person" would cobble together for an app of any complexity from "toys" of today would be a complex web of half solutions. They'd be trying to do the same stuff that can be done within AWS, and often what they cobble together would be worse off having it made of disparate components from maybe dozens of different vendors. I can't see how it's any easier to connect all those dots than it is to do it within AWS. If you're new and doing simple stuff on simple systems that's all well and good, but don't expect it to scale easily or at all, and if you try you're in for a whole lot of dev-ops and networking and other bullshit just to get things to talk to each other. There's a lot less of that in AWS. Usually you just copy an ARN and paste it into another box, and the things are connected.
>Barriers to entry are something you have to consider.
There's no barrier to entry for AWS, unless you can't afford $0 per month. Anyone can sign up and use free services, and they are very well documented. There's tutorials galore, probably more than any other current toy platform(s). There's tools and tooling and all kinds of support out there for it. But sure, some toy platform might be more fun to use for your hello-world task tracking app if you aren't building anything serious.
So ridiculous.
Use a workflow engine which abstracts queues for you so you can "just write code" without managing jobs/schedules/state/retries.
https://www.inngest.com/blog/how-durable-workflow-engines-wo...
(disclaimer: I'm biased as I'm an author of a workflow engine)
The code becomes "wait until this time", vs "enqueue this function with this state to run at this time via this message broker, which may enqueue other jobs in ways you can't see".
There's no real silver bullet that will let you run code in the future without specifying "when" that code should run — but workflow engines are by far the more productive of the two.
But people are way too afraid of simple tech these days. If there aren’t dedicated QA and Ops teams, at least 2k GitHub stars, it’s not web-scale™ and meant to run on a fleet of 726 servers at minimum, it’s not to be trusted!
But at the infra end of it, it's a different story. AWS is suited towards a very different scale of development effort than Deno currently is. The effort required from scratch to securely and reliably get to a point where a dev can just name a queue is substantial in AWS. You really need account hierarchies, guard rails, roles, permissions, delegated IaC etc etc in place to be able to do that - otherwise it eventually gets out of hand.
AWS is geared towards larger dev teams or groups of dev teams, while (to me at least without using it) Deno seems far more oriented at the smaller scale set and forget PaaS oriented teams - eg Heroku refugees etc. Those teams could very well outgrow Deno and need AWS at some stage though.
Very infrequently. Even more infrequently as we move to IAC and use terraform or even cloudformation.
> user code.
A past life has taught me that users will never properly understand at-least-once semantics. Anytime you redeliver messages, you will get a flurry of user complaints and breakage.
Either you do the impossible and invent a way to do exactly-once semantics. Or you should always redeliver 0.1% of all messages, just so that users don't come to depend on messages being delivered once.
For the local version you could get multiple queues by calling Deno.openKv("db-2.db") with different SQLite file paths each time, but that feels like a lot of overhead for a pretty common need.
I guess this is a Deno architectural style thing - maybe when you build complex apps on Deno it's expected that you'll have a microservice style architecture where lots of different scripts work together, each of them with their own KV store and hence their own queue?
One of the core devs has confirmed this on their discord: https://discord.com/channels/684898665143206084/115671428253...
Quoting here:
> Correct. Currently a single queue is supported. You could multiplex multiple types of messages on the single queue though.
I thought Deno was some sort of node.js replacement. What am I missing and can I use this either locally and/or self hosted without paying for it?
> Since Queues are built on Deno KV, it uses SQLite when running locally and FoundationDB when running on Deno Deploy for maximum availability and throughput.
I wrote a bit more about this pattern here: https://til.simonwillison.net/deno/deno-kv
Although, from reading the Foundation DB docs and checking the Deno KV API, I honestly suspect it is a thin layer.
Self-hosting FDB is somewhat inscrutable though, so their value add is in not having to handle infrastructure while being backed by FDB.
I keep very detailed notes on everything I'm doing already - either as a VS Code scratch document or Apple Notes or often a GitHub Issues thread. A lot of my TILs start by me pasting those notes into a Markdown file and tidying them up.
(I'm not a production customer, and I haven't thought through how this ought to work in deno-deploy-land, I've just seen a lot of painful ACL set ups. But I also like knowing that lambda function foo written by the intern can't read every DB table)
But I won't write servers in Deno and put them to production in my own servers. (I don't like serverless)
You see, Deno(and Bun)'s business model is built around serverless (Deno deploy). And Deno deploy is ANOTHER runtime.
It's clear with KV and queues... Locally and if you self host, you have a barely working version backed by another tech...
This means you're self hosting a version of Deno that's different from the majority of the Deno customer base.
How many people will use Deno to run servers on their own hardware? Since that number is low, then I'd rather use Node.
I could maybe see the appeal if you prefer to have a single company in charge of your language runtime + compute hosting + data storage, but I'd personally want to avoid that, especially when the company doesn't have a track record in it.
About Deno Deploy, I completely disagree with the analysis that it's just like any other lambda service. It's not just about easy deploys, the service itself is simply better. There are no cold starts and they don't just do that by keeping a vm up all the time. It's really magical. You should give it a second look.
Do you know how they're doing that, technically? Is it running on Cloudflare under the hood, or if not does it have a similar set of tradeoffs and benefits vs Lambda? Or is it a third completely different way with different tradeoffs/benefits?
Deno Just Works for me. `deno run file.ts` and you are good to go.
Not to say it's been 100% smooth sailing; I have hit a few rough patches with deployment into my own infrastructure (some within Deno and its dependency management, some within libs like sqlite3) but overall I am happy to not deal with nodejs.
I don't have the time to try to port any of my meaningful projects to it right now, but I look forward to the day when I can do so.
With deno all that setup/configuration is handled by deno. There’s obvious trade offs but I think what deno is doing is pretty cool.
- ScyllaDB
- Riak
- Couchbase
- Cloudflare KV
- Many different AWS offerings (DynamoDB, ElastiCache)
- Similar alternatives from Google & Azure if you are in those ecosystems
Of course there's nothing unique about a KV store. You can use literally any SQL or noSQL service out there, or set up your own in any way you want.
They also all have the advantage that you will get bindings for every language, not just Deno.
[1]: https://docs.deno.com/deploy/api/runtime-broadcast-channel
In case anyone wonders what this would make possible out of the box: https://apple.github.io/foundationdb/design-recipes.html
It does show a level of understanding by the language creator that they know how the language can and probably should be used - I like how KV defaults to SQLite locally and takes on new meaning in a hosted environment. I haven't seen, but as long as you can actually override the behavior to use your own KV/queue technology (so long as it satisfies the interface) I see absolutely no issue with this.
There is a huge benefit where the code always looks the same for common things we all need to do. I wouldn't mind seeing other languages take an overall stab at this so there is some choice other than JavaScript.
Some questions:
* Looks very powerful, but at the core is it just one single queue? Does Deno.openKv() create a new queue or re-use the same queue every time it's called?
* Usually I need multiple queues of different priority, so when bottlenecks happen the highest priority jobs run first.
* Maybe they guarantee infinite worker capacity so you don't need queue priorities? No bottlenecks if you can pay for the jobs you enqueue?
When you use it locally the database is only on your device however all deno processes can access the same dB so you can use it to pass data around just like you can for localStorage that deno also supports.
When you use it in the cloud (deno deploy) then saves are replicated across regions.
Deploy is server less.
KV is simply a wrapper around sqlite with that you get atomic transactions that you wouldn't with local storage
The API is standardized for both.
[1] https://www.google.com/search?q=Use+database+as+a+queue+is+h...
A bit of an aside, but not sure we want at-most-once delivery for email.
Enqueuing a Message: Each enqueue action translates into a KV write operation.
Receiving a Message: Every received message entails a KV write, and a single request charge.
I really like this
With grand promises rode the ploy
Venture capital, ahoy!
We make a lovable product to capture audience
Vendor lock them at the earliest convenience
Hey look at our awesome developer experience
Add ANSI and emojis and comments saying it’s genius
Some next level tech, plebs just don’t get it
Next round of funding, we’ve already spent it
I took the big dough, got puppets and framed it
R-O-I is bad though but layoffs gon’ fix it
Never mind the drama *cough* are ya gon’ install it
Run it, ship it, kill yourself with working on it
Entangle your business with it
Tell the client to deal with it
Bring it to the big wig
Get ’em to sign off on it
Crown your own sh*t
Dogfood is tasty, ain’t it Human brains proudly present
Real poetry? Grouchy resent!
My rhyme is not gradient descent
After all, it is barely nascentI understand its only when using their cloud and there is a local implementation, but I just couldn't imagine using a feature in LLVM, for example, where if you were to upload the binary to a specific cloud it would use a different implmentation that costs behind the scenes.
It just feels odd to me. I guess it's kind of cool. Maybe different clouds can implement the underlying KV and then this becomes cross-cloud with different underlying benefits without changing the code. But I'm not entirely sure how I feel about this.
Deno was released 4 years before Deno Deploy and existed just fine.
Is there any full stack solution other than Fresh?
For one, this is just a simple queue with no routing capabilities. It's a much simpler tool.
For two, and more importantly, it's built into your application and introduces no additional operational dependency.
You could probably achieve a pretty similar API with a custom AMQP library, though!
edit: people seem to be getting hung up on the "runtime" v "programming language" distinction. I'm not sure why--it's weird to me that this is "part of the language + runtime" at all. Clearly, reasonable people can disagree about that, but hopefully on points more substantive than pedantic
If tomorrow, someone started a for-profit company making a faster Python runtime and introduced features like this, it would be weird, and it would feel as pointless to me as Deno is
KV is free to use locally as I assume this will be too.
(It didn't really happen with App Engine, but this seems like a cleaner API?)
> Deno KV databases are replicated across at least 6 data centers, spanning 3 regions (US, Europe, and Asia). Once a write operation is committed, its mutations are persistently stored in a minimum of two data centers within the primary region. Asynchronous replication typically transfers these mutations to the other two regions in under 10 seconds.
Generally: the terms "programming language" and "runtime" are synonymous. There are a few examples where this is not the case (Javascript is definitely the best one; Java/JDK; python). There are hobbyist runtime re-implementations of some languages. But generally: Only academics worry about differentiating them.
Language is more than syntax. Language is the combination of Syntax and Meaning; its a system of communication. The word "Fooloofol" is syntactically correct English, but meaningless within the English language; and thus is not English. The english word "bark" is syntactically identical across more than one meaning; a dog barks next to the tree bark. Language isn't just syntax; its the entire system of understanding.
The statement "fetch("https://example.com")" is syntactically valid in any ECMAScript-compatible language. Its also syntactically correct Python, Go, Rust, and probably a few other languages as well. But: it is not meaningfully correct code in many of the runtimes which implement that language. It was only valid in NodeJS after version 18, after all.
What you're really asserting is: Node is not a syntax. That's more accurate; but still inaccurate. Node does have its own syntax. For example; if you were to execute "await myFunc()" in NodeJS v6, you would get a syntax error. So, if Node isn't a syntax; what syntax does it use? JavaScript? By what definition? ECMAScript? Certainly a subset of the global standard, or a version of the standard released in the past. They're quite bad at keeping up. 99% syntactic compatibility with another syntax is still a different syntax; and thus it has its own syntax.
The old "you're writing HTML and CSS, that's not programming" runs in a similar vein. It is programming, by any reasonable definition of programming. You're instructing a computer to do something. The real assertion is that its not procedural programming; which seems like a pretty dumb differentiation to lose sleep over to me.
But, ok: Node advertises itself as a Javascript runtime, but the standards body against which they are tracking makes it an ECMAScript runtime; not more, not less. Probably not more, I'll grant you; but certainly less! You can pick any formally standardized version of ECMAScript, and find that Node.js has incomplete support, in some cases even for years after publication [1]. Again; its close enough nowadays, especially outside of ESNext which really doesn't count, that we're not talking about a useful or meaningful difference; but rather a pedantic difference.
"Languages" formally specified independent of implementation are not languages. Language is the implementation; and the specification guides it. I'm not just talking about programming languages. I'm talking about spoken and written language as well.
For lack of a better specification; The Oxford English Dictionary is not English. The dictionary is both more and less than english. It has a very large center-of-the-venn-diagram. But: commonly spoken English words will always exist for which it doesn't yet have definition for (it's as of yet blissfully unaware of "bussin" and "rizz"). Simultaneously; it will have words that make no sense to modern speakers, or definitions that have fallen out of use.
It's the same situation with programming languages; the language is the implementation. The specification, if one exists, guides the implementation; but without absolutely no-more-no-less 100% coverage of the specification (which has not happened in NodeJS), and absolutely no-ambiguity-in-the-spec (which has never happened in any formal specification), they are meaningfully and inherently different. Because zero-ambiguity is impossible given fundamental constraints in politics, communication, and in a very real way physics: the specification isn't the language.
Here's an interesting fun fact: TypeScript; that programming language we all love. It has no formal specification. Yup! People have been asking Microsoft since 2016, when they last published the specification, to update it, but the team has (explicitly or not, I don't know) taken the stance that the implementation (and its test cases) are the specification. The language is the implementation.
So; what language is Deno-compatible code written in? The only accurate, formal, academic, correct answer is: Its written in Deno. The language is the implementation. Productively and usefully; its written in a language that is approximately similar enough to TypeScript that no one will notice.
That's not really what Deno is. It's more vertical than purely a Python runtime-equivalent.
Having said that, the CPython runtime includes SQLite, so it's not far off anyway.
Can you explain further?
Thanks
> Opening a database > In your Deno program, you can get a reference to a KV database using Deno.openKv(). You may pass in an optional file system path to where you'd like to store your database, otherwise one will be created for you based on the current working directory of your script.
In fact, only the last page in the documentation speaks about KV on Deno Deploy.
But, I think this section hints at how it could be done [1]. Its phrased in the context of connecting to the DenoKV-hosted database from outside Deno Deploy, but it seems like the reverse could also be accomplished with the same API.
[1] https://docs.deno.com/kv/manual/on_deploy#connect-to-managed...
> Don't want to use Deno Deploy? Deno KV works on any VPS, so you can deploy to your favorite cloud hosting service.