Big changes ahead for Deno
deno.com
deno.com
npm compatibility is huge for Deno. It is basically the one major drawback to Deno, which gets hopefully fixed with this feature. It also looks like this compatibility layer is implemented transparently in the existing module management, which is a big bonus. Nobody wants to deal with node_modules anymore when working with Deno for an extended period, just love to see this.
That level of incompatibility is the banner headline not some minor footnote, which is effectively how the bun team decided to communicate that little detail.
If they're going to do things all-over-again...again in the JS community couldn't they get it right this time and go with speed, stability, compatibility...
Instead of hype and cute mascots?
(No idea if there's any relationship between the projects, just makes me think that article's a bit dismissive.)
Core features for npm compatibility though (like implementing most of the npm api, including require(), and its special globals) has been making progress for a long time.
There is even a near complete project to support Node's native api (which bun also supports): https://github.com/denoland/deno/pull/13633
Relative to the the existing node compat efforts, the url import (and accompanying package download code) will be fairly small. The most difficult part is probably the extra stuff that node throws in the global namespace. How to handle that without having to pollute the global namespace for all programs (even those that do not import from node) is unclear.
Ideally these globals would be visible only from code in modules imported from npm. But the spec does not really allow for this unless the npm code is loaded in a different realm, but cross realm code causes a lot of headaches, which could only be avoided by having the realms share most globals and intrinsics (and sharing intrisics is not allowed by the spec).
There may be some other way to hack this into working, or perhaps programs with such imports that actually use NPM specific objects will need to be run with the "--compat" flag. It is really unclear at the moment.
The development is well under way, we expect to ship first iteration of this feature in the next release (v1.25) in the coming weeks.
Like you said, don't know if Bun had an influence on the timing of this announcement, but great to see innovation in the JS server space. Kudos to both the Deno and Bun folks!
BunJS does of course have a slightly different use case, packages or apps that still need to make use of Node APIs. I can see a future where we say "if you have an old Node app, you can run it on BunJS with a codemod and get an X% perf boost for free", as well as "if you are starting a new project, just go with Deno because there's less BS to think about".
Don't personally know anyone who is treating Bun as anything more than a curiosity.
Any thoughts on speed reliability concerns comparing the two?
NPM is a giant pain point, and I suggest that Deno and a lot of other people want to move on from it.
But the gravity is just too much.
Sorry.
Feel free to gather and build your dependency trees by hand, if you believe this to be worth it. Most packages have their sources and build instructions readily available on GitHub/Lab :)
Don't downplay it's importance and it's disadvantages that came along.
There are too many examples where changes to packages have had serious consequences.
Left pad and node-ipc to name the better known.
I don't think NPM is the best way to create a package manager, in fact there are many choices they made that I think are rather stupid and led to many more problems than there needed to be (whoever decided that the default for `npm add` should be a caret dependency instead of a tilde should never be allowed to work in the industry again, IMHO). But I'm not going to blame them for a problem that definitely existed in the package management world before them and will continue to exist long after everyone forgets about NPM.
Which is why I use `yarn`, I guess.
This is rather unfair.
Your previous comment referred to NPM in the broadest possible sense, so what else would you expect its audience to do but to interpret it in the same way?
It is primarily a package manager and so I assumed you didn't like it for how it functions in that capacity.
That is not to say that I disagree with you - it's their priorities and stubbornness on major issues that I myself take issue with, but I'm complaining too much already in general so I'll leave it at that.
Until I get a customer requirement to use anything other than nodejs, then I will consider another look into them.
This seems like an acceptance that moving on simply won't happen. The existing ecosystem is too big and too powerful to be ignored. If I were a particularly cynical person I'd also point to Deno taking on millions in VC funding - once you've done that you can't be content to create a perfect, gleaming sandbox. You have to bring in money.
Not being able to use these packages is a detriment to Deno's adoption, not a boon. IMO, the biggest issue with Node is Node itself, not its ecosystem. All Deno does is bring the JavaScript ecosystem to a more modern standard. Being able to pick and choose the npm packages you want to use while slowly adopting the Deno stdlib and Deno specific packages will make it easier for developers to make the switch.
Once Deno ships these features I'll try Deno again. Right now I'm really turned off by JavaScript development. The last couple of times I've tried to start up a project on Node it's been an absolute nightmare with its ESM madness and lack of TypeScript support. On the other hand, getting started with Deno was a breath of fresh air... up until the real development starts.
I've always loved Node.js. But the unfortunate reality is it sucks to build new projects in it in 2022.
We are consuming sources as they are provided in npm using "compat layer" that is part of Deno's standard library. We had to provide special module resolution, but besides polyfill for built-in Node APIs, it works very similar to how Node consumes these packages.
When contemplating how I might do it, I came up with centralised reexport, a file deps.ts containing:
export express from "npm:express@5";
So then I looked up the docs and found https://deno.land/manual/linking_to_external_code#it-seems-u... and was amused to find the same solution, even down to the file name “deps.ts”!
Is this how people tend to work with Deno?
I see it also supports import maps, https://deno.land/manual/linking_to_external_code/import_map..., which could readily solve the problem too. Do people use that?
In fairness, the two examples, Deno and Go, are successful projects, so maybe it's not a bad initial strategy.
And now because you've hard-coded imports at every level, you can end up with a dozen different versions of the same import, causing potential conflicts between the various components of your app.
Package managers evolved to be the way they are _because they're useful._ It's not like people said "let's add all of this complexity for fun!"
Direct imports failed as a solution for Go, and they already have been acknowledged as an incomplete solution for Deno (import maps are already a larval form of package.json, but without the ability to inherit child import maps, which means it's still broken, but presumably it will eventually be fixed).
Not having a standardized place where you can see the dependencies of a project, and not being able to distinguish which are _development_ vs _runtime_ dependencies, is also rather broken, and would require a package analysis tool that's far more complex than a package manager to use to trace the dependency tree reliably.
Deno has a concept of locking dependencies to specific versions with a lock file: https://deno.land/manual/linking_to_external_code/integrity_...
Its the same stuff npm does but it doesn't depend on npm's CLI. It sounds like you're mad they're moving the cheese a bit. IMHO I'm glad to be rid of the npm ecosystem baggage.
> Deno can store and check subresource integrity for modules using a small JSON file
Isn't this also "a larval form of package.json" as the parent suggested? (I want to learn about Deno)
Having a file to manage locked dependency versions isn't bad. Having a file that has grown into a crufty monster with millions of uses can be bad.
In that respect, package.json is a strict win. Your lack of willingness to use `git blame` to see why you added a line, or lack of reasonable git comments, is not to be blamed on the file.
Complexity is unavoidable. How could you write a tool like license-checker [1] for a Go-based project without having license information in a standardized location? Without the scripts section, how can you create a tool like husky [2] that automatically installs git hooks for a project? Every single part of package.json is there for a good reason; at best you could argue that putting some of it in other files would be aesthetically superior, but that's just bikeshedding.
Complexity isn't de facto bad. Some complexity is required if you want a certain level of functionality to become available. Deno (and Go) are slowly accumulating that "cruft" as people realize that those functions are actually useful or even critical to a mature ecosystem.
The lock file format is also TypeScript-file based, which ... well, since dependencies can point at arbitrary git repos, you could easily end up in a situation where the version tag on the repo is changed and then you can't find the file you need to download. It's a pretty worthless lock file if the file can just disappear from the internet; a lock file should exist to guarantee that a package can be rebuilt exactly, not just prevent it from being rebuilt if it can't be rebuilt exactly, which is all the Deno lock does.
npm on the other hand will prohibit anyone from deleting an older version of a package if anyone else is using that package. "leftpad" can never happen again now in Node. Having a centralized repository is actually a Good Thing; being able to search through all the packages in one central repo and determine relative popularity of each is also useful.
Also: The lockfile is a generated shrinkwrap file. That's only a tiny piece of what I'm talking about. What I want to be able to do is set the exact version of an indirect dependency without needing to fork and rebuild every dependency. See yarn resolutions [1] for an example in the npm ecosystem.
No, I'm not mad because it's different. I habitually chase the bleeding edge up to and including jumping between frameworks and build systems and even languages and editors in constant search of ways to improve my development workflow. I thrive on learning new ways of doing things. I was enthusiastically digging into Deno shortly after they very first 0.x announcement, but quickly saw a number of showstopper flaws (including lack of npm compatibility) that caused me to write it off.
No, I'm complaining because Deno is missing crucial features. "Lack of features" isn't a feature on its own, especially when people use those features. (See also the Go language.) I also would miss the "scripts" section of package.json, since it's a centralized location for the various commands that a project might need; things like "generate a new migration file" or "run both the client and both servers" vs "run only server A".
This is all complexity that was swept under the rug by Deno and that will need to be replaced by community standards that won't be standard.
[1] https://classic.yarnpkg.com/lang/en/docs/selective-version-r...
And to comparing to other languages: It's not like "ok, projects became to big we need better package management" or something, but to me reads more like "there's too little deno stuff and migration prevents people from going here, so let's try to embrace the other runtime's stuff so maybe more people can use our runtime without having to start from scratch"
I'd say it is on a successful trajectory. The main issue with it was NPM compatibility so I think this is a very very wise move, even if it is pragmatic rather than ideal.
I'm very interested in how packages will manage runtime support in the future. Right now we are seeing an explosion of JavaScript runtimes from Cloudflare Workers to Bun, all with sightly different APIs. I wonder if there will be an easy way to get cross-runtime support without it being much extra work for library authors.
Right now, there has already been conflict with some packages being browser-only and some being node-only. I feel like there needs to be a better solution than the one we have now.
Browser, node (OS-level), or serverless worker, these are all fundamentally different roles and need different API requirements. Node and serverless don't have a DOM. Serverless can't have unfettered access to the host OS. Node should be as capable as any host language (Python, Ruby, C, etc.) with an API equally powerful. The browser obviously needs a more limited sandbox.
There is a parallel universe out there, where Sun had success with Java in the browser, the JVM evolved to better support distribution and HTTP standards as well as replaced the need for WASM. And we all lived in a happy world with Java on the server and Java in the browser and everything is cross-platform and Just Works(tm). Cats and dogs also live together in harmony in this world. We had big dreams in the '90s.
Hermes might show up in the engine radar due to how prevalent RN is, but other than that, you can bet 99% of projects run Node.js or Deno now.
As long as these are actual standards, all the other runtimes also get a fair shot at implementing them. I think it's a move in the right direction.
> There will be no node_modules folder, no npm install; the packages will be automatically downloaded in the Deno cache.
My understanding of deno's package system (see https://deno.land/manual/linking_to_external_code#it-seems-u...), is that basically, at compile time it will go and fetch external URLs and cache those files locally. So this change basically makes it so your import:
> import { assert } from "https://deno.land/std@0.152.0/testing/asserts.ts";
Is magically generated from npm when you do this instead:
> import { assert } from "npm:testing@2.0.1";
Ok cool. Whatever.
...surely this isn't the actual problem?
There seem to be certain pretty fundamental problems:
- not all npm packages are typescript
- a package is not just an 'assert.ts' file, it's often an amalgamation (eg. using rollup)
- a package often has many dependencies
- a package may use the node API, that deno doesn't have (it has a more-or-less compatibility mode, see 'differences that cannot be overcome' -> https://deno.land/manual/node)
So bluntly, how on earth is this going to work?
I mean, I'm a fan, the folk working on Deno are smart and motivated, and if it works, I'm more 'wow' than 'I don't believe it'... but I'm very surprised to see:
> the vast majority of npm packages work in Deno within the next three months
That seems... ambitious.
I can't see how the approach is really sustainable, given that basically, it means 'anything node supports we have to support to'; doesn't that mean you're forever playing catchup and 'doesn't quite work'?
How is vendoring going to work without a package lock file? Doesn't this just mean you've reimplemented npm and you'll forever be playing catchup to the various npm features that are required to make packages work?
Imports don't need to be written in typescript. They could have manually written type definitions (which is how people using typescript in the node ecosystem already handle this), and packages without type definitions will have the imports typed as `any`. Sure that makes them a bit annoying to handle especially in strict mode, but it entirely possible to use such packages.
Deno is actually working on adding the native node API (https://github.com/denoland/deno/pull/13633), so the website was a bit overoptimistic when saying "Deno will never support Node plugins". However it is possible for some plugins to be incompatible still, for example if they try to use node internals rather than just the API, or if they try to use utilize the libuv event loop, since deno is not libUV based.
All Javascript is Typescript isn't it?
Or to put it another way, JS is only a subset of TS in as far as you're willing to forgo the benefits of TS.
From the docs:
> Typescript is a strict syntactical superset of JavaScript
Typescript contains more than JS. But all JS is valid Typescript.
That, and the emphasis on speed (esp quotes like "We aren't optimizing for a handful of edge cases, but for overall real world performance") makes me wonder if Bun is lighting a fire under them a bit. It's still a long way from catching up, but it's been closing the gap at a blistering pace and getting enormous amounts of hype. The more competition the better.
Exciting times
It almost sounds like I could write a self-contained deno command-line "app" that does this. Does that seem right (for those that know better)?
[1] https://raw.githubusercontent.com/microsoft/TypeScript/main/... [2] https://unpkg.com/typescript@4.7.4/lib/tsc.js
This is just for personal projects. But I sometimes write JS code for the browser and I tend to not want to use any 3rd party libraries. But I do like the extra type checking. So I just want something that does the type checking and does TS to JS -- but without installing npm (which I have some irrational dislike for -- maybe just because I don't understand it enough).
npx is installed right alongside npm in Node installs for some time now.
Learning npm is still often a good idea when working with Node. One thing that may be useful to learn here: npm has a concept of a development dependency needed only for development, not run time. You can install one with `npm install --save-dev typescript` for example. (Or `npm i -D typescript` if you want to save some typing.)
You can clean out dev dependencies with `npm prune --production`. Or if you are in a fresh install situation (a fresh git clone, for instance) and you want to skip developer dependencies you can `npm install --omit=dev`.
I find the distinction between production dependencies and developer dependencies quite useful.
This will give you tsc as a global command line to compile typescript to javascript.
ts-node is optional, but it is the standard node cli that can run typescript
After you install these two you can just ignore that you have npm forever.
(... but yes you can also just download the js file as the person above suggested if you're really stuck on being anti-npm)
TS "compilers" basically strip out the type annotations, they don't care about the validity of the types, so you'll be missing out on the main reason to be using TypeScript.
1. Switch everything to esbuild, remove 10k dependency security nightmare, bloatfest and slugishness of pretty much every other node based build system / bundler.
2. Switch to deno.
Haven't quite got to deno yet, but #1 was liberating. esbuild also has some preliminary deno support, but not sure how stable it is.
The main thing I like about esbuild is that I feel like I wont ever have to "migrate" build system again (and I have to deal with an unusually high number of repos so this is a big deal), not much in my repos have anything esbuild specific, migrating was more a matter of removing build system specific smells from repos, it's all just a js and css bundle now. i.e there could be something better than esbuild in the future, and switching to that should in theory be effortless. This has never been true for all of the NPM build systems prior, (I was there at the beginning with grunt).
> - Provides web platform functionality and adopts web platform standards. > - Supports TypeScript out of the box. > - Has built-in development tooling like a dependency inspector (deno info) and a code formatter (deno fmt).
Also they have a build-in testing library, so for me is mostly get rid of the tooling as dependency, the only thing that is missing is a frontend bundler (I think they have one, but intended to use for the backend only), but there are swc and rebuild that hopefully will catch the feature parity with webpack and friends.
I guess I'd want a Typescript compiler that does the "type checking" before stripping out the type annotations. Like that's how TS is supposed to work.
If ESBuild doesn't do the type checking, then it probably doesn't fit my need.
Edit: In fairness, if my IDE does the type-checking, this is almost good enough.
This is really huge and will be a huge boost to the Deno ecosystem. On the other hand, I quite enjoyed that it wasn't jacked into NPM. There were reasonable alternatives like https://jspm.org/. This is a big swing at Node and I'll be watching with maximum curiosity.
As an aside, the only thing preventing me from adopting Deno as my daily driver has been the lack of AWS Lambda support (requiring a lambda layer, slowing down cold starts). Would be great if they could leverage connections and advocate for native AWS support.
Here is just one from the community https://github.com/hayd/deno-lambda
Using your own image (i.e. without using the base AWS image with the layer) you'd get even worse cold start times.
Furthermore, will it support unifying transitive dependency versions to minimize bundle sizes?
It's also interesting to note how the tone changes after receiving funding. May just be a coincidence, but this looks like a reactionary jab back at Bun, which is barely out of the gate.
> It's also interesting to note how the tone changes after receiving funding.
seems on-topic to me.
I'm excited to see this, and I'm sure they'll work this out, but I sure hope I don't have to keep dependency versions in sync across all of my import statements. I don't have much love for package.json and node_modules, but it is nice to have one source-of-truth for all the dependencies of a project.
Deno's philosophy is to be aligned with web browser standards. I suspect import maps will be the path forward for most projects as it's what browsers will or are supporting today too.
Again looks like package.json with many manual steps in between (where to get those import maps is the first question).
Also, apparently non-composable etc. https://news.ycombinator.com/item?id=32469280
> Deno's philosophy is to be aligned with web browser standards.
And yet they want to be able to run npm packages...
You create the import map by figuring out what dependencies you want to use and crafting it and your code to reference them. You can do anything--you can import five different versions of the same library but under different names if you want. This was not something even possible with node and npm. You can make it as simple or as complex as your needs require. For most people they'll keep it simple and drop a few import urls directly in their source, never even needing to touch a tool or other file to worry about dependencies.
They're giving users what they want, an ability to run packages which haven't been ported to work with deno yet. The blog post explicitly makes this clear. This is not some grand conspiracy like so many replies seem to be trying to imply.
Ah yes. Truly the same amount of effort as package.json.
How exactly do you figure out all dependencies (including transitive ones) and "craft" that?
Edit TIL you can install as many versions under different names as you want with npm: https://medium.com/weekly-webtips/how-to-install-multiple-ve...
> This is not some grand conspiracy like so many replies seem to be trying to imply.
There's no conspiracy. All the awkward workarounds with tedious manual steps to recreate a package.json "dependencies" field are frankly laughable. Deno should bite the bullet and automate this since they already have the dependency resolution built in. No need to "craft" anything.
I LOVE TypeScript, and was so excited about Deno when I first saw it, but the lack of package.json "feature" is a huge turn off for me. I sometimes feel like I'm the only person that has no problem with package.json. I think it works great.
Apart from the fact that you don't create package.json by writing "export * from" by hand.
Anyway, besides that, I'm sure you can use different versions of a library within one codebase well enough; especially in NodeJS land, nowadays, differences between library versions are small.
Global find & replace isn't a real solution here. For example, if I'm reviewing someone else's code, it's harder for me to verify that they didn't miss an import.
In general, anything that replaces a single source-of-truth (packages.json) with a process (“run find-and-replace with this query”) will increase the amount of energy spent on being vigilant about that process at the expense of other things.
Sounds like you want a lock file. Deno supports that just fine.
I'm not rewriting my work so someone can have fun playing with VC money and abandon mature software.
Supporting npm is one big step forward, but without a package-lock.json that locks all child dependencies, it’d be a nightmare using the new compatibility in production
They are not clear if you need to have npm installed or not for this to work. I sincerely hope not - if it requires a locally-installed npm then they've shat the bed I think... that would be a major turn-off for me.
Can't say I've ever been concerned by the performance of Deno, but looking forward to seeing this playout anyway since faster is always nice to have.
You will still have both of those when deno resolves all dependencies and downloads them all.
It will just be hidden from you by "magic".
I get it, I understand it, but it is depressing to know that we're going to be dealing with horribly ridden legacy NPM code for the next 800 years.
A lot of the code on NPM is just bad. And now we're going to keep this legacy of just, bad JavaScript .
This is why I prefer dart and flutter for making mobile apps. Over react native.
For reference: https://news.ycombinator.com/item?id=32457587
Btw, I like where it's going but I find it quite sad that there is no official linux ARM64 support yet - which means I can't try to use it on AWS lambda for example.
Here's the problems:
1) Often, you have lots of code that only runs once (e.g. UI initialization), or runs rarely (click handlers). By the time the JIT has had an opportunity to observe this code and optimize it, it may be too late.
2) The dynamic nature of JS means that making your code JIT-friendly requires deep knowledge of what language features to avoid.
3) Performance isn't only driven by having the tightest possible machine code. Memory layout is often more important on modern processors. Indirection causes cache misses, and JS objects contain far more indirection than equivalent structs defined in C/C++/Rust. Further, in Javascript you can't really create cache-friendly collections of objects like vectors, BTreeMaps, etc.
1) Gathering information at runtime competes with the actual program for resources.
2) Optimizing at runtime competes with the actual program for resources.
3) Most importantly, JITed languages tend to be pointer-heavy and full of indirections, and there's not much a JIT compiler can do about that.
But in practice they take a long time to warm up, may or may not get faster, and in general don’t attempt total program optimisation. As some parts of the node standard lib is native code and can’t be optimised at runtime.
Can this really be right? What is the number for Nodejs?
But while "the vast majority" of npm packages don't require a gyp build step for native addons, some of those modules are pretty important, and I see no indication in the announcement that they're also going to be implementing the Node C API or the gyp build process.
Right now I'm working with a machine learning project, and XGBoost [0] is a direct Node.js extension [1] through the binary interface.
So this does bring things a step closer to being generally usable, but there are still significant roadblocks.
A WebAssembly build of XGBoost could work with Deno, but aside from some guy's unsupported side project/proof-of-concept for use in a browser, I'm not seeing an XGBoost WebAssembly build. And generally when deploying something like a machine learning model I'd rather use well-supported tools than to need to dive into the rabbit hole of maintaining my own.
And yes, XGBoost will likely eventually have that kind of support for Deno, but then the next bleeding-edge project will come along and only support Node.
Even assuming Deno eventually hits a tipping point in popularity where everyone wants to release Node _and_ Deno support in their bleeding-edge projects, there are still things that I miss from package.json that don't seem to exist in the Deno ecosystem.
Things like the "scripts" block: A nice centralized place to find all of the things that need to be done to a project, plus auto-run script entries that can trigger when a project is installed. And inheritable, overridable dependency maps (see the yarn "resolutions" block).
I'd love to jump into Deno, but I think there has been far too much "baby thrown out with the bathwater" to its design. It's the classic development problem of looking at a system and seeing a ton of complexity, but not really understanding that all of that complexity was there for a reason. Maybe when it re-evolves 80% of Node's and npm's features I'll be convinced to make the jump. I'm a huge TypeScript fan after all. But it still strikes me as a violation of "As simple as possible, but no simpler."
[0] XGBoost is a _very_ promising approach to machine learning, training models much faster and with much more accuracy than traditional approaches.
I still don't understand why they have this feature and still consider themselves security focused.
Do we know what will differentiate the 10-20% of packages that won’t work?
curious what kind of npm packages won't be supported in Deno
Not only I will be able to choose package manager, frontend flavour, backend flavour, build system, unit testing framework, now I will be able to choose which runtime to pick as well. I can't wait to see how productive my team will be when this ships.
I feel unease reading untestable claims (with no numbers) about future releases. I'm made more uneasy by the fact that this is the second bullet point of the main tl;dr. It makes me wonder, what are the organizational incentives driving such a publication?