Node.js 16 Available Now
nodejs.medium.com
nodejs.medium.com
import { setTimeout } from 'timers/promises';
await setTimeout(1000);
console.log("awake");
Deno basically does this with the Deno standard library, e.g. https://deno.land/std@0.93.0, and I agree that I think it's the right approach.
There is nothing "special" about wrapping setTimeout as a Promise, indeed pretty much everyone has done it at some point, so would be nice if there was a single, blessed standard version that I could just add as @node/standard in my package.json as long as I was on any supported Node version.
I prefer my batteries included. Also importing from "timers/promises" isn't a global namespace.
The most common stuff I like to be part of the standard lib, either node or javascript, but it works fairly well now I guess so no biggie.
That said, I'll probably spend more time searching the node docs for how to import/use it than just implementing it myself.
OTOH I use promisified timer in almost all of my web scraping scripts, to reduce the load on the server, so I'm glad I'll be able to drop this thing from my utility library.
/\*
\* A sleep function that returns a promise.
\*
\* @example
\* Sleep for 100ms
\* ```
\* await wait(100);
\* ```
\*/
export async function wait(ms: number) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}const wait = require('util').promisify(setTimeout)
[1] https://nodejs.org/dist/latest-v16.x/docs/api/util.html#util...
I prefer
async function foo() { return await new Promise(...) }
as opposed to
function foo() { return new Promise(...) }
They're the same thing for the most part but the latter I have to potentially dig deeper into the function to confirm it returns a promise compared to the former.
For documentation purposes, I recommend block-commenting the `async`:
const foo = /async/ () => new Promise(...);
> async function foo() { return "hello"; } foo().then(console.log); console.log("after")
after hello
It doesn't need to be that tricky.
My initial thoughts were “urgh I just want to have .js files in my projects” but I’m wondering if I’ll warm to them given that they make things a lot easier.
Are people eventually just all going to use .mjs files for everything?
I’m thinking about new projects rather than converting existing projects.
One benefit I can see to .mjs is that if all extensions are .mjs it’s clear what type of project it is without the need to open up package.json.
About new projects, I'd have to read more about this new extension, for some reason it feels like a temporary solution that'll just get merged to the normal js extension in a future release.
From what I’ve read though, there doesn’t appear to be much of a plan apart from the current solution.
My initial reaction is that I want to keep all my files with .js extension.
One thing that could potentially be problematic is for files that need to run in the browser and on the server.
Really curious to see where the community’s tea leaves settle. My guess is that people will eventually just start using the new extension.
Node is not for UIs, so they're definitely not using Node for that. It was confirmed that the UIs in the Dragon capsule (which, sorry for being pedantic, is not really the "rocket") ran on top of Chromium. It's possible SpaceX uses Node under the hood somewhere, but I don't believe that has been confirmed anywhere.
Also kind of interesting to note that it's only the Dragon capsule with humans that has controls, the capsule (and rocket) are both autonomous. The controls on the capsule are only there "just in case".
So if they’re running their UI on Electron, it could be on Node.
Also as a data service for connection pooling databases that don't support connection pooling with supplied drivers or server-side.
I said Node is dominant in web/APIs, but Fintech I don't expect to go for Node, if we're talking trading, but also for CRUD work.
DDEX is the industry standard for metadata communication (artist, track, licensing, credits, etc.).
One 20-track Xmas classical music compilation album could easily have a single 150MB XML file!
Companies started 10 years ago tend to have Java / Ruby backends, but companies started in the past 2-3 years will often have NodeJS/Typescript backends.
Were I to do it over again, I probably wouldn't choose node, but my problems with it haven't had to do with static typing or type errors--we don't use typescript. Where we've continually struggled is indeterminacy in process control and error handling and writing robust services in light of that.
* Async operations destroy stack information.
* It's very easy for someone to miss an error handler and end up with your process in an ugly state.
* Default error handling is "crash the process", no matter what else is going on.
* Lots of libraries that rely on buggy native code.
Using Node.js for (large scale) mission critical backends, mostly in JS, but (on the topic of typed stacks) more and more of it is becoming Typescript.
> Wonder if Node.js will ever get wider adoption like Java got.
I'm not sure if it it will ever go as wide but it does seem to be going that way.
IIRC, the performance of Node was ok but clearly worse than Go/Java/etc. Uber was using JS not TS back then, but the real issue (at least when I started) was lack of a defined interface for the API/mobile app communication. That was eventually addressed by adopting a forked version of Thrift.
It also used Node for a very core part of the app, and it was Node 0.10 to boot, but my understanding is that that's on the way to deprecation.
Microservices themselves are all in go or java these days.
I personally ran mission critical node services that interfaced with over 100,000 simultaneous compute nodes in AWS.
But this is (way) less than 1% of code and typically performance is not the problem with Node, even for our use case. ~Everything we build feels fast the first time. If on rare occasion it does not, it’s a matter of rearranging the building blocks, not swapping them out for something else entirely.
Having the same language across our entire stack has huge, enormous, gargantuan benefits that shouldn't be underestimated, especially for a small team. Being able to easily move between backend and frontend code bases has had a gigantic positive impact on team productivity. Couple that with auto-generating client and server-side typescript files from our GraphQL API schema definition has made our dev process pretty awesome.
Our app doesn't have a huge need for caching, but we use a mix of in-server-memory caching ( https://github.com/isaacs/node-lru-cache ) and Redis when we need a global cache.
Writing a migration tool is pretty trivially easy - all knex does is let you write an up migration and a down migration, and it keeps track of which migrations have been applied in a DB table. Main thing is that all of our migrations are each in an individual versioned file in our source repo.
We use postgres COMMENT functionality to apply comments to all of our table columns.
We have services that don't ORMs and I have to wonder if it's worth the cost.
Basically, in my opinion ORMs just make the easy stuff slightly easier, but they make the hard stuff much harder, and when you're stressed out trying to fix some critical production DB query all they do is get in the way.
Joking aside, in a dynamic language like Javascript, especially in modern coding style which is not OO anyway, you don't need an ORM.
Joking aside, I do write SQL statements (or use a query builder, which is not the same as an ORM).
I don't "turn the response into classes by hand in every app I write" however, because the responses are perfectly usable as they are (in a more functional style), and OO is not the best way to model records anyway.
I know Ruby and Python have ORMs, ActiveRecord, SQLAlchemy and so on. They're not really needed. Heck, I've read authors of ORMs saying you don't really need one...
I think the answer depends on the type and load/volume of app you're working with combined with the dynamics, size, and skill level of your team(s). I'm extremely comfortable writing, profiling, query planning, and debugging SQL queries. Others aren't, and therefore having an ORM to query data in the DB with the syntax of the language you're using in your projects makes way more sense, if nothing other in order to speed your team up.
2. project search and replace on $COLUMN_NAME
If your column name is a common keyword, variable name, etc. in your code base and it's difficult to find using project search, that's unfortunate, but we organize our backend code and tests in a logical enough way that it's never taken longer than an hour to create a PR to create a migration to rename a column and update all places in code that reference it.
IMO an hour for a change you have so little confidence about is not acceptable when the alternative allows you to do it in a second with full confidence.
Keep in mind I've installed and used an ORM in projects where the ORM is used only for migrations, but not used in application code and this is absolutely a fine reason to use one imo. But adopting an ORM for migration purposes and forcing or using in application code simply because it's installed isn't necessarily a good approach.
ORMs are to SQL what static types are to programming languages. The conversation we just had was me giving you one example of the benefit of static types, which happens to showcase a huge weakness in dynamic typing:
- How painful is it to rename a class attribute?
=> It's a long search & replace exercise which results in a less-than-certain outcome.
The obvious take from this isn't that "renaming class attributes is rare".
ORMs have very basic support of current SQL standards and database specific features.
This means that using an ORM reduces the power of the database choice you made.
Also things like arbitrary SQL support imply you have to manually creat return value typings.
Having to leave the nice ORM wrapper functions for arbitrary SQL means you lose all the ORM niceties like soft deleted or updated_at fields
Overall I see very little use for ORMs.
We have a TypeScript Node.js API in production, and we wrote Zapatos to be that something: https://jawj.github.io/zapatos/
same here, except Apollo GraphQL, we still use REST. it is indeed nirvana. wish the community finally settle down on a stack at least for a decade.. shifting the backend every few years doesn't do good for dev productivity.. (had to recently use Go due to peer pressure. while performance wasn't a concern it was purely due to the "feel" that Node.js is not good enough for serious backend work - lack of multi threading , potential future performance and scalability etc.)
That aspect of Node.js can actually be a very good thing for serious backend work, as it encourages process decoupling using API's. Decoupling ends up scaling better later on: you build a distributed system that can be scaled out horizontally across a fleet, rather than running up against vertical scaling limits of how many threads you can get running on the same local hardware.
Whenever I've seen applications that use worker threads for backend systems they end up regretting it and wished they had decoupled into a separate process that could have been scaled independently onto other hardware over the network. Spinning up new threads on the same machine is a temporary crutch that bites you later on in your growth.
I don’t think this is a particularly compelling argument for the lack of multi-threading. The worker thread pattern is fairly useful and there is no reason a well architected application couldn’t use both horizontal and vertical scaling. Independent scaling is only really useful if there is a large variation in the amount of work done by the workers per query. If it’s well bounded, then it might be easier to just horizontally scale the entire backend.
There are also other use cases for threads that don’t fit into the worker pattern. For example, background tasks that happen outside the serving path.
Would you mind enumerating them? I have an idea of what they are but curious about other perspectives.
My sense is that code sharing is not that common between a Typescript frontend and backend. You mostly need generated request/response data types but I don't think there's that much shared behavior because you can't import any of your backend-y logic (database, auth, external APIs) transitively into your frontend.
I think primary gain is what you've hinted at: 1 ecosystem and it's easier to onboard fullstack devs.
// shared code - implement the algorithm
export async function loadPageChunk(
args: LoadPageChunkArgs,
loadRecordValue: loadRecordValueFn
) {
// ...
}
// client code - use the algorithm, provide client-specific IO
// eg on Android we'd use Sqlite.
const records = await loadPageChunk(cursor, SqliteService.loadRecordValue)
// Server code - same, but use the server's data stores.
// Behind the scenes, these loaders batch, etc
const records = await loadPageChunk(cursor, useCache ? CacheService.loadRecordValue : PostgresService.loadRecordValue)
Previously I've done this with 2 implementations (JS on frontend and Java on backend), but then keeping the logic in-sync is a nightmare. With a single language you can just share the library.
You are correct, though, the biggest benefit I see is not sharing code, but making it trivially easy for a front-end dev to add a small piece of backend code that they need without needing a back-end dev to do it, and vice versa. It just makes the overall team much more productive because there is very little "waiting on the back/front-end" to do it.
Oftentimes we'll have either team write up the schema for a new endpoint using the GraphQL schema definition language, then from that we autogenerate the TypeScript types, then usually the front-end team creates a simple mock in the backend so that they can fully implement the UI, meanwhile the backend team works concurrently on the real implementation. This process allows for much more parallel productivity than if, when something is broken or needed from the other team, you just have to submit a ticket and wait.
Here was the meeting:
https://www.meetup.com/pdxnode/events/142646682/
* Ben Acker will share about some awesome drawings and tales of Nodejs within Walmart Labs.
Live medical data processing, large ETL pipelines, and coordination systems that set 100% uptime as a goal and any incident would have _very_ thorough RCA, retrospectives, and accountability reports.
While traditionally these were built with things like Erlang, Java Ecosystem tools (Camel, etc), the only time that using node had serious downsides was when integrating to particular languages ecosystems as a second class citizen.
Examples:
• Kafka. Until recently the node-rdkafka wrapper had quite a few bugs compared to the java client. We ended up writing our own internal client in typescript. Performance was worse, but we had easy scalability of our producer services and we were able to track down and resolve any issues quickly.
• z3. There hasn't been an official build for node yet. We had built a wrapper to use a specific fork of z3 that we had, which allowed identification and distribution of shared subsets of problems. This worked with an etcd-like streaming consumer where you would get live-pushed keyed-problem subsets and utilize those in a local in-memory cache to speed up solvers that overlapped.
• Distributed Actor-Like framework. Obviously Erlang, Akka, Akka.NET, etc are the prior art here. It didn't take too long to have our team analyze these and build out mimics in typescript.
• Wrappers for specific C++ statistics libraries. Some of our ETL pipelines would enrich data with a pass on certain identifiers/classifiers/aggregators. For a few of these we created js/ts wrappers, but it would have been nice if they existed.
• Standard Library. We took a microsoft-like approach and just created a standard library for ourselves. Tested and with lots of features, it meant that we rarely had to reach outside of our ecosystem for Collections, Encoders, Crypto, etc, and that they were all documented to our internal standards. If there was an issue, you had someone you could ask and get an answer within the hour.
Unfortunately all of the above is proprietary and we weren't allowed to release any of it.
We did retrospectives on technology choices and limitations once a year, to learn from decisions made going into the future. Each time when Node came up, the general consensus was "we could have done it in Java I suppose, but the Typescript/Node combination was much quicker to iterate on and we felt more confident in the solution after the fact".
Huh? Node is almost dominant for all kinds of API and web backends...
[0]: https://jobs.lever.co/substackinc/69f5ed72-9a51-404d-9db1-20...
That is, where "mission critical" means "critical to some particular business". Not guiding rockets or critical medical use, etc.
The dramatically larger ecosystem and community is great, but notably the static typing is far more powerful in practice. I really wouldn't pick Java to improve type safety nowadays.
Problems I personally have with it:
1. no exact object types in ts as in flow - means they have to be emulated by destructing, sad, but you can live with it/you have to be careful
2. transpile times - but recent experiments with swc for tranpilation and deferring typecheck to run concurrently while tests are kicked off after swc finishes look promising
3. type system could be a bit smarter in few places, but no blockers so far
I'd recommend but with caution - spectrum of developer's competency is closer to php (almost anybody can do it) than the one of languages like ocaml/haskell/rust and others (where entry bar is higher). Vet your dependencies, hire competent developers and it can work very well.
Some of libraries we're using:
- https://github.com/appliedblockchain/assert-combinators - light, runtime type assertions ("parse, don't validate" style to avoid illusion of type safety at io boundaries)
- https://github.com/appliedblockchain/tsql - functional, tagged template based combinators for sql generation
Typescript/Node.js/GraphQL back-end with React/Relay/Typescript on the front end.
https://github.com/coralproject/talk
It's pretty nice having the whole code base share types, syntax, structure, etc. It's served us well for many years!
Some of our clients include: The Washington Post, New York Times, Wired, USA Today, and Financial Times
I know it's possible and that some teams do it, but the story wasn't great with (much) earlier versions of node. Some teams just wrote their stuff in another language and just use a child process in node to call it, serializing everything as a string and DE serializing it in the other language. The problem with that though is that you suffer a pretty decent performance penalty serializing and deserializing, and though it still might be worth it, it's also not great since some teams actually just called similarly to how you'd call a shell script.
Is it much better than that now?
That said, in my own experience it was seldom worth it to rewrite something in C++ for performance sake. After rewriting some computationally heavy part as a native addon, I often ended up gaing only ~20% more perfomance at best when compared to properly optimized JS implementation, and even that was not guaranteed since V8 improved rapidly. That was not a good enough reason to keep a whole different tool chain around, so I'd end up going back to JS.
[1] https://medium.com/netscape/javascript-c-modern-ways-to-use-...
Since the bottleneck with native addons is usually data copying/marshalling, and we have direct access to WebAssembly memory from the JavaScript side, using WebAssembly on this "shared" memory might become the best approach for computationally heavy tasks. I wrote about it a bit here[2].
[1] https://github.com/zandaqo/iswasmfast
[2] https://medium.com/swlh/structurae-1-0-graphs-strings-and-we...
You have to scroll down all the way to "Node.js ES2021 Support" to start seeing features that work in Node 16 but not Node 14 (the current LTS version). Of course, it's possible to use Babel to bring those features into Node 14, but I enjoy leaving it out of my toolchain when possible.
Apple presenting some major hardware news today. Perfect time to release v16 :)
I'm curious, because I'm useless at RegEx.. But will this break current RexEx implementations ??
> ..We propose the adoption of an additional indices property on the array result (the substrings array) of the RegExpBuiltInExec abstract operation (and thus the result from RegExp.prototype.exec(), String.prototype.match, etc.).
> This property would itself be an indices array containing a pair of start and end indices for each captured substring. Any unmatched capture groups would be undefined, similar to their corresponding element in the substrings array. In addition, the indices array would itself have a groups property containing the start and end indices for each named capture group.
> NOTE: For performance reasons, indices will only be added to the result if the d flag is specified.
https://github.com/tc39/proposal-regexp-match-indices
From given example:
const re1 = /a+(?<Z>z)?/d;
const s1 = "xaaaz";
const m1 = re1.exec(s1);
// indices are relative to start of the input string:
m1.indices[0][0] === 1;
m1.indices[0][1] === 5;
s1.slice(...m1.indices[0]) === "aaaz";The correct way to do this is to write a native addon that uses libuv to create your own thread (`uv_thread_create`) that does all the interaction with the GUI APIs. You then manage your own message queue to pass messages between your GUI thread and the V8 thread (using `uv_async_send` to invoke a C function that dispatches events from the queue into JS-land).
"The right tool for the right job" and all that jazz.
Observables are neither a part of the language nor a part of the Node api. I suppose that was what the parent's criterion.
Also please be sensitive and don't use "kill" in new code. ;) We already have to deal with "abort".
Yes, a way to terminate down-stream. It turns out to be a pretty messy problem. I didn't know AbortSignal was in Node now. It's been a while since I revisited this issue. I should read more.
> Also please be sensitive and don't use "kill" in new code. ;) We already have to deal with "abort".
Point taken!
I invested quite some time on node.js and eventually bailed out and now am using other alternatives. It did not work out as not all applications need those async-logics which made code unnecessarily difficult.
Nowadays for me, nodejs along with npm/yarn are just more of a frontend tool, which are still very useful and essential.
Is it a good idea to make hype a relevant factor in choosing web or app's architecture?
Yes! Hype = active community and support, interested developers, possibly novel solutions to problems, etc...
If something has hype it's worth investigating, absolutely. But it should also be treated with a great deal of scepticism; there's often a lot of money in it for businesses and individuals moving you onto the latest shiny tech.