Hard-won lessons: Five years with Node.js
blog.scottnonnenberg.com
blog.scottnonnenberg.com
JavaScript has a lighter weight feel than Java and the jvm stack. Single threaded with a top notch event reactor. You can write just a few lines in node and have a web server serving some json.
There are a ton of JavaScript coders. There is something to be said for having a common technology throughout your stack and you ui is almost certainly in js.
It's easy to get something simple going quickly, there are tons of libraries and you're going to need to write go, Java or .net to out perform it.
The big downsides, imho, v8 is designed for client side stuff. Both it and dart vm are limited to relatively small memory footprints (think 1GB) which just isn't good for some workloads and problems. Likewise a lot of server tasks can make good use of real threads, you're out of luck with node, you can have child processes and send messages but if they have to share big amounts of data the marshaling becomes a bit of a bottleneck. If you work is database crud, or light weight, or horizontally scalable, or able to be modeled as streams then node isn't that bad. Js feels kind of limited in how certain things are modeled, there isn't much data hiding or abstraction; the extreme simplicity almost creates more complexity for some things. There are lots of different opinions on basic code behavior, some node modules do work on import, some need explicit initialization, some alter prototypes and it's a concept that seems to be lost on many js devs; I've burnt hours debugging something because someone reordered the imports and it screwed init logic. And you'll completely screw yourself if you don't stay up to date with your depends, things change fast and sometimes they change a lot. Both node and dart vm seem really well equipped for tooling type things that are usually done in bash, Python or perl but that doesn't seem to be as popular.
It's definitely not for everything but it's worth a look. It is really popular.
Ain't nobody is going to compare performance against Erlang for numerical number crunch.
It was never built for that. It's not even a fair comparison.
It's like comparing a race car against a hybrid prius and saying the race car suck at mpg. They're built for different purposes.
Ilk implies as if Erlang is some crap flavor flav tech. It is not.
Erlang was made for concurrency.
NodeJS was made to solve for concurrency too.
And Erlang beats NodeJS hands down in concurrency.
The actor model is a much better model to do concurrency and preemptive scheduling is amazing. It's easier to mentally think about actor and the code isolation in each processes makes it much more cleaner, object oriented, quoted by Alan Kay himself.
NodeJS chose some existing language to do concurrency where as Erlang was created from the ground up to do concurrency. Hence the reason why it's VM is so amazing on top of the fact that concurrency constructs are first class and primitives.
No, but it means I can write relatively computationally intensive server side code in JavaScript rather than having to switch to C and use an FFI. It's always nice to have decent performance, even if it's irrelevant most of the time for web programming.
Erlang is great but its ecosystem is tiny compared to Node's, and as a programming language it's at least as quirky as JavaScript.
I disagree with this; while the syntax is unusual for many, its semantics are extremely simple and clear. I don't think it has anything quite like:
* JavaScript having `null` and `undefined`. * JavaScript lacking have proper integers. * JavaScript strings being UTF-16, so things like "".length don't work. * Scoping of 'this'. * The unmanageable automatic typecast rules * Semicolon insertion. * JavaScript `with (x) {` (maybe a cheap shot since it's so uncommon, but we are talking about language design).
Main quirks I can think of for Erlang are when people trip up over "strings", <<"binaries">>, and iolists. But I'd love to hear where others run into proper semantic quirks.
And even records are hardly a quirk like the ones I listed for JavaScript: as long as you remember that they're just syntax around tuples, you're hardly ever surprised by what they do. You'll never get a runtime behavior you didn't expect.
This hasn't been true for a while now.
node --max-old-space-size=8192 server.js
See also http://prestonparry.com/articles/IncreaseNodeJSMemorySize/
I've heard that a lot but advocacy always seems to be based on artificial benchmarks. Most server apps are I/O bound and the ones which aren't tend to bottleneck in native code – i.e. a gzip implementation would run faster in V8 than python, ruby, etc. but since they're just calling a C library, it's a question of whether V8 is faster than GCC/Clang.
One area where this is more interesting is async support, but the argument then is that it's easier to write async code in JS than other languages and reasonable people disagree on that point.
I'm not saying that V8 isn't an impressive bit of work, just that most of the advocacy I've seen has reminded me of the JVM hype cycle before most people learned that a JIT mattered a lot less than their architectural choices.
"Tons of modules" is also a misnomer, because a much larger than normal portion of these are low-quality, small, and/or poorly maintained. Just the way it goes in the age of GitHub, where everyone wants a badge of honor in their repo list.
Seconded.
I'm working for a company building an Ionic app, and they hired two junior programmers with "angular experience" (both of whom had built angular apps before) to be the developers on the project.
I'd used KnockoutJS but never ES6, Angular (1 or 2), TypeScript, written a mobile app, plus the RxJS concept was new to me, as was thinking asynchronously (I told them I wasn't a good fit for the project).
Yet while the developers have struggled to do their work and the schedule has slipped, I've been able to pick up the concepts, write half the app and solve the juniors' problems despite my unfamiliarity with it all.
Glad to see nearly 20 years' experience programming makes a difference, even though none of it was with today's "hot" technologies.
and... the 'impostor' thing... quite true. easier for these people to slip in to larger companies, I think, but it's a problem all around.
I would expect someone with 20+ years of experience to be able to help juniors troubleshoot in any language/stack, though, whether they expect it or not. :)
I can with one, and have cracked his management style too, so he's gone from being unproductive and the butt of office gossip to meeting schedules.
The other is new to this country, doesn't speak much English (e.g. not enough to ask questions) and understands little English too. At a loss for how to proceed...
This, was actually one of the things I liked most about Node.js - npm allows a developer to specify all dependencies (`pacakage.json`), versioned, similarly to maven but not as clunky. I was able to get up to speed project very quickly and easily using just `npm install`. I had one or two versioning issues to iron out, but the advice it provides about what is broken and where was very helpful.
It'd be interesting to hear a fair, modern comparison between npm, ruby-gems, LuaRocks, pip/PyPI/cheeseshop, and the other big ones I'm sure I'm forgetting.
The JavaScript community has had a lot of competitors on this front, particularly for frontend asset management, but it does finally seem to stabilizing a little bit. It took forever.
I mentioned NPM in particular in the context of this being a node.js discussion. I've seen a few comparable tools in my time but IMHO NPM may be one of the cleanest, considering this as a feature of node.js - it does add to its attractiveness as a development environment.
Yeah it took the JS community ages to sort this out, but in NPM there seems to be an answer.
Not to nitpick details, but V8 is a runtime, not a compiler.
A compiler checks your code and tells you if it contains errors before you run it. V8 does not.
So it's a compiler which incorporates a compiler? Compilerception much?
Seriously though: Can I tell V8 to compile my JS and tell me any errors found before I deploy it to production and runtime? Yes or no?
Just because V8 internally JIT-compiles the JS-code to something eventually executable which the machine can run doesn't change the fact that it's interpreter and execution-engine (also known as a "runtime") for Javascript code.
Just like Python and all those other platforms you listed as not being "compilers" (in which case you're absolutely right).
No, what I meant is that the concept of a JIT compiler incorporates both a runtime and a compiler.
> Seriously though: Can I tell V8 to compile my JS and tell me any errors found before I deploy it to production and runtime? Yes or no?
That's not the job of a compiler. It's a feature of compilation, but a compiler does not by definition have to compile your code before deployment. What would you be compiling the JS to? If you're wanting to check your syntax, your IDE should be able to do that, or you can use a compile-to-JS language like TypeScript. Yes, it's dumb I know - that's one of the many reasons I use TypeScript.
> Just because V8 internally JIT-compiles the JS-code to something eventually executable which the machine can run doesn't change the fact that it's interpreter and execution-engine (also known as a "runtime") for Javascript code.
You seem to be hung up on the idea of a compiler as something that checks your code is correct. It isn't - that functionality is incidental to the compiler's job, because it can't translate invalid syntax into whatever destination language it compiles to.
> Just like Python and all those other platforms you listed as not being "compilers" (in which case you're absolutely right).
I did no such thing. That was someone else. Any language which internally uses a bytecode representation is a JIT compiler, including Python, Java, and PHP. You could argue I'm being nitpicky, given that pretty much all performant interpreted languages use a bytecode representation - it's true in a sense, but I'm just trying to highlight that you seem to have a misunderstanding of what a compiler's job is.
And for the record, AOT languages also have "runtimes" provided by a platform. If someone combined gcc and glibc, would the combined product no longer be considered a "compiler"?
- To get around the 1GB cap use Buffers, they are outside the v8 heap.
Here's the major caveat, out of, say 100+ PoC/prototype projects I've ever done, I've taken two to 'production'. I put production in quotes because they were actually internal services as part of our build and delivery pipeline. It's a major pain in the butt to monitor and keep Node.js running production-like in even remotely similar ways you would with Java, .NET, Python, or even PHP for that matter. I don't expect such drastically different technologies to be exactly alike for production workloads, but there's usually analogous ways to do X, Y, or Z.
You know where Node has been awesome? You land a project with a real tight deadline and its more of a fire and forget project (think interactive 'experience' at a major gaming conference for instance). It needs to run pretty well for a short period of time with a narrow scope of functionality, but some of that functionality is non-trivial. I absolutely love Node for that.
It clearly tells you in the README it's not for production, but I have this recurrent discussion with developers who tell me they have only ever run Node.js this way.
What that tells me is there is quite a trend a towards exactly what you describe, where a lot of what gets written never goes into production.
No, it is worse than that I would have thought. The cynic in me (who is more often right then wrong) suggests that is means there is a lot of "proof of concept" level of code out there in production environments potentially running important things.
Why?
Writing Java is really very fast these days, especially with IDE's.
I can get a small web site serving JSON REST requests and memcache up in ten minutes with a simple Maven/Gradle build and full IDE integration.
Between having a REPL, near instaneous stop/relaunch times, and doing in 8 lines of Node.js what takes me 70 lines using CompletableFutures, I'm still going to reach for Node to prove the idea.
Now, when it takes hold and looks like it might stick around, it immediately gets turned into a Java project. Or, if the event we were building for is over, it basically just gets deleted- which pretty much says it all in my opinion.
It's not a toy language, there's some real usefulness to it, but just like anything else, there's a time and place. Currently, that place is NOT production.
It was really fun work, because even the 'simple' stuff we built had to address some of the big traffic issues you'd normally have at a much larger scale and the work was usually very high concept or cutting edge. It would've been a nightmare to maintain for longer than a week though.
I prefer python in general but for zero-to-simple-web-endpoint I may pick Node.
Sure, but that wasn't the argument; GP talked specifically about async and websockets, which are areas where Java is particularly weak, but areas that I was surprised would be required for a PoC at all.
Asking, because I do most of my development in Clojure and find it much nicer than Node both for POC and production. There's also self-hosted Clojurescript now that runs without a dependency on JVM.
I've only just played around with Clojure and I've written a middling amount of Groovy, but the vast majority of it involved Jenkins build pipelines, so I don't really count it.
Java guys always say how they have really developed tooling and look down on other languages. The problem is those nice tools have become a hindrance as well. You can't develop in Java/Kotlin if you don't use those huge complicated tools.
I can't speak for all, but at least for me it prevents me from ever using Java tech. I can develop just fine using a lightweight editor in any of the languages I use. I don't need huge tools or complicated build systems. Because my languages have nice tooling that I can write manually and fully understand.
I tried using Kotlin and step one is always install Intellij.
Oh please. Let's be real, Maven is not even remotely more complicated than the typical package.json, webpack.config.json, .babelrc triple you need for any useful NodeJS project (though you can put the .babelrc into the package.json, I've heard that's the way to do it at the moment). I was so taken back that nowadays JS needs a build step too. But it's a "transpiler", not a "compiler" - cause that's something completely different.
> I tried using Kotlin and step one is always install Intellij.
So? Atom and Visual Studio Code did need to be installed too on my machine. This mentality reminds me of someone using an axe head as a hammer "I'd have to pull the hammer out of the toolbox" (install it) - that's too complicated, I prefer light-weight tools!"
https://maven.apache.org/guides/getting-started/maven-in-fiv...
Maven quick start first command:
> mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
Then it generates stupidly deep Java structure.
and next you have to write a verbose xml. They even say:
> The POM is huge and can be daunting in its complexity
All of this stuff is verbose and built to be generated by IDEs. It's not clear, it's not concise.
You misunderstand my point completely. I was trying to say the Java ecosystem is complicated and the only way it works is by using large IDEs. Saying how awesome Java IDEs are is disingenuous because Java is unusable without them.
The thing with Java is the amount of boilerplate, but the thing with Java is that all of that is ridiculously well documented or demonstrated ad nauseum. Are you building a jar with Maven? Your folder structure looks like this. War? It looks like that. Before Yeoman or any other generators even existed Maven has had archetypes that do the exact same thing.
If you compare Maven, heck, even Gradle to Webpack + Babel for even fairly trivial projects, there's no way that it's more complicated. The tools for Java are truly portable, I can't even recall the hell we used to have to go through when we had to run gulp builds on Windows because developers of various libraries had hard coded *nix style paths, etc.
Most importantly, Maven and Java really aren't a moving target. 9 months from now I'm not going to have to overhaul my entire build process because a wide swath of my plugins have completely changed directions. Stability is how you wind up with the time to make things run smoother in production. It's how you wind up with repeatable patterns and better instrumentation. You get a chance to take a breath and formulate an honest to goodness process vs. literally sprinting as fast as you can to simply keep up with the changes.
This is based on what? It is totally untrue from my experience. Quite the opposite.
IDEA and Eclipse compile your code as you write it. And with Hotswap, you don't even need to redeploy anything. They also all offer REPL's that are much more sophisticated than anything Javascript has to offer.
I think you haven't used a JVM language and its ecosystem in a very long time.
Except when you have to restart the JVM because you added a class or a whole bunch of other things that it can't hotswap in the new code.
Dropwizard is a great example of this. It's an opinionated (convention over configuration) Java framework for building REST web services. You can get a 'Hello World' web service up and running in literally 10 minutes following this tutorial http://www.dropwizard.io/1.0.0/docs/getting-started.html
Why is it faster to write code in a dynamically typed language?
Most modern statically typed languages have type inference so you don't even need to write type annotations for the most part, assuming these extra characters were the reason.
So what's that mysterious reason that makes writing dynamically typed language faster?
I literally don't know the answers to these questions so the same task would take me hours, possibly days. Getting from zero to JSON api in node.js is fast by comparison.
Note I am not arguing whether or not it's better in the long run to be the guy that went from zero to node.js or zero to Java, that's a separate discussion.
You have similar concerns with node (What's npm? etc...) These are peripheral and necessary knowledge on both platforms.
> Getting from zero to JSON api in node.js is fast by comparison
It's only fast because you already have the knowledge.
Two equally proficient developers in Java and node will set up and get a server up and running in about the same amount of time.
The idea that it's a faster to achieve in a dynamically typed language is a myth.
The code in question looks like:
iter.next().ok_or(ErrorKind::NotEnoughArguments.into())?.parse().chain_err(|| "cannot parse argument")?
the solution is of course to do
iter.next().ok_or::<Error>(ErrorKind::NotEnoughArguments.into())?.parse().chain_err(|| "cannot parse argument")?
in a dynamic language, you wouldn't have to worry about this; in fact, you wouldn't need to call .into() either - there's just less typing and compiler errors
Less ceremony.
Do you read JSON from ajax post bodies? Do you access fields in that JSON without sanity checking it? Try passing JSON that's missing one of those critical fields (as, say, a griefer might do). What happened to your app?
- Logging
- Metrics/Instrumentation/Tracing (memory usage, transactional performance monitoring, etc)
- Surviving exceptions
All of these are either very immature in Node or simply so far behind they may as well not exist. By nature of running in an application server, you do get a lot of benefits right off the bat for all of those, but even if you're running an executable jar instead of deploying to Tomcat stray exceptions will in almost no cases cause your application to simply die for all requests.
Also, say what you want about the verbose try/catch Exception model, but at least you can count on it. Am I getting an Error in this callback or is an exception going to get thrown? Oh, they leaked an exception when I was prepared to handle an Error and now my entire server is down. Does your process automatically restart? Not likely unless you spent a lot of time hand crafting your service to be daemonized, which is generally FAR beyond the scope of most developers I've met in any language (which also upsets me).
Also I thought something like PM2 (https://github.com/Unitech/pm2) or Forever (https://github.com/foreverjs/forever) was pretty much standard for production Node apps?
We use PM2 in production for web apps, long running processes (console apps with rethinkdb change feeds) and cron to spin up command line apps for data transformations, SaaS integrations, bots and more.
Most of us use VS Code now (I still use Vim)...so yes, having done enterprise work with a full blown IDE and tooling can be very helpful as the Node debugging and profiling experience is not great (but getting a ton better).
In the past we had a ton of C#/.net in production, then Ruby and now JS for the indefinite future. We build things that end up in production for 10 years or more, so having that talent pool to pull from in the future will be great.
* http://jdebp.eu./FGA/unix-daemon-design-mistakes-to-avoid.ht...
A crash certainly draws attention to a problem much better than an error in a log that no one usually looks at.
Yes it's a pain, but you deserve it for writing buggy code and then not catching it in testing (that's what I keep saying to myself, anyway). ;)
Exception handling in C# is amazingly good, you can't run C# production level apps.
It suffers none of the problems that are being discussed above. All you're doing is confusing the discussion with a view point I haven't heard in a decade.
What language doesn't do that?
If a TypeScript ORM with automatic schema migrations and decent expressiveness comes around, I'd be willing to give NodeJS another shot.
But so far, nothing comes even close to the productivity of Django + ORM + Django Rest Framework. For your typical CRUD app, this will walk circles around any current JS solution. Even if the lack of static typing eventually becomes a bit of a pain.
Relationships must be done manually, including joins, traversing, and many to many.
Sometimes doing:
w = Widget.objects.get(pk=2) w.category.author.name
Is just all you need. And Django makes that trivial.
But if you need a regular web site, not a real time app with 100000 users, then nothing BEAT those techs.
They are mature, stable, productive, and incredibly flexible.
If you use anything else and you need something that is not "the typical hello world", you will have to write it by hand. With DRF ? You google it, you find a module and call it a day.
I'm not really up on the state of DotNet Core or whatever they are calling it this week, but when that is shaken out, I'll be interested so see what kind of inroads it makes on Java and Python in the Unixy server world.
Now, in terms of bindings, bindings to libuv are also fast. So node is not so much of a problem in itself.
However the problem is how the language is being used:
To say you can write a serious library or server code because you know JavaScript, is the same as saying you can write a paper in cardiology research because you know English.
You will find a lot of people that completely disregard the domain knowledge required to write safe, workable code. While this happens in many languages, JavaScript is the lingua franca for those kind of people.
Because of this, be very careful of the libraries you use. You will most likely have to read the code to ensure there is some thought behind it.
Then if you are hiring node developers, prepare to receive an avalanche of resumes from people without a degree expecting to learn on the job at your expense while receiving a 6 figure salary.
This makes no sense.
They're all dynamically typed languages (avoiding the term "scripting languages" since it's pretty vague).
The only reason for the difference in speeds between these languages is due to the quality of their interpreters, not the languages themselves.
That's not entirely true; there've been fairly good write-ups on particular elements of language semantics for which support inhibits VM performance; for instance, there were particularly good write-ups a while back by the JRuby team (Charles Nutter, specifically, IIRC) on Ruby semantics that are problematic in that respect.
Obviously, that doesn't mean that the level of investment by major firms in the interpreters doesn't have an impact, and Node is definitely riding high on Google's investment in making V8 speedy for Chrome. But language differences do matter.
Repeating myself just for you: I was referring specifically to benchmarking language aspects excluding bindings to native code such as files, networking, processes, etc.
e.g: https://benchmarksgame.alioth.debian.org/u64q/compare.php?la...
Then sure, JavaScript has many implementations, so does Ruby and Python. I was referring to v8 (used by node), CPython and Ruby MRI in particular.
A functional prototype is software that implements its functional requirements (e.g: features) but not its non-functional requirements (maintainability, stability, scalability, performance, configuration, monitoring, security, etc.)
What is "a lot of baloney" is people selling functional prototypes for the price of a finished application, or not knowing the difference.
Node.js matched the developers' skill sets and gave us the ability to quickly write a server that could easily handle several thousand concurrent requests from the clients. On the client-side, Node.js gave us cross-platform support for nearly free (as well as being within our skill sets).
Looking back, I do not think the project would have gotten off the ground in any other platform in that specific organizational environment. The server on Linux was particularly well-suited as it was architected to scale horizontally but still had room to scale vertically.
On the client side, while we were able to get the Node.js service out pretty quickly, it eventually had performance and integration issues, particularly on Windows. It was super frustrating having to shell out to external processes or write native C++ modules to get access to some needed Windows APIs. In hindsight, I would have tried to rewrite the Windows client sooner in C++ or C#. On the plus side, it was a huge advantage that the entire Node runtime fit in a single executable (no dependencies on a huge JRE or other runtime).
Overall, I would recommend it as an option to consider if most of the following is true:
* Need to get to market quickly * Application is I/O intensive rather than CPU intensive * Dev team already knows JavaScript * Target environment is not Windows
Personally, I decided to move away from Node.js and these days if I was faced with the same problem I would lean toward a JVM language or Elixir.
That one is still my main problem. I know, I know, Windows is evil and so on, but our customers use it. We are remarkably free in choosing our tools or whatever, but Windows is usually a constant (Intranet environments) and every day I crash against the absolute "Windows is a second class platform" problem of NPM modules. For an environment which is supposedly platform-agnostic the amount of "only on Unix" is astonishing.
Maybe that you use a lot of modules with c dependencies?
I'd still go reach for the JVM and languages like Java or Kotlin if I was going to be hammering on a piece of code on an everyday basis for two or three years. But in 2017 (and probably 2016, but 2017 was when I got to it), it just became good enough to get it out the door with.
(Regarding TypeScript: I feel like it's an incomplete thing and I'm happy that it exists, but for the stuff I use ES6 for I can hold the entirety of it in my head and I'm not worried about typing. For things that matter to me...well, I wouldn't be using ES6 in the first place.)
Between ES6 fixing [0] many of the day-to-day warts that made JavaScript a joke language barely suitable for scripting DOM elements and V8's complete obliteration of other dynamic VMs in speed, I am being forced to accept that JavaScript is maturing into a serious platform, and starting to gradually introduce Node.js into my workflow.
Just today I was discussing how hard it is to pull the stick out of my butt regarding JavaScript, but I think that the evidence is in favor of it. I'm having a strong impulse to try to use Dart instead just so I don't feel as dirty anymore.
On top of that, NodeJS is very, very fast. Compared to Python or Ruby, straight computing is significantly faster, but that's not even where the meaningful gains come from. Because NodeJS is non-blocking, the core is IMMEDIATELY available to serve the next request once an asynchronous call is made (read from DB/Cache/HTTP call). Because of this, each core can serve thousands of simultaneous connections.
Because JS is single threaded, in order to take advantage of multi core hardware, you just run the cluster module to effectively launch a concurrent instance of your application for each core.
As far as the language itself, once I understood it, I loved it. At first, I thought it was terrible, but when I went to ES6 syntax, and fully groked how to write asynchronous code, it rocketed to my favorite language.
I started with Java, moved to Python, and although it took me a while to come around to JS, I would absolutely never go back.
FYI, anyone who has worked with Elixir or Erlang views these sort of statements about Node as completely ridiculous. The only languages right now that are making a serious effort to bring real concurrency to modern programming are Go and Elixir. Node with its single threadedness and global heap doesn't come close, since those are fundamental problems with JS. The papering over the defienct language concepts with attempts like promises don't address these issues at all.
With a global heap, you also become memory bound far sooner than you expect as well.
Spring Boot microservices that don't do much shouldn't need more than 32MB heap -- see https://spring.io/blog/2015/12/10/spring-boot-memory-perform... for details.
Of the top of my head, concurrent asynchronous tasks are simpler in Pony, Go, Erlang, Elixir, Haskell, and Oz than in JS (and these generally also handle parallelism beyond "run another instance of your app", too); and lots of other languages have concurrency constructs equivalent to JS promises (and, often, also async/await), so JS is at best no easier than that larger group.
"any other language" is a bold claim. I heard Haskell gets concurrency more elegantly, or Erlang.
What languages are you comparing against? It seems like asynchronous code is more difficult to write in NodeJS than in C#, Go, or Rust, and about the same as Java. I can write code in Java in a style similar to promises using ListenableFuture (Google Guava) or CompletableFuture (Java 8) [0].
I should also clarify that I don't think promises are the optimal approach to asynchronous code. I think you can do better, and the primitive "await" is an example of how. Bob Nystrom (munificient) wrote an article called "What Color is your Function?" [1] that does a good job describing the difficulties of promised-based async, and how await improves on it, and how Go does even better.
(From what I've read, JavaScript ES8 is supposed to include async/await, and NodeJS seems to have support for them. At that point you'll be able to program in standard JavaScript and use await, which will bring NodeJS up to parity with the first-tier languages.)
> On top of that, NodeJS is very, very fast. Compared to Python or Ruby,
Perhaps that's true compared to Python and Ruby, but not when compared to other high-performance runtimes like Java. The following three paragraphs are from a comment I previously wrote on this topic [2].
Folks might also be interested in the TechEmpower web framework benchmark [3]. The top Java entry ("rapidoid") is #2 on the benchmark and achieves 99.9% of the performance of the #1 entry (ulib in C++). These frameworks both achieve about 6.9 million requests per second. The top Java Netty server (widely deployed async/NIO server) is about 50% of that, while the Java Jetty server, which is regular synchronous socket code, clocks in at 10% of the best or 710,000 R/s.
NodeJS manages 320,000 R/s which is 4.6% of the best. In other words, performance-focused async Java achieves 20x, regular asynchronous Java achieves 10x, and boring old synchronous Jetty is still 2x better than NodeJS throughput. NodeJS does a pretty good job given that it's interpreted/JITed while Java is compiled, though Lua with an Nginx frontend can manage about 4x more. NodeJS is handicapped by having to orchestrate everything from its single thread.
I agree that asynchronous execution can provide an advantage, but it's not the only factor to consider while evaluating performance. If throughput is someone's goal, then NodeJS is not the best platform due to its concurrency and interpreter handicap. If you value performance then you'll chose another platform that offers magnitude better requests-per-second throughput, such as C++, Java, Rust, or Go according to [3] and [4]. But I'll grant it has better IO performance than many other dynamically typed languages.
Asynchronous execution also does not necessarily require explicit async programming. Other languages have as good or better async support -- for example, see C#'s `await` keyword. [1] explores async in JavaScript, await in C#, as well as Go, and makes the case that Go handles async most elegantly of those options. Java has Quasar, which allows you to write regular code that runs as fibers [5]. The code is completely normal blocking code, but the Quasar runtime handles running it asynchronously with high concurrency and M:N threading (similar to Go). Plus these fibers can interoperate with regular threads. Pretty gnarly stuff (but requires bytecode tampering). If async is your preference over Quasar's sync, then Akka might be up your alley instead [6].
Lastly, high performance servers don't necessarily require asynchronous code at the application layer; performance sensitive logic can often live in the library or framework e.g. Netty, reverse proxy e.g. Nginx, or in the OS's context switching.
> Because NodeJS is non-blocking, the core is IMMEDIATELY available to serve the next request once an asynchronous call is made (read from DB/Cache/HTTP call). Because of this, each core can serve thousands of simultaneous connections.
This also describes regular synchronous code with many threads. Once your application makes a blocking call, the operating system context switches to another thread (process). This is how boring old blocking Java code running on Jetty outperforms NodeJS in request-per-second benchmarks [3]. And you could take advantage of this in NodeJS too if it weren't for the fact that JavaScript execution occurs in a single thread!
For most typical applications, overall performance is more influenced by the efficiency of the runtime than it is by the concurrency/async strategy.
[0] https://github.com/google/guava/wiki/ListenableFutureExplain... and https://docs.oracle.com/javase/8/docs/api/java/util/concurre...
[1] http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...
[2] https://news.ycombinator.com/item?id=12341886
[3] https://www.techempower.com/benchmarks/#section=data-r11&hw=...
[4] https://news.ycombinator.com/item?id=12268988
[5] http://docs.paralleluniverse.co/quasar/
[6] http://akka.io/
secondly, what do non-plain-text benchmarks give us?
There's a lot a capable developer can accomplish with Javascript, with it's first order functions, collection operations, generators, promises and now, async /await. Granted JS is not idiot-proof, but I'd any day use a powerful language than a highly restricting one (think Java and Go).
I think Node is absolutely fine. But I see why someone who values robustness and error-reduction as primary goals or someone that has a firm grasp on other asynchronous models might not see many advantages to Node. Just as some people value stability higher than progress, neither is necessarily evil, just beliefs.
People have different priorities and Node delivers on some and is not geared towards others.
But I don't find Nodes async, single thread model very interesting or revolutionary. It is very pragmatic and definitely useful but I found the Erlang VM and Elixir a much more compelling solution for example.
The primary reason anyone would use Javscript over say Erlang would be the ubiquity of the language and the availability of libraries or strong familiarity with the language. But my point is Javascript provides you the tools to write functional code that is robust (if you are disciplined) and generators + promises (and async/await) makes the single threaded async model painless. I certainly did not claim that it was a silver bullet or a revolutionary model of computation.
.then( you realize it's just a callback..
Why use it:
- Server and client side in the same language
- Simple enough for you to feel free
- Enough hammers not to forge your own every time
- Fast enough
- Good enough to structure larger programs
- Browser is the only cross-platform GUI toolkit
- Web applications
- Network clients and servers
- Computation (CPU) heavy programs
- Very high loads
- Shell scripting
- Raw packet requirementAnd what I like about NPM is that there are a LOT of single file modules with a narrow, well defined scope. There are exceptions to this, like Ember or Angular, where they try to package up everything you might need. But there are also smaller frameworks like Vue.js and even smaller packages still, which do a single aspect of data binding or rendering or whatever.
And it's those granular little packages where I think the NPM community ends up exploring just about every possible way to do get something done, and often comes up with a new idea that can teach me something.
The organizational and software complexity behind an interface like Rails or iOS or Ember I think discourages app developers from questioning the interfaces they are using, and participating in shaping the landscape under them. I think that's a shame. The NPM community does it better.
1. most devs know JS (so it's easier to hire & onboard new devs)
2. while everyone claims node's dependencies are hell, I love the choice available (although over time, it's tiring).
3. The community is generally awesome (helps during meetings, online etc.)
4. Developer Exp: With TS, Tslint etc, devx is not too bad IMO (and there's no dearth of opinionated articles around on the correct way to do things)
5. Finally - we can quickly churn out prototypes for features to production (but I guess that's more experience than the tools?).
6. In general, we'd be using JS on the frontend anyway (React), so it just made sense to use one language everywhere.
7. In general, most tools have JS SDKs and JS language support is always around (an exception I found was Tensorflow) - as an example, AWS Lambda supports Java, Python and JS.
So... at one place I work, we use nodejs. There are so many modules added using so many files, that we ended up not being able to use our packaged application - extracting the package into a holding dir before moving into place meant that we ran out of inodes on the filesystem (stock 8GB ubuntu ext4 cloud image)! The filesystem could not hold two copies of our app, basically, due to inode exhaustion. The easy workaround was "increase the size of the disk", which seems silly given it's 80% free space.
This has apparently been fixed in recent times with flatter npm structures, but watching the nodejs community run into every single packaging roadblock that has been solved before is just disheartening.
Deeply nesting packages was specifically designed to fix the issue that other packaging systems have: two dependencies needing different versions of a third dependency.
Flat-where-possible structures (ie, npm v3 and up) were designed to improve this. They're excellent and npm is the only place I've seen them.
We also have had issues with node_modules takes minutes to download to the point that we basically had to cache it in our CI builds - which in itself was a really bad idea (thankfully Yarn seems to fix a lot of those issues)
That's when those download/install steps don't just fail for some unrecorded reason.
It's nice to be able to do server side rendering (i.e. run React there). Also you can run all the build tools in your application server during development which can be super convenient - Probably the most compelling example I've seen of this is Next.js.
InterSystems Caché is easily the worst tool I've ever used. A typo in your code often caused the compiler to bork your entire runtime. Not kidding.
Node is almost that bad. Stately as nicely as possible: it's syntactic vinegar for a type hostile flow of control obfuscation framework.
If you have some irrational need to use libuv (callback hell vs actors, CSP, multithreaded NIO, or AIO thread per connection), you're much better off just using 'C'.
Also a good example of how it needs to be okay that some people just don't like Node. While also illistrating that they could also, arguably, be nicer about it ;)
What area do you generally work in? That often influences the view of which tools are suitable.
Also, context switching - the projects I work on tend to be front-end heavy, with a little API access and data storage on the backend. I find it a lot easier to do that backend work with the same language, tools and debugging environment as my client code.
So these days where most front ends are JS, it makes a lot of sense to have a JS back end too.
A lot of the problems mentioned in that article are the author misunderstanding libraries and not actual issues with NodeJs itself. For instance the problem with the Postgres library was that he didn't read the docs. The lib uses a session pool by default. You have to release the session when your done or else it will not become available again until the default session timeout is passed, which is quite large in Postgres. In respect to using Coffeescript "Classes", forcing JS to work like other languages will create unexpected behavior. It uses prototypical inheritance, not class inheritance.
Of course, Node.js is only suited for I/O-scheduled but not CPU-heavy workloads.
It's a fast, well-designed runtime written in C++ with the event loop as a central construct. It's an ecosystem where async is the default and you don't have to work hard to get performance and scalability in that area. And it has a fantastic package management tool. The quality of the packages that npm installs may not always be high, but the tool itself is so much better than pip, maven/gradle, gem, cpan, etc.
I've moved on from Node because I've found nearly all the benefits of Node available in Rust. Rust has the performance of Node's C++ runtime, easy event loop with tokio, and cargo is the one package manager that I think betters npm. And with wasm, we're even starting to see the possibility of sharing code between the server-side and client-side. Add to that the ability to use it anywhere that Node can be used (I'm using Rust now with Lambda without a single line of JavaScript), and there's now no reason for me to put up with the hassles that using such a poorly-designed language as JavaScript imposes.
[1] See eg the bit about NaNs in the article.
Even back in Koa 1, co/yield/function* closed the gap for me. Nowadays it's much better with ubiquitous promise usage and async/await.
I miss various things about Clojure like the editor-as-a-repl development cycle, but in my opinion, they still amount to small technical differences that play second fiddle to business concerns.
defmodule HelloPhoenix.UserController do
def index(conn, _params) do
users = HelloPhoenix.Repo.all(HelloPhoenixUser)
render conn, "index.json", users: users
end
end
For websockets, streaming, and other stuff beyond request -> database query -> response, then competing abstractions get more interesting.
Javascript of course ;)
You can leverage the same language for server side stuff and browser programming.
So, you just read/write to a database wherever? Or do you have to remember the context the code is running in?
There may not be syntax switching, but I think you stil usually need to remember where the code in question will be executing (well... I need to, anyway)
Javascript is easy. Express, NPM, Bower, <insert framework of the month> are tough.
If you check http://www.modulecounts.com/ you can see that npm has as many modules as all the other package repos combined. And it only seems to be gaining momentum. (screenshot: http://imgur.com/a/adqNJ)
Meteor has more stars on GitHub than any other web framework: https://github.com/search?p=3&q=stars%3A%3E1&s=stars&type=Re... Nodejs, socket.io, and express all appear on the first few pages.
That's a big misconception there. I absolutely don't mean this in an elitist way - and I completely agree that some degree of accessibility is a must - but this is in no way a benchmark for a good backend language. This is how we got PHP.
> [WebPage screenshot -> visual Diff -> send mail] > Try to do that in any other language.
I fail to see what's so difficult about this. I'd wager that in the "heavyweight stacks" you'd not even have to use modules outside the runtime libraries for a basic variant, and if you want to, you'd have them at your disposal.
Don't get me wrong, the large centralized ecosystem of NodeJS definitely is one of its biggest pluses.
But - and yeah, now I'm sounding a bit elitist and jaded - I've heard a variant of "It's so much simpler in X" or "Try that in anything else than X" too often when it only meant "I'm actually not really proficient in anything else but X".
Node was decided upon before I joined the team, because some of the senior developers on the team were big members of the node team, and they also wanted to crank out something new in this greenfield project. In retrospect, and in my reading of the history, Node was sufficiently different and new enough that it was "ooh, shiny new toy".
JS/Node might be nice for scripting up a new demo that doesn't need to be maintained. I'd use Python/Django or Ruby/Rails myself, but whatever. Small projects that don't need to be supported to me seem pretty language agnostic. Do it in whatever tools you're most familiar with, and don't worry too much about fit.
For large production systems that need to be supported and maintained, it's awful. First, javascript is changing incredibly rapidly. It seems like every couple of months, you're updating your libraries and there's a new "best practice". You're faced with updating all your already written code to use the new canonical way of doing things, or you're stuck with a mishmash of ways to do things all in production, and new engineers have to constantly ask what the new hotness is.
Stack traces are useless. Yes, this function got called with invalid argument values from the event loop. What function called it, and what arguments did it have?
Javascript itself basically has objects that are just JSON objects. When you're passing them back and forth, you get the irresistible urge to add "optional" fields just to pass one extra parameter for a use case. You generally end up with a big ball of wax parameter object being passed around with fields with unclear purposes.
Libraries are often half-baked. They work most of the time, and try to do the right thing, but sometimes just behave unexpectedly. I'm not quite sure whether to blame the people who wrote the libraries, or if it's the language itself that tends to encourage things like that. I'm guessing it's six of one, half a dozen of another.
I can't stand NPM. A lot of times, two libraries that you want to include will depend on incompatible versions of another, shared library, and you can download a NPM dependency tree for each library to overcome that. Everything hangs together, sort of.
It's very difficult to reason about code, and things are generally defined by convention, until they're not. The callback is generally the last argument passed to a function, except in async control functions where it's the second to last one, and the last one is an array of the results of previous clauses.
Saying that using the same language for the front-end and back-end allows engineers to work on both parts is silly, IMO, as front-end and back-end work require two disjoint skill sets. Just because surgeons and chefs both use knives, it doesn't mean that you'd want a chef performing a liver transplant. Also, node.js actually reads a lot more differently than most front-end code I've seen. Back-end code written in Java and front-end code written in Javascript tend to have the same sort of data flow and control structure, as opposed to Node which is on an island all by itself. I could go on further, but you get the gist.
Good it's not just me. Working in TypeScript I find it even worse; sometimes the stack trace uses my .map files and other times the compiled JS file features in the stack trace.
> Javascript itself basically has objects that are just JSON objects. When you're passing them back and forth, you get the irresistible urge to add "optional" fields just to pass one extra parameter for a use case. You generally end up with a big ball of wax parameter object being passed around with fields with unclear purposes.
This. Working browser-side I find it gets even worse when you don't want a pure data object but to add methods - but somewhere down the line the object gets persisted to localStorage. Not a JS problem, a problem with storage in any language - but because a Class and a data object are interchangeable, I see it happen more than usual (and cause unexpected bugs)
I don't think it's changing as quickly as you say, especially over the time since ES6 released/acquired first-class support, but you are right, this has been an issue.
>Javascript itself basically has objects that are just JSON objects. When you're passing them back and forth, you get the irresistible urge to add "optional" fields...
You use classes incorrectly / refuse to use classes and then complain when they don't behave like classes?
And then you blame the language?
Honestly, your two points preceding this were pretty reasonable. But I really think your bias starts to show yourself in the next ones.
>Libraries are often half-baked
You're really going to need to flesh this one out. React is half-baked, and so is the create-react-app surrounding it? Express is half-baked? Loopback is half-baked?
I won't deny shitty packages exist; Javascript has a low barrier to entry, and things can be mildly functional and kinda broken pretty easily. But I mean, it's obvious which packages don't suck if you just look at the amount they've been downloaded off npm and their issue lists on github.
>I can't stand NPM. A lot of times, two libraries that you want to include will depend on incompatible versions of another, shared library
In the last 18 months of developing in Node.js, I have not run into this issue once. Before that, sure, but I don't really see that as an issue lately.
>It's very difficult to reason about code
Promises or async/await fix all those issues. Again, this seems like a complaint 18 months out of date
There's issues with JS as a backend, to be certain. It's not nearly as performant as Java/Go/whatever language you fancy is capable of and it's single-threaded.
However, I don't think it's changing as rapidly as you think lately. I honestly must say that you sound like you had the issues of 2 years ago that have been improved on a lot by ES6 and a little maturation of the ecosystem.
The language is very flexible, and that definitely lent itself to a rapidly changing ecosystem at first. But if you want to take that flexibility and see how far you can bend it, then complain when it breaks, it's not a good look for a criticism of the language.
Anyways, this is a strangely impassioned defence, which I guess stemmed from the fact that your first two points resonated with me, but then your following points came up a bit short. I think you maybe haven't looked on the language with fresh eyes for a while, and will find it is not quite as easy to criticize these days.
I'm working on a very old node codebase - it's over 3 years old, which is absolutely ancient in terms of node and the pace of its development.
And you're correct, we're not using classes, promises, or async/await. However, this being a large system that's in production, we don't exactly have time to rewrite everything to take advantage of new features, and introducing another way to do something when we already have 5 ways of doing the same thing that are different with nuances seems silly.
And refactoring a dynamic language is just orders of magnitude harder than refactoring a static language. With a compiler, you have reasonable confidence that you've found every use of a particular function or field. With dynamic languages, you're never quite sure, especially with Javascript.
That's why I said that for a long-running production system, I would advise against Node. I'm sure something else would come down the pipe that would fix your particular complaint, but might introduce one of its own. And once you already have a large mass of code, you're sort of stuck with that version unless you have time to do a complete rewrite with the new coding paradigms that might introduce bugs. Heck, we tried to just use the latest version of Node, and all of our tests passed. When we tried to deploy it to production, we had to hastily roll back because it started sigseving after a while under load.
One of the largest advantages that people tout for node is that it's easy to write large quantities of code for. But that's also a curse for stuff you have to maintain, as it almost seems like you're taking a snapshot of the Node ecosystem at the time that you start writing the program, and it's extremely hard to update/upgrade it. Hence my dislike of it for any sort of long-lived production system.
EDIT: for an example for half baked node libraries, let's take a look at handling command line arguments. The two leading contenders are minimist and commander. Minimist gives you very few features, as befits its name. Fine. But if you want anything with more feature, you get to use commander. Commander doesn't really handle required arguments properly, always uses -h as a flag to trigger help (not configurable), and when you pass in two strings as flags to set the value of a named variable, it always sets the value of the one that's listed last. (e.g, if you want -version and -ver to indicate the same flag, the library parses what you pass into it but only sets the -ver variable when it parses things). And this is the second most used command line parsing library in node, as far as I can tell.
For work that is almost entirely IO bound Node is really fast and easy with asyc concurrency that works really well. Compared to most other solutions out there will you get a very fast solution very quickly. For problems which basically reduce to
-Wait for GET/POST request
-query a bunch of databases based on request
-return result of DB queries as JSON
But you could also achieve this by transpiling another language to Javascript of course.
Also I've had experiences with Typescript code compiling with an error then immediately recompiling without any errors. Then errors when I switch to my browser. The immediate recompilation is probably just webpack but it doesn't inspire confidence when errors are non-deterministic.
And I addressed some concerns with Typescript in a sibling comment. It is much better than plain Javascript, though.
It works fine for simple cases, but those are cases that were simple to begin with (Ok, maybe they would not be fine with CPS, but async/await is just thin sugar over promises).
To really solve the async problem, you need either Observables or monadic constructs. The former is much more mainstream and mature in the JS world right now. It's not as cute as async/await and you'll be typing more functions, but it really solves the problem altogether.
Is there any better workaround to that these days?
Node.js's own libraries are a mix of events and callbacks. Some of it can been "promisified", however they have yet to start a real async conversion AFAIK. I've had to wrap some of this stuff to create async interfaces for my own code.
Ditto for apply your own types atop the results. Most are already there and only a few edge cases need to be added.
As far as writing your own types on top of those libraries, how do you ensure any amount of correctness?
Am I missing something important here?
I think places like HN are probably distorting your view of how popular it is.
I think your average corporate developer is just trying to earn a paycheck and also has a petrified fear of meaningfully learning/using more than one language. Telling him "use this and you can use JavaScript everywhere!" sounds like a big difficulty improvement to him, especially since JavaScript is something he "already knows" and has been familiarized through years of being the exclusive client-side development platform.
Since this type of dev cares only about keeping his career on cruise control until retirement, Node.js is engaged with enthusiasm. In fact, it's one of the most enthusiastic receptions by BigCo devs that I've seen.
In one sentence: You can build anything with NPM + copy paste in very little time.
Its easy to quickly put up a prototype. There are a ton of libraries for everything, and most come with copy-pasteable examples. Yes, this is frowned upon, and with good reason - but for prototyping, its exactly what you want. The prototype you write will also work much better than your average dynamic language prototypes (except for Erlang).
With generators and async/await the code became fairly pedestrian and un-convoluted. The only additional element in the mix are the few await/yield keywords sprinkled around.
The language is flexible enough that you can also write code in FP style (higher order functions, combinators, etc). For example, RxJS code looks pretty natural.
ES6 and above has all the modern bells and whistles: decent module system, classes, short lambda syntax, template strings... As a result code is pretty much as pleasant to write as Ruby or Python
When your project grows large enough, you have the option to painlessly convert to TypeScript. Besides getting rid of one annoying JS flaw (implicit conversions), this will also improve development tooling tremendously. TypeScript again has a wide dynamic range: you can start out with a fairly lax type system that allows a lot and then turn the dial up by turning on more and more checks (no any types inferred, strict null checks, etc) TypeScript's type system is surprisingly powerful: generics, unions, intersections, type guards, discriminated unions, control flow analysis with exhaustiveness checks, string/integer literal types, "mapped types" - the list is quite large and growing: https://www.typescriptlang.org/docs/handbook/advanced-types.... . But what makes TS different is that it models and formalises idiomatic JavaScript well, so your existing code will not need to be adapted.
TypeScript is where client/server code sharing starts to shine. Shared types enable end-to-end type-checking. You can arrange your code in such a way that data fetching is injectable, and use the same code with fetch on the client and direct method calls on the server (thanks to the same interface). If you have a very demanding client-side app, there will definitely be a lot of code reuse opportunities. (Nowadays I suppose this is also possible with Scala.js and bucklescript)
The lack of threads is a mixed bag. Most people already covered the disadvantages very well so I'll mention some advantages. Its much easier to reason about stateful code, since you can treat each synchronous code chunk as uninterruptible, and the possible interleaving points are always obvious (e.g. await/yield keyword). There are also several techniques and libraries to work around the disadvantages (e.g. dnode for RPC to separate processes for intensive calculations) and while none of them are very convenient, they usually end up being what you have to do eventually anyway. Threads will only get you so far with servers running CPU intensive jobs - soon you will need more than one machine and then you can't take advantage of shared memory any longer.
If all else fails, you can probably throw more processes at it and put a load balancer. Thats an interesting project, and AFAIK not yet solved (at least not in the node ecosystem). You would need a load balancer that is aware of the current state of the node processes - round-robin or random algorithms will be far from optimal to be helpful. I started some work on this here: https://github.com/spion/least-latency-balancer but I'm sure that its not the best approach and that it can be improved a lot. Might be a good idea to look at what the OCaml folks have - I'm pretty sure they've been dealing with a somewhat similar problem due to the lack of multicore.
Another way to attack this problem is to have better monitoring for long CPU-bound tasks e.g. https://www.npmjs.com/package/long-task-detector
But I will admit that in this regard, there are languages/runtimes where the situation is far better: Haskell, Go, Erlang to name a few.
Especially one with different paradigms
Web development is about many things. HTML, CSS, performance, accessibility, security, networking, supporting different browsers, polyfills and workarounds, keeping up with new features, UX and usability. Then there are frameworks and build tools, which are all of course JavaScript-based.
Only a small part of web development is actual programming, and the fact that frameworks most of the time lay out the patterns for you, makes the “programmming” aspect even less significant in the overall picture.
So no, I don’t think that learning another programming language is a good way to spend your time if you’re a web developer.
It's not encouraging stuff. On paper the platform looks decent but aside from a few use cases it seems that there's more hype than it merits.
https://github.com/npm/npm/issues/10999
https://github.com/npm/npm/issues/9633
I have personally hit both while trying to build (not even write!) Node software.
The former bug is an example of "broken by design", and it's interesting that it remains open (therefore admitting that it is a problem?), but no solution came yet. Although I believe Yarn doesn't have it - but then why doesn't the entire Node community move to Yarn?
The latter bug is, to me, just insane. Several dozen bug reports (and even more if you google for the error message); and when it shows up, it can be an outright blocker. It was for me - I plainly couldn't build the software that I wanted, and no workarounds helped. And no fix, 1.5 years later and counting.
It's been the single least productive coding environment of my life. When you're taking care of large systems of backend servers, Javascript is pretty far down on the list of languages I'd pick to use. Add to that Node's explicit handling of asynchronous operations instead of just blocking and waiting.
Also, it seems to get worse the more code you have. I'm sure it's possible to write really good readable javascript in large codebases, and the same for Node, I just haven't seen it done.
Another subject the author brings up is "the ecosystem". Javascript has so many libraries that keeping up with them, their updates, and using them in a canonical way throughout a large codebase is a full-time job for at least one engineer.
I love the language, but this is absolutely true.
Also, there is a lot of research in design, and I do design and development interleaved. I could carve out slices of that work to call "pure development" or "pure research", but for me, design, engineering, and research are all part of one tightly coupled process. I call that process development for shorthand. Is there a better word?
When you have to worry about the environment unexpectedly reusing internal variables, when is it considered foolish to use a framework where you have to question every relationship between every concept? Just stop using it.
(I don't think JS "this" is that bad...it's just not needed at all)
Personally, I interpret it as meaning he believes that the average developer's baseline calibration for how much is to be assumed is too trusting and people should verify more assumptions than they do-- starting with the most dubious ones of course.
> I finally added extremely verbose logging ... started adding logging ... Verbose logging to the rescue ... I had the right logging in place ... My first step was to jump in and add some key logging statements ... added extremely verbose logging
Weird to see people happy to be limited to such stone-age work-flows when fully capable debuggers have existed for almost all proper languages out there the last 30 years.
Based on this, it's hard not to state that current Node-developer are the new PHP-developers: Unsophisticated, completely unable to see when the tools at hand are lacking, and happy with a with whatever they can get running.
Comments like this are not the problem with HN.
The fact that they get voted, literally, to the top, is.
And it'd often be quite a simple logic bug that I could see straight away.
Regularly using logging for debugging is often an inexperienced developer's crutch. It's a red flag, the sign of the desperate, that they can't comprehend their own code or don't seem to understand the business logic.
It can be a bad substitute for replicating the problem, reading the code + using a debugger.
Sometimes it really is justified. The author seems to have relied on it a lot though.
Like in the situations in the article. That's why the comment in question is not just blunt. It's a bad comment.
A good comment would take a specific instance of using logging and said why setting a breakpoint / using a debugger would have solved that particular mystery faster.
NaN bug doesn't sound like it needed logging
Mutability bug doesn't sound like it needed logging
Dependencies and versions bug doesn't sound like it needed logging
The Documentation and versions bug doesn't sound like it needed logging
For example, in the documentation and versions async.js bug, he added "extremely verbose" logging where it appears that a simple step-through would have immediately revealed to him the passed arguments were wrong and the signature of the method didn't match the documentation. Which would have made him realize he had the wrong version.
The only one where I would expect to have to use logging is the tests bug.
Even after you've read the code and think you've spotted the problem, you should step through to 100% confirm you've understood the problem.
It's simply confirming the bug and the solution, it's good science, it's good practice. It's empirical.
Even in those scenarios, I rarely have to look at live data, reading the code is often enough to identify it or playing around a little bit. The patience to replicate a bug before trying to fix it.
This is the essence that sets apart a good debugger from a bad one. A bad one resorts to littering the code with print/log statements because either they can't replicate the problem or they simply can't run the code in their head.
I'm being unfair, it's not even that, they simply need more experience. When I was younger I remember complaining that exception stacks were utter gobbledigook, utterly useless. Now, I read one and often know exactly what the bug is. I was completely and utterly wrong, impatient and unwilling to admit my ignorance. At the time I thought I was wise and knowing. They're very useful, I simply didn't understand them.
My "simply" from above belies the years of experience I have.
Because 17 years ago, I had this same naive development workflow. With PHP, exactly as OP mentioned.
Whenever someone mentions static typing, the canned response from JS developers is "Oh, haha, we actually have unit tests for this" (usually accompanied with a patronizing smile for the obsolete waterfall programmer). Yet when dealing with a real world dynamically typed application from some mediocre company, there are (surprise) no unit tests at all.
I think the best compromise is optional typing. The language should encourage it by default, but allow some variables to be declared as dynamic/untyped. This greatly simplifies things like parsing of JSON data, which should essentially all be considered a string until there's some reason to consider it something else, but enforces consistency and safety through most of the application. Haxe does a pretty OK job with that.
Like C#'s dynamic keyword (not to be confused with its var keyword, which just does type inference).
A debugger is a tool like any other, and sometimes it provides unique benefits, like if you're new to a complicated project that lacks appropriate logging and/or you'd like to step through code to try and understand the code's flow, but for tackling individual issues, I would be concerned about non-junior developers who consistently rely on a debugger for solving problems in code.
This is quite a loaded question, so let's unpack it a bit.
1. I made no implications regarding anything, my point is explicitly laid out with clear reasoning and examples.
2. I never made any remarks regarding how the use of any particular tool reflects on one's professionalism. I am guessing you're conflating my remark with regard to junior-level experience with the qualities of professionalism, and if that's the case I'll clarify that being junior does not in any way reflect on one's professionalism.
3. Your flippant dismissal of logging as "a bunch of print statements" is unfortunate and broadcasts a blind-spot in your knowledge of how robust logging is a critical component of any properly engineered software project, especially network services of any kind. It's not an either or question, logging is a must-have business/operations concern, whereas a debugger is a nice-to-have developer concern.
> Debuggers are incredibly useful in tracking certain things down and don't make you write and then remove a ton of logging statements.
You shouldn't be writing and removing a ton of logging statements, logging should be a planned and permanent part of your application from the beginning with appropriate severity labelling for debugging as well as run-time analysis purposes. If you want to use a debugger, that's fine, but a debugger is not a replacement for logging.
> Sometimes logging is easier, sometimes debugging is, and not just for "junior devs".
As I already stated in this comment, that's a false choice, logging should be the minimum, the debugger is there if you really want/need it. I never stated that the debugger is only for junior devs, I said I'd be concerned if a non-junior dev consistently relied on a debugger for solving problems in code, which is not at all the same thing. The use of a debugger is appropriate at any experience level, but generally speaking, the debugger is a slow troubleshooting tool that is best suited for exploring code flow or unusual bugs that seem to defy bedrock assumptions about how the application should be functioning. They are not the best tool for everyday bugs and troubleshooting.
There are fancier setups than "print statements", like binary lock-free logs to circular memory maps. But in essence, print statements, yes.
And heh, I just realized that if you're using a debugger with reversible execution tracing, you're using EXACTLY a binary log file.
> Based on this, it's hard not to state that current
> Node-developer are the new PHP-developers:
> Unsophisticated, completely unable to see when
> the tools at hand are lacking, and happy with a
> with whatever they can get running.
Btw the tooling in the Node ecosystem is actually pretty good.
You also seem optimistic, unit test don't catch new bugs, they just test for typos and the bugs you already knew about.
You write a new unit test using the data which caused the bug but expecting a non-error state. The error that is being thrown/rejected will have a stack trace you can inspect to find the line which threw. To find the real cause you need to read the code before this to find out how the bad state was created.
If you kept things small and simple, it is not so difficult to work out what went wrong. If you let things grow in complexity too much, you can however end up within a breakpoint/log quagmire.
Programming anything beyond a small app is ultimately managing complexity and often fixing how the various moving parts don't quite work together as you intended.
You simply can't, and shouldn't, unit test that stuff.
Even though node has a debugger with breakpoints, etc., an effective programmer will still use logging in situations like the ones in the article. Taking the first example, before he knew that NaNs were causing the problem, setting a breakpoint would have taken forever to reveal anything. You can't set a conditional breakpoint until you know what the condition is. Logging was the fastest way to get useful clues.
In others granted he's working locally and probably could have found the problems without logging, but that just comes down to preference IMO.
Node does have a debugger though and recently added official (though experimental) support for attaching the Chrome DevTools debugger [1]. Doing so was doable via 3rd party packages previously.
Edit: updated as author wasn't only talking about in-production.
[1] https://nodejs.org/api/debugger.html#debugger_v8_inspector_i...
PHP has had debugging for a long time, PHPStorm is pretty much now the standard in php development and includes full productivity in that area. It also doesn't face the issues web-JS has with pre-processing, post-processing and compilation of JS code that has to have a source map back to the original source.
There are also debuggers for PHP :) No horse in this race, I write Python.
Also, while I agree mostly with you, there are cases where logging is more useful for debugging than using step-through debuggers. However, such problems are usually related to concurrency where a debugger may alter or even hide the problem, not stuff like this here.
Particularly with real-time, concurrent code where you're interacting with an external system, bringing a thread to a halt in your debugger causes all sorts of false-positive timeout bugs. You get about one shot to break with the debugger, then your state is trashed and you have to start over.
I would bet that most developers know more than one language. So what if you develop with Python, C++, and Node? Does the singular fact that you develop with Node make you a shitty programmer?
You can execute everything inside a promise like with Koa, which amounts to something like:
http.createServer((req, res) => {
handle(req)
.then((response) => res.send(response))
.catch(() => res.send(500))
})
In a Java app you except all the way to the request response.
In a Go app you return all the way to the request response.
In a Node app, you have to call your way back to the request response. Or you don't, potentially leaving the whole process in an inconsistent state. The amazing concurrency of node comes at a price.
They're soft-deprecated in Node now though, and AFAICT work on a replacement has stalled.
Ie. Be okay if a syntax error crashes your server.
I came to this after building C#, Scala, and Java webapps, so it surprised me that this these class of coding errors could take down a site (both screwing up the try/catch or not handling promises).
There is probably some room for Node/the language/Express to make the experience better, either way it's something you'd need to know if you're new to the platform :)
This can be problematic at times if you need to open a very large number of concurrent connections (crawlers, slack connections etc), and can force you to organize a "cluster" of processes before you really want it.
>>Ie. Be okay if a syntax error crashes your server.
Really? I need to set something up to stop a syntax error not crashing the server
These rapid prototype arguments don't hold either imo. I can knock out quick and dirty java / c# apps plenty quick enough that work fine for their needs. In my forays into dynamic languages there is no rapid development due to all the errors, the lack of IDE capability, the compiler replacing tests I have to write
However with Reactjs + React Native, you have an unavoidable component of your company that is in javascript (or will have soon). Why would you not do JS based server runtime ? With typescript/flow, you actually have an excellent language with no compile steps. Plus, and probably the most important, you have a cross pollination of engineering talent and time .
I think its the same case with data science - it is much more productive to build a python based data science startup, because of flask+sqlalchemy+jupyter+pyspark+numpy+tensorflow+keras. Yes, you can use scala or java... but you lose that one-language-to-rule-them-all productivity multiplier.
From a quick prototyping point-of-view, i think golang would have theoretically killed nodejs.
When you asume something in JavaScript, for example that the second parameter in the function is a callback function make sure it is!
if(typeof callback != "function") throw new Error("Expected callback=" + callback + " to be a callback function!");
if(isNaN(someNumber)) throw new Error("someNumber=" + someNumber + " is NaN! someState=" + someState);
if(something == undefined) throw new Error("something=" + something + " is undefined!");But .. I have this niggling feeling .. as my code develops I'm noticing the expressiveness of Javascript lets me do things that look nice, and impressive but that I know will come back to bite me when it comes to enhancement and maintenance.
In this respect Javascript kind of feels like perl with OO and functional semantics built in from the ground up. It really is a lovely language but I wonder how well a JS-based system will scale over time. I've heard people say "never use ruby in production" and I wonder does the same reasoning apply here.
The argument goes that building apps that use the same language client-side/server-side carries the virtue that you can write "pure" apps - i.e. that share code front-end and back-end. That is definitely something that makes sense, especially if you need shared libraries you won't need to write the same code twice for each end.
But beyond that, I feel the expertise required at both ends is quite different. As a systems programmer I feel that strongly-typed rigid languages are very much the way to go because they give you a better sense of how robust the code is. Typically on the back-end you want to get something to work once, and leave it, so you need to be sure it's right.
My more limited experience working client-side suggests that you need far more flexibility there. Issues are far more easy to spot front-end and the key virtue is to be able to make changes and enhancements quite quickly and you don't have the same stringent code-confidence needs as you do on the backend because the quality requirements are different. Niggles and gotchas can be spotted, diagnosed and worked around on the front-end but on the back-end they become a royal pain.
So this, is what I think Javascript is great for on the front-end. I do love the simplicity of bootsrapping Node.js on the backend and the actual reactive framework for building apps is really cool. But yeah, for systems code "at large" I see limitations there for sure: The same we had with Perl, Ruby and Python. Perhaps what we need is some alternative language support e.g. Haskell or something that gives the same code-confidence, but which can interoperate perhaps with the "pure" javascript libraries similar to how say Scheme and Java can interoperate.
I like the fact that I've been able to follow along as it's matured and this gradual bit by bit learning process has made me a better developer because of it.
I have been absolutely disappointed by Node's tremendous number of gotchas in deployment and massive complexity of build pipelines needed to get around the inherent issues of the platform.
My deployment with Django, for example, have been much simpler to handle.
you can't derive Eq (transitive equality) on f64
but you can derive PartialEq (intransitive equality)
"And I found a whole lot of references to req.randomThing and even some req.randomFunction() calls. I then proceeded to search back through every single middleware function which had run before, to figure out what exactly was going on."
In a statically typed languages names are bound at compile time, so you can actually jump to the definition of each name using tooling. A JS tool can only make an educated guess.
Which is true, but ultimately useless
Can someone recommend a Nodejs performance monitoring solution that works great?
Devs these days are craptastic coming from dev->systems with devops nonsense. Just stop forcing this terrible language into server and js is as fine as it will ever be client side.