Interview with Ryan Dahl, Creator of Node.js
mappingthejourney.com
mappingthejourney.com
This means deeply nested callbacks such that the indentation wanders steeply to the right. When you do loops or have exceptions programming got very confusing. In effect you have to do a continuation passing transform. It's not difficult but the results are utterly unreadable. For example a simple for loop could be transformed into recursive function because this way it is easier to keep state between callbacks. It's a nightmare.
What I found surprising in the early years of Node that some people in the mailing list had a «Real programmers don't use Pascal» attitude.
Bruno Jouhier's developed streamline to simplify and automate the continuation passing transform. One could almost program in a «blocking» style. I found his work really amazing. However for that Bruno got attacked in the mailing list.
Then Marcel Laverdet developed fibers. A different and equivalent way to solve the callback pyramid are fibers, coroutines and generators. This way we don't need the continuation passing transform. But the reception in the mailing list was at best only lukewarm. Anyway, the developers of Meteor saw the potential and decided to base the server side of the platform on fibers.
And now JavaScript has generators and async/await. And lo and behold: with these official solutions the hard core programmers swayed and accepted «Pascal».
In my opinion, Ryan Dahl has missed an opportunity. Node was by large not ready and finished when he left. He should have tried to convince the community to find a solution for the callback hell.
I think I understand people like him. They are always on the lookout for fresh ideas.
I was the primary instigator of generators for Python. Part of my motivation was to do something like CSP (e.g. call-with-current-continuation, call/cc) without having to redo the whole language runtime and also kill off alternative Python implementations.
Generators in Python are limited to a single stack level. They have been extended to better support async IO coding. They work and are probably nicer than using callbacks but are still pretty ugly IMHO. The message passing that Erlang does is the best solution to this kind of concurrency. You can't really bolt that on a side of an existing language though.
Anyhow, I think Node is successful largely because Javascript is the language understood by web browsers. The non-blocking coding style is not some kind of magic bullet as Ryan originally suggested it was.
It always felt like every other code-nesting problem to me.
I mean, nobody talks about conditional hell, everyone acknowledges that you should flatten out your conditionals.
This https://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/spr... is not an exaggeration and I still would not use it to showcase how I felt when I had to work with Java developers.
As far as I remember, we were never so much against the idea of it, but we didn't believe an actually good implementation was possible.
The primary concern was scalability. At the time supporting high concurrency networking (10K+ connected clients) was a top priority for us.
The solutions proposed by Laverdet and Jouhier used libcoro or fibers to achieve the desired "cooperative threading"; this may not involve actual OS threads but it does require the creation of a thread stack for every fiber. To handle 10K connections you would need 10K stacks so you have to allocate 40GB memory right there, too much.
We were aware that there were more effectient ways to do it, but this would have big changes to the V8 javascript engine. We didn't have the resources (nor the skill) to get it done and take on the maintenance burden.
There were also concerns about the surprising language semantics it created; suddenly callbacks might run "inside" a function, e.g. after it's called and before it has returned, not something a javascript user would normally expect to be possible.
Nowadays with async/await the callback-hell problem is actually solvable (to the extent that node-fibers solved it, anyway). It'll take time though before that becomes noticable, because a lot of packages need to change their APIs to take advantage of it.
But I agree that js attracted "Real programmers don't use Pascal" kind of programmers at first. Pascal style programmers perceived js as pure hell and treated it as abomination. Through, I am not sure whether they were swayed or rather moved elsewhere in disgust.
And conversely, you can reduce nesting to, basically, zero in any imperative language, if you replace all your conditionals and loops with a bunch of labels and gotos.
Which is to say, how much code nests is very much a language design issue. Good PL design should not cause unnecessary nesting where it's not warranted by the structure of the solution.
I love callbacks. The best thing about them is that you always know precisely what will happen when a line of code runs. Not so with promises.
Another nice thing is that you always know which line of code will run next. Not so with async.
I see async in the language spec (and class too) as a concession made to people who couldn't be bothered to learn how to use JavaScript the way it was intended. Like burgers on the menu at a Mexican restaurant.
And no judgement against them: I'm happy for them that they got the ruby-like JavaScript they always dreamed of.
But I wish those people would be a tad less boastful and understand that they turned people like me into exiles in our own language community. I used to have a language that promoted small single purpose modules with functional interfaces. That's gone. It's mostly synchronish OO interfaces now. And bigger and bigger frameworks. My folk are going to have to basically move out. I don't see an alternative at this point but to fork NPM.
https://github.com/andreas-gone-wild/snackis/blob/master/sna...
Then came node.js. Look at the ecosystem it came into. JS itself has very little standard library, and nothing for I/O. It has a highly optimized VM supported by the resources of Google. It's a language known by many developers. node.js took this well developed clean slate and built a non-blocking I/O server on top of it. It offered a concurrency model on the server side that's completely new to many developers, an alternative to the traditional threading model everyone knew. Since server-side JS didn't exist yet, it forced all libraries to be written in this non-blocking way. Every package in NPM is written to support the core design of node.js which is non-blocking I/O. And it was exciting to many developers, it was a reason to rewrite everything in this new way.
C# in particular does a great job at conflating concurrency and parallelism. Both fall under the same Task namespace and it drives me mad to no end. I've also seen a couple instances of Execute and ExecuteAsync both blocking because the underlying driver didn't support async, with no indication other than my async calls blocking.
It's quite different from the async-by-default sort of thing that's going on in the node.js world.
Rather than screaming, perhaps read this (from the developer of the Python world's most popular ORM) to see why they might be suggesting it:
http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...
tl;dr: If your database is properly setup and provisioned it shouldn't be a source of blocking I/O (and if it is, throwing thousands of concurrent async connections at it isn't going to help you). For standard CRUD applications, your program will spend much more time in CPU processing the database response than it will waiting on the database to deliver it.
Furthermore, when querying a traditional RDBMS like postgres or mysql, safe concurrency is ultimately enforced by the ACID capabilities built in to the database itself i.e. transactions. Whether you're using async, threads, multiple processes or something else entirely doesn't matter. Given the database itself controls how much concurrency it can support, real world tests (at least for Python) show that threads give you better performance than event driven callbacks.
Of course, threads are completely wrong for the HTTP side of things and this is where async (e.g. nginx or node) offers all of its advantages. That doesn't mean it's wrong to put your DBMS behind a thread pool.
Although I'll be happy to point out why Go's model isn't even good. Go's lack of ability to perform meaningful abstraction combined with the low level of its concurrency primitives means you end up having to copy/paste and slightly tweak dozens of lines of flow-control logic every time you use them.
This[1] is an excellent critique of Go's concurrency model.
[1]: https://gist.github.com/kachayev/21e7fe149bc5ae0bd878#so-wha...
No such thing as perfect, only trade offs.
This is a general problem with all "green thread" abstractions - they are not OS level, or if they are (as e.g. fibers are on Win32), nobody cares to use them.
Promises, on the other hand, can be mapped to a straightforward callback-oriented C API, thus allowing a fully async execution stack end-to-end across different languages. You can even go one step further and define a uniform API for all such callbacks, as WinRT did.
https://msdn.microsoft.com/en-us/library/system.data.common....
> Providers should implement this method to provide a non-default implementation for ExecuteReader overloads.
> The default implementation invokes the synchronous ExecuteReader method and returns a completed task, blocking the calling thread. The default implementation will return a cancelled task if passed an already cancelled cancellation token. Exceptions thrown by ExecuteReader will be communicated via the returned Task Exception property.
Found that after several hours of tearing my hair out trying to figure out why my async calls were still blocking...
What? I am sorry but this is not true.
Coroutines have been a concept for...well a long time:
https://en.wikipedia.org/wiki/Coroutine
Green threads also have been around for a while.
> but these all had a critical flaw, which is the rest of the ecosystem didn't support it. Basically every library, almost all existing code used the threading model. And when you mix threaded libraries with a non-blocking I/O server, it completely falls apart because the threaded code blocks.
I am not sure how you can say this while using Python as an example. Python has the GIL which, typically viewed as a limitation, actually prevents the situation your assertion suggests. In fact, one of the largest barriers to removing the GIL is the wealth of libraries and C-extensions to Python that implicitly rely on the thread safety guarantees/limitations of the GIL (as covered by Larry Hastings as part of his first Gilectomy talk).
To be clear, there are reasons Node became popular, but let's not pretend that it brought some sort of concurrency revolution to the "server-side landscape" in 2009.
Edit: Heck, NGINX was an asynchronous, event driven web engine released in 2004 and written in C:
What web frameworks were built around using them? Was there a good ecosystem of libraries that were easy to use with the web framework without introducing blocking I/O?
>I am not sure how you can say this while using Python as an example. Python has the GIL which, typically viewed as a limitation, actually prevents the situation your assertion suggests. In fact, one of the largest barriers to removing the GIL is the wealth of libraries and C-extensions to Python that implicitly rely on the thread safety guarantees/limitations of the GIL (as covered by Larry Hastings as part of his first Gilectomy talk).
Aren't all python threads backed by OS threads? And pointing out that threading has been used by too many Python libraries to change supports the previous comment's assertion that non-blocking I/O was hard to use exclusively in languages like Python because too many libraries used blocking I/O.
>Edit: Heck, NGINX was an asynchronous, event driven web engine released in 2004 and written in C:
Nginx is pretty specialized. Not really sure it's useful to compare it to a programming language / web framework with an ecosystem of compatible libraries like PHP, Ruby on Rails, django, nodejs, etc.
Ironically, now that we have async/await to replace nested callbacks, Node finally has the code readability that gevent users took for granted all this time.
Quite honestly, it wasn't very good. It worked magically until the magic ran out. Mysterious breakages caused by stuff deep in gevent interacting badly with other libraries hampered my ability to use it effectively. For instance, you couldn't use it with python-requests directly, you need to use another library that makes requests compatible, unless you roll your own magic, which was often very complicated. And, at least in my experience, gevent stuff tends not to work with Python 3.
Now that asyncio is a thing, I'm glad I never have to debug gevent-related problems anymore.
The language I'm referring to is none other than Haskell (which has been around in some form since 1990).
Here's the earliest paper I know of detailing the internals of the Glasgow Haskell Compiler's IO event handling system (from 2010, I believe): https://static.googleusercontent.com/media/research.google.c...
And here's a more recent update: http://haskell.cs.yale.edu/wp-content/uploads/2013/08/hask03...
(Just in case of link-rot, the titles are respectively: "Scalable I/O Event Handling for GHC", "Mio: A High-Performance Multicore IO Manager for GHC")
Here's a short example:
main = do
putStrLn "Hello, what's your name?"
name <- getLine
putStrLn ("Hey " ++ name ++ ", you rock!")
It drives me crazy when people tout Node's approach to concurrency as being ideal, as if callbacks are a necessity.
They're not.
Haskell, among other languages, is an existence proof that you can have lightweight threads and asynchronous IO without callback spaghetti. And I haven't even gone into Haskell's threaded parallelism, Software Transactional Memory, etc.
No, I'm not missing the point.
> Your example shows a simple hello world and doesn't speak to the point being made.
Hmm... Let's do an analysis:
* My example demonstrates reading from a given file descriptor (FD 0 -- stdin -- in this case), and that reading happens asynchronously (as I had pointed out, and even provided white papers describing how this works). Again, this is despite the syntax looking like a naive, synchronous program. The GHC Run Time System handles asynchronously reading from the FD and scheduling threads as appropriate such that OS threads aren't tied up waiting on what would otherwise be a synchronous read(2).
* As a bonus, it also demonstrates writing to standard out.
Now, one could take exception to the fact that my example doesn't give any evidence that similar asynchrony should be expected in a web stack. One would be wrong: as anyone with, say, as little as a year of experience writing web applications on POSIX systems should know, the Berkeley sockets API exposes sockets as file descriptors, so the same generic async event handling system in GHC's RTS applies equally well to networked IO. GHC and Node (via libuv) uses similar primitive kernel APIs at the end of the day, the only difference being that Haskell doesn't require that you reify the callbacks in the source language.
Idiomatic? Check.
By default? Check.
So no, I think I do get the point, but if you still disagree, I'm all ears.
If my tone sounds harsh, it's because I spent the time to provide links to two papers and even gave a small example of what the syntax would look like for a minimal program that demonstrates IO. In turn, you're telling me that I'm missing the point. I would argue that, if you read either of those two papers, and you knew about the implications with respect to networked IO (if you didn't, you could easily research the Berkeley sockets API, kqueue(2), poll(2), etc -- all mentioned in the papers), you would see what I'm getting at with my example code and the papers referenced. I can only posit that you haven't read the papers, or if you did, you must have not done so in earnest. Hopefully you can see how I would be frustrated when I've put in time to research this space, and when I share something, I'm met with what (in appearance) amounts to a dissimilar level of diligence and, ultimately, an empty dismissal.
If something else is going on, I would be more than happy to issue an apology.
> I am sure you can get a more relevant example going in Scotty, but it appears that it didn't show up until 2012.
Open source web frameworks for Haskell go back to (at least as far as) 2009.
Happstack: (2009): https://hackage.haskell.org/package/happstack-server-0.1
Yesod (2010): https://hackage.haskell.org/package/yesod-0.0.0
Snap (2010): https://hackage.haskell.org/package/snap-0.3.0
I didn't provide a "hello world"-like example of a web app because my command-line example only requires a trivial inference on the part of the reader to realize that the same asynchronous treatment of FDs there applies equally well to sockets, and thus web app programming in general; I deemed a full fledged web app example as more complicated than necessary to get my point across.
For what it's worth, here's an example in Happstack, straight from the documentation:
module Main where
import Happstack.Server (nullConf, simpleHTTP, toResponse, ok)
main :: IO ()
main = simpleHTTP nullConf $ ok "Hello, World!"
$ curl http://localhost:8000/
Hello, World!
> It's that you could quickly write web services idomatically that did so by default.
I wasn't questioning the rationale for using Node back in 2009. My point is this: eight years have passed since Node was released, and alternative solutions have since been released. Somehow the Node community (and the greater web dev community in general) are oblivious to these alternative approaches.
I suspect this is because most web devs see the callbacks as evidence that IO is asynchronous in Node, and at the same time lack the requisite experience to intuit how another language might implement asynchronous IO while providing a syntax that looks like the usual synchronous model. Now, they could actually read the research papers put out, but they don't, and so everyone keeps going on about the technological revolution that is Node and its approach to IO, oblivious to alternative solutions that (arguably) provide a more convenient programming model (and is, if we throw in support for multiple threads, more computationally powerful, too).
Such laziness and/or lack of curiosity is pervasive in the industry, and that's frustrating.
Your indinctment that the reason people aren't using your language of choice as lazy is ironically lazy. There are plenty of practical reasons for using different languages and tools.
function sendMessage(message, toClients, callback) {
var messagesSent = 0;
for(var i=0; i<toClients.length; i++) sendTo(toClients[i]);
function sendTo(client) {
client.send(message, function messageSent(err) {
messagesSent++;
if(err) console.warn(err.message);
if(messagesSent == toClients.length) callback();
});
}
}In the interview, he mentions himself that Node was inspired by Nginx.
The ONLY reason to do callback-based code is for performance reasons: for network-IO-heavy apps there is less OS-level overhead from managing thousands of connections with a few threads using the OS-provided non-blocking I/O operations than managing a connection per thread. Even then, if you actually want to do better than connection-per-thread, you have to avoid the POSIX API for non-blocking I/O and use whatever OS-specific API is available for better performance (epoll, kqueue, or completion ports).
The greatest joke of all though is that Node.js is dirt slow because Javascript is dirt-slow. Yes, you may be able to invent some microbenchmark where Node.js looks good, but the only reason it does in that benchmark is because the Node.js app is spending 99.9% of it's time executing code in a C library. Any real-world Node.js app is going to be slow because Javascript in general, even on V8, is anywhere from 10-100x slower than C or even well-written Java.
And worst of all, Node.js is single-threaded! If you want to scale onto all your cores you have to go multi-process, which is an incredible pain in the general case. So, you switch to a runtime and framework that forces a more difficult way of programming on you (callback hell) but which is better for general networking performance, but you use a dirt-slow, single-threaded runtime while doing so. You are basically ending up with the worst of both worlds, and any decent development team will run circles around you in performance and productivity with just about any other backend stack.
I know of one startup that had a fortune 500 company sign up to their service and their dotnet APIs couldn't hold the load and the cost of scaling them wasn't sustainable. They rewrote the hardest hit APIs in node over a weekend and were shocked at the improvements. I also know another startup that had a similar experience, they moved off of Ruby to node and were shocked at the speed increase that they were getting. They couldn't be competitive with their competition with their old codebase.
Despite what I just said above speed isn't necessarily the most important factor. Infrastructure is cheap and developers are expensive. If I'm asked what the best programing language or platform is my answer is usually the language and platform that your team is most productive with. Only a few companies get successful enough that speed is the deciding factor in platform choice, if we say that productivity is the most important factor then the JS/Node combination is right up there as one of the more understood and productive systems.
Callback paradigm has not been used in node development for 3-4 years at this point.
return someHttpCall()
.then(response => someOtherHttpCall(response))
// Do 2 async operations in Parallel:
.then(response => Promise.join(
someCacheThing(response),
someDBThing(response)
))
.spread((cacheResponse, dbResponse) => sendBackToClient());
Clustered express application puts the performance well within what you would expect from Go/Phoenix, all of which are operating at an order of magnitude above Python/Ruby.
If you've ever actually implemented a promise it's even easier to see how little promises actually get you. Given a callback-based API, it's absolutely trivial to wrap it in a Promise-based API. Fundamentally this is because promises give you very little in the end on top of callbacks beyond some syntactic sugar.
If you want to change the topic from "callback hell" to debugging across stacks, then we can do that. There was some work with "domains," since moved to AsyncListener/Continuation Local Storage to keep context across stack calls, but I don't disagree that improvements in this department are the place that NodeJS paradigm would currently benefit from most.
A different programming paradigm, for instance, would be the use of delimited continuations such that the API you invoke suspends the current call stack, executes the I/O operation, then resume the call stack with the result of the I/O operation. Very few languages support that. One reason is that natively supported continuations in the programming language has far reaching consequences in terms of performance.
I'm not an expert on golang, but I believe this is essentially how goroutines are implemented: https://groups.google.com/forum/#!msg/golang-nuts/t3g8NbITYE...
Also keep in mind that I'm not 100% against callback-based programming. I have, for instance, implemented a non-blocking client for a custom TCP-based protocol using Netty. The client performed better than a blocking version that it was meant to replace. I played around with using Promises in conjunction with the library but ended up preferring callbacks to Promises for a number of reasons, some of which are specific to the JVM.
guile has recently gotten guile-fibers which works like this. It is really an amazing way to do concurrency.
Then you're not using promises as they are intended to be used.
https://pouchdb.com/2015/05/18/we-have-a-problem-with-promis...
If we're trying to show the simplicity of modern nodejs code with non-blocking I/O, I would suggest we lead with examples like:
const foo = await queryFooObject(id); await writeToFile(foo);
True, there's still plenty of value in promises, especially for things like:
await Promise.all(severalParallelQueries);
Beyond that, there's things like Erlang and Elixer, which I am less familiar with. My understanding is that the Erlang VM is significantly slower than JVM or CLR implementations but generally faster than scripting language runtimes.
In my opinion, Node.js became popular because people wanted to run Javascript in a server environment.
The new thing that it brought was an entire environment that used non-blocking I/O exclusively without any pre-existing thread-based code and libraries. With node.js, you don't have to wonder if a library you want to adopt supports non-blocking I/O.
Also callbacks for all that they're derided now were an easy paradigm to understand to get into the whole philosophy of async. The first prototypes of node supported different paradigms before callbacks were settled on IIRC.
At some point Isomorphic Javascript became the main reason to use it, as golang has largely stolen the non-blocking IO crown with its implementation of green threads.
JavaScript on the server before Node was a second-class solution.
On edit: so yeah, a nice way.
I think the other parts of it are: 1) many people wanted to write server-side Javascript, 2) people did not want to use Java, and the concurrency stories for PHP, Ruby, and Python were (and still are) really crummy.
Each language had some niche library that did
non-blocking I/O [...] Netty for Java)
More then Netty supported it. Netty, Jetty, Dropwizard, Grizzly, VERTX. Apache Tomcat supported non-blocking IO 5 years ago.
Node.js isn't quite as old as you make it out to be, there were plenty of mature tech stacks out there with PHP/Perl, Ruby, C#, etc. Many of them perfectly fine to work with. Node.js came along with a very different conceptualization of scalability relative to other web-servers, and that was what drew people to it. Today we have things like varnish, nginx, fast-cgi, etc. as well as much faster servers which make it a lot easier to scale out web stuff without having to redesign it from the ground-up, but node.js was for its time a hugely advantageous way to approach scaling. And it still is for many use cases, but today it's matured into something that has a lot more to offer.
I had the feeling before Node.js, everything was PHP.
I'm being petty, but serverside JS did exist before node (e.g. Narwhal, which I believe is where jsgi and a lot of the commonjs stuff originated)
I think node.js finally made server-side JS take off because 1) it was based on V8 which was the new, performance-oriented JS engine powering Chrome, and 2) it was self-contained (i.e. didn't rely on Java on one hand, wasn't just a scripting layer of something larger on the other).
yea, I remember the woes of building CouchDB from source....about half the issues generally seemed to be SpiderMonkey related.
It was basically "ASP-style" server-side javascript that ran on Netscape's "Enterprise" web server. It was a little unstable, but pretty advanced for the time (had DB connectivity, etc.)
It was once true, but that was 20 years ago.
People have a short memory, but at the time getting single to double digits of requests per second on a beefy server was entirely typical. Yes, someone somewhere could demonstrate an alternative, but node completely changed what was normal. It was extremely high performance for the time.
Umm, what? It was not a challenge to get to double digits of requests per second on a beefy server in 2008, at least not in any popular language (including PHP and Ruby). It was (and still is) easy to scale requests of "blocking IO" languages by spawning additional threads at the server level.
A completely standard developer on .NET or PHP in 2008 was building pages that rendered at less than 10 requests per second. This is experience, not a guess, given that my role was improving the performance of those disasters. A completely average developer on node was building pages that was 10x to 100x better performance.
Secondly, simply spawning threads is laughably non scalable.
I ask because in 2008 I was only about 3 years into professional web development, and even I knew about stuff like the C10K problem[1] and that I/O Completion ports were apparently one of the few things that Unix people admired about Windows.
https://blogs.msdn.microsoft.com/rickandy/2009/11/14/should-...
How else are you going to do it with a non-threaded language? I've ran well into the 10s of requests per second running "standard" PHP code on practically vanilla Apache configs. This was far from atypical and is no way any kind of challenge on a beefy machine.
My job was not to clean up disasters though. Maybe if your job was to clean up disasters, your typical was much different than mine?
To get cpu bound parallelism in Node.JS you have to spawn child processes. There's a built in module in NodeJS called "cluster" that abstracts this a bit. When talking to a child process you could just as well be talking to another machine over the network, the code will look the same, and scaling horizontally across many machines will be easier, compared to languages that solve concurrency by using threads instead of non blocking IO.
There's a learning curve though. It took me about six months to learn how to do async programming, manage messages between different processes and machines, and callbacks, I do remember the "callback hell", but now it has become a second nature, it's like brushing my teeth, but more fun. I don't think half a year is that bad, you would need just as long time to get comfortable in any other language. And I consider JavaScript very newbie friendly, you don't need a CS degree to write JavaScript.
No it doesn't. The only way to do truly multithreaded code in PHP is using pthreads.
Python and Java threads don't block on I/O, only on CPU. Event loops (i.e. node) also block on CPU.
AFAIK, node's advantage over threads was just that it offered much higher concurrency for I/O constrained tasks for the same memory (and was easier to programme for people coming from the front-end world).
He showed us that we are doing I/O completely wrong and also taught us how to build software using pure async programming model
Really interesting. I can imagine others would refuse to give up on the thing they'd worked so hard on. But Dahl has the self-awareness to just step back and say "huh, guess this isn't so great after all".
Huge props to the guy.
But, sadly, part of the saga of nodejs could have been avoided by people looking at the history of computing. If you look at Go's concurrency model it was entirely mapped out in the 1970s in CSP. There wasn't really any need for go the 'single thread, non-blocking' route that nodejs took and evangelize it as nirvana. There was a ton of distributed systems work done in the 1970s. Just look at Lamport's clocks paper.
I'm not criticizing Dahl directly here, more a sense that people ignore history in a field that has only had history for far less than a century.
The reasons it ended being very big are clear in hindsight but are hard to predict:
- Sharing code between server and client
- Server side rendering
- Lower barrier to entry than other platforms (learn once write anywhere)
- Well-designed package system
It was also helped by the fact that folks decided to write JS tooling in JS, be it minifier, transpilers, or test runners, without nodejs we would've built then on top of something else.
It was my understanding that "server side rendering" was the way the web worked since the beginning, with the server generating the markup that is provided to the browser to render the page. Nobody called it "server side rendering" until we had "client side rendering", where Javascript executed in the browser provided the markup instead of whatever program was running on the server.
I'm not trying to lecture or be snarky here; I'm genuinely confused as to what people mean when they say "server side rendering" as a new thing. Like, is there something novel that I'm missing because the term is overloaded with a definition I don't know?
Previously you had to either duplicate the code in different languages or pick one or the other (either use angular or rails).
I agree that it's a bit of a misnomer, but I can see why people would call it that given the path we took to get here: first we rendered almost everything on the server, and then it became popular to render everything on the client, but now we're finding that a compromise is best, and that compromise is easiest to achieve when you have similar technology on both sides.
("We" in the above is intended to represent some vague idea of the web development community at large, not any particular individuals or groups.)
What I came to understand, was that "server-side rendering" refers to whole application lifecycle steps: - The server generates initial HTML markup, based on the request. - The browser can parse & render the already generated HTML (instead of waiting for the client-side JS to load first, then have it create / modify the DOM). - From that point onward, all changes in UI happen through client-side rendering.
You could do this with Django template / JSP pages / Jade templates as well. Just write your templates, and some jQuery sprinkled here and there, to dictate how user interaction should change the page.
But that's not all there is to it.
We slowly took away SSR, with client side JS frameworks, like Backbone, Angular 1, Ember etc. Applications started to have client side routes, even with ugly hashbangs in the URLs.
It was easier to write entire application logic of your UI in a client-side framework with two-way data bindings; than writing the server side in a template language, and put in some JS here & there to handle interactions. Especially, if your application was big enough.
But this started having problems with SEO, and initial load times.
Say, your app has a homepage URL, of the form https://company.tld/, and a products URL, of the form https://company.tld/products.
If a user goes to homepage, and clicks on the "products" link in the navbar, the page content of /products route would load in the client side, via the framework.
But, if a user directly hits the /products URL from their browser (can be from history and omnibox prompt), the browser would first load the home page with assets, then the client side JS would take over, and route the user to the /products URI endpoint.
New server-side rendering paradigm gives you best of both worlds - get to maintain a single codebase to render your page (be it server or client), while give the initial fast loading, so that your page is interactive quite fast.
This is not easy to do. One big challenge for most server side rendering solution is that, after initial render, the JS in the page would try to re-render the page; thereby creating an impression of a "flash".
Yes, if you're using a framework like React or Vue, which uses VDOM, this won't happen in most cases, because of how a virtual DOM based renderer works. But there are still cases, where you might need to explicitly do something to prevent the re-render.
There are other challenges too. For instance, having the page server-rendered, and hydrating the state.
In the search of maintaining a single unifying codebase for both server and client (isomorphic rendering), we often do things that cannot work on server. For instance, chunk splitting your front-end bundle based on routes. Or CSS media queries.
To sum it all up, "server-side rendering" isn't just about initially rendering the page on server. It's so much more than that. It's mostly about how we can maintain a single codebase (now that we have JS on server too), and run on two different platforms.
I think this is the main reason Node caught up.
SSR and code sharing are only useful in full stack projects, not in APIs. I don't know the numbers, but my conjecture is that there are more RESTful APIs written in Node than any other type of server project.
It's progres bar was the bottleneck causing slow downs.
The size of node_modules directories is basically a meme now.
Although other languages are catching up they have to do it as add-ons or third party tools -- for example python pipenv https://www.kennethreitz.org/essays/announcing-pipenv
One is an implementation detail of a secondary part of its operation (progress indication), and the other is about packages having many dependencies, not about npm causing node_modules directories to grow itself.
Not a node user since many years, so I can only guess, but isn't the cause of this is bad community practices (micro-packaging, many different things that do the same thing, etc) rather than the package manager itself? You can get hundreds of deps in any language if the practices are similar.
Non-Blocking IO is still the best we have because its a limitation in syscall interfaces. Node.js, nginx, go, etc all use epoll and their ilk under the hood. In fact, in go goroutines only ran by default on a single OS thread until go 1.5. The ergonomics of call-back based non-blocking io is the issue here, not non-blocking io itself. Because of javascript's history on the browser where callback based events are the only apis it made sense to copy this style of api for IO so stuff like `setTimeout` worked in both cases. Node's selling point with regard to non-blocking io was never about ergonomics, it was the fact that you had a dynamic scripting language where the entire ecosystem was using non-blocking io using the same standardized event loop. Contrast to the other popular "web" languages of the day, Ruby and Python which have had some non-blocking io for years but usage is isolated because most of the ecosystem doesn't support it at all, and when they do they completely different and incompatible low-level io stacks.
A lot of tech knowledge isn't easily discoverable. Heck, sometimes things don't even make sense without understanding their historical background. The filesystem layout in the unixes is a great example, on my system I have: /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, /usr/local/sbin.
[1] http://lists.busybox.net/pipermail/busybox/2010-December/074...
The fact is that green threads aren't a panacea either, and exploring the callback based pattern led to some really interesting development, evidenced by the explosion of javascript on the server .
At the time that node took off, a huge amount of server code was python with the GIL, and there needed to be something to shake up the status quo.
JS on the server was the means, not the end.
If you're doing a green threads implementation for a high level language runtime, those restrictions don't apply, to an extent. I think e.g. Erlang, Haskell or Go are examples of "green threads done right".
(Now, I do think Rust did the right thing in getting rid of green threads, but that IMHO is more a result of the space Rust is in rather than a general indictment on the utility of green threads)
Green threads done right strikes me as something like Windows' user mode scheduling (or Google's switchto patch that sadly never made it upstream), in which threads really are 1:1 as far as the kernel is concerned, but manually scheduled by userspace. This requires kernel support, but it fixes every issue except preemption, which is better to just not fix. (Few Go programs actually benefit from multicore CPU scaling; for CPU bound tasks a better use of optimization time is just not writing in Go, which tends to result in better speedups than trying to scale Go due to Go's compiler being relatively immature.)
In my ideal world, kernel-scheduled and user-scheduled threads would exist in the same language, and programmers could choose which one they want on a thread-by-thread basis. I/O calls would be equally compatible with either threading model (which could be done in a zero-overhead fashion with the proper kernel support), eliminating the problem of sync/async incompatibility. This can only really be done today on Windows, unfortunately...
This was the case for Dahl and many others. It's definitely possible to catch up on the history of the field while you're working in it, but not everyone can find the time to do so.
To get CSP you needed green threading built into the runtime, and that's a much taller order.
So it's hard to say that the not-critizing-directly Dahl should have learned from the past. We'd probably never have heard of him if he'd set out to build a new esoteric CSP language or runtime, just like we haven't heard of anyone around that time other than the golang team.
Isn't that Go's problem, though — that there isn't a single thing in the language that wasn't entierly mapped out in the 1970s?
what learned lessons, like pragmas and struct tags?
As someone who is occasionally accused of being too pro-Go on HN... I actually have no problems thinking of Go as a really nice and refined 1990s language. Another type of "Java done right". (I say "another" because I think C# has a really good claim to that as well, albeit in a very different direction.) There's a place for that in the world. As much as I love the cutting edge of programming languages too, I'm not sure that we're anywhere near as far along on knowing how to build really big systems with them as a lot of people think we are. (There's only one way to get there though, and I absolutely encourage people to keep trying.)
Given the fact that C is basically the blueprint for Go, I'm not sure what things were taken away from C, syntactically speaking. It's more like Go added garbage collection to C, and a few other bells and whistles and told people : "this is 21th century programming", without actually thinking about what C got wrong at first place.
Pointer arithmetic. Go has "pointers" but they're really more references. Some other things depend on how you look at them; Go "removes" manual memory management from the language by adding GC, it "removes" the fact that in C values are essentially untyped and very easy to penetrate down to that untyped layer by making all values carry their type around regardless, they "remove" the limitations on function pointers to provide real closures, they "remove" trivial array overflow by adding checked access, there's a lot of rough edges removed from C at the language level even if it required "adding" some runtime support.
I mean, by the time you've removed pointer arithmetic and manual memory management, you are by no stretch of the imagination in C anymore.
What do you call new? generic programming, nullable types, type classes? all pioneered around 1973? Can you explain what "new" is?
And CSP was invented by mathematicians for themselves, not for real life programming. As opposed to Erlang, which was invented for real life programming of telephony applications later in 1980s. But people keep ignoring history and think that CSP is suitable somehow, and not actor model. CSP is an even bigger mistake, don't try to make it look like it isn't. Learn from history.
I'm not a huge fan of Goroutines spawning threads in the background. I think that spawning threads and processes should be explicit because there is a big performance penalty when multiple threads have to share a CPU core because of context switching.
With Node.js, you have more control over the process count so you can minimize the amount of CPU context switching that happens by making sure that each process gets its own CPU core.
That said, I do think that on a conceptual level, Go feels cleaner than Node.js... But since async/await was introduced I feel that Node.js has the upper hand again.
Plus, he said he was using Go before he joined Google.
The guy is obviously at genius levels because he is a really advanced programmer even though he didn't really get involved with it much later in his life.
I've been programming on and off my whole life, and I can barely understand 1/5 of the concepts he's talking about.
Reminds me of that scene from Silicon Valley "He's a 10x, and I am a 1x".
That's just some "god" given talents right there.
But yes, I generally agree success should not be predicated on what school you went to.
I was primarily pointing at the fact that he started in Community College, which is sometimes looked down upon.
So for Math and CS, yes, this mostly means mort of the Ivies and 10 or so other schools, including MIT and Stanford.
In general I'm of the opinion that there is a lot of "top school" inflation in the United States. It's better to consider schools on a major by major basis, especially at the graduate level, where your advisor might be more important than the school.
The reason I restrict the top to the top 10 (and top 20 for leniency) is because the admission standards are vastly different the first 10, then the next 10, then the next 10 - 20, and at that point there isn't a significant difference in "attainability" anymore. I term it this way because I think the word "top" is only useful for signaling, not for real qualitative comparison.
As a specific example: NYU is listed as a top 30 school for computer science, but while it's a good program, it seems odd to list it as a "top CS school" - is it really that much better than the next 10 or so, or is the term just diluted? Likewise, NYU is in the top 10 for mathematics, which actually seems sane from my perspective.
And in the top 25 schools for CS is Rice University - you need a new word to call the group that MIT, NYU and Rice are in, because "top" is no longer all that meaningful.
"I think Node is not the best system to build a massive server web. I would definitely use Go for that. And honestly, that's basically the reason why I left Node."
Go may be an excellent choice for massive non-web servers, I don't have enough experience in it to say. For the product I work on, though, Node.js is the way to go. It's the best framework that allows us to use the same exact code on the server and on the client to create a fast progressive site.
Go lets you do multi threaded non blocking.
That's a pretty substantial delta in the server world where you can easily have 24+ cores available.
Memory and cpu footprint of go is also drastically lower to node as it is a compiled language. That makes a difference when you are running in a cloud paying per GB of memory and per core.
I say this while actively developing in node.
Furthermore, not everyone is developing cloud-based apps and trying to optimize for metered cost structures.
If you need raw, parallelized, computational power, then I will without-a-doubt agree that Go is a better choice. However, to say that one would always use Go for any server (even webapps? come on...) truly doesn't make any sense.
The only other benefit-of-the-doubt explanation I could come up with for such an explanation is that it seems like the creator has spent more time in academia than in "the real world" of development, and perhaps the persuit of "the ultimate async system" is more important to him than practicality.
https://api.soundcloud.com/tracks/340298200/download?client_...
There are still a lot of codebases that are callback heavy. For one, callbacks are faster than promises in a lot of node versions by a very large margin. In other code bases, legacy rules still apply.
Promises don't work well for some more complicated logical flows (though they're pretty damn perfect for the common ones) and async is still pretty new.
Just because it's not a problem you deal with doesn't mean that it doesn't still affect plenty of devs.
New... in the 1970s https://en.wikipedia.org/wiki/Model–view–controller#History
It boggles the mind how little "web devs" know about the history of the field. No wonder they keep reinventing the wheel.