Things I Regret About Node.js [video]
youtube.com
youtube.com
It's very popular of course, so I'm definitely not arguing that metric. However, the stuff that was originally called examples of tooling that exhibits unneeded bloat and complexity (Maven) is now reimplemented in Javascript, but poorly (npm).
(Amusingly, my iPhone autocorrect replaced "roots" with "Russ". Russ Cox is an engineer who works on Go. :)
I'm not trying to say that Java has no accidental complexity of course, I don't want to open that can of worms :)
This is all just the process of evolution at play. What seems obvious today wasn't yesterday - applies to biology, material science, medicine, engineering, art, music, architecture, design, taxi services, marketing, government, politics, and so why shouldn't it be so in computing?
Sidenote: I love the humility of this video. I remember the days when node was first unleashed. I could not have imagined how it has changed the way we all work. It all seemed so obvious from day one, and here we are today. What a brilliant contribution.
We're definitely reinventing the wheel a lot, though.
Can you expand a bit more? Not sure what this means.
And the list goes on and on IMO. What's disappointing is that these were lessons learned a long time ago and now they're being re-learned.
But anyways, [2] is at least a problem in many other package repositories. [1] would probably be a problem for many - given legal pressure (vendor your shit, that's the solution). [3] was a bug, not a design issue - no package management system is immune to bugs.
The one thing Java has is that it uses namespaces, which may help with [2] (but barely). [2] certainly has been a problem in PyPI.
Certainly all of this could happen to PyPI. We see it happen with js more, I think, because js happens to be extremely popular so there's a ton of packages for it and it's also much younger (especially node) than others.
He does have it in his slides.
Slide titled: "Regret: package.json", last 2 points:
> Ultimately I included NPM in the Node distribution, which much made it the defacto standard.
> It's unfortunate that there is a centralized (privately controlled even) repository for modules.
Just trying to clarify that he does actually talk about NPM and his regret about it.
I know. [2] and [3] have nothing to do with a package repository you can’t delete things from.
There is something exciting about developers using a language in ways it was never designed. Then having the language change to support the changing ecosystem...
(as it happens 1.7 was released recently.)
A large majority of people you’re chiding for not learning from others, don’t even realize those other things exist.
The car industry probably wouldn't be as big, if you had to learn a new tool for every new car.
The reason this is a problem is because web tech is constantly changing, to the point that so many of these projects end up in the scrap heap far faster than other tech. It causes problems with long term service due to compatibility issues with ever changing dependencies.
And sometimes it’s just resume building or intellectual curiosity itching.
Here's a concrete example. Your classic node.js or express.js sample app is something fairly simple like a hello world, or an IM server. A more complex sample probably looks something like that venerable nodecellar app from a few years back. In all cases the spiel is, "Hey, look how easy it is to create a web server with node."
Except that I'm looking at my node server source right now - for an honestly fairly simple app containing a handful of pages and a blog - and here's what I have:
- Routing (obviously)
- Cookie and body parsing
- Session management
- MongoDB integration
- Passport.js for authentication with a couple of providers (FB and Twitter)
- File system access
- HTTPS and SPDY/HTTP2 support
- Compression support
- Logging with winston and morgan, including loggly integration
- Referer spam filtering
- Pug templates
- Hexo blog integration
- Path resolution support
- Request validation and redirects
- Static content support
- Stripping headers such as X-Powered-By, and adding other headers such as the all-important X-Clacks-Overhead
- Error handling
There's probably a couple of other items I missed, but you get the idea. It seems like a lot but, as far as I'm concerned, this is express.js app MVP for anything you might want to put into production.
I haven't even mentioned the gulpfile I use to build all this, which targets desktop, mobile, along with embedded versions for a particular mobile app due to launch in the near future, and has become something of a behemoth[1]. Nor have I mentioned that I have Cloudflare to sit in front of this, primarily to deal with the heavy lifting of some of the media files I serve up.
On the face of it, this might feel like "bloat" but it's all necessary to run the application and, like I say, a lot of it is the bare minimum for an MVP web app in node.
[1] Yes, I know I could/should switch to webpack, but gulp works, and switching to webpack "just because" doesn't justify itself with the value it might add.
It's a shame these tools keep being rewritten because there are definitely good ideas in all of them, but for some reason they can't seem to be unified.
Webpack pulls dependencies into one file and deduplicates them. This is obviously even more complex since now you have dependency resolution logic as well as dealing with the various module systems JavaScript has invented.
While Webpack is a little dense, it appears to strike the right balance between complexity and customizability (and probably more important for longevity, library buy-in). It doesn't seem like anything on the horizon is going to unseat it anytime soon... certainly not Brunch.
I don't know, I thought the same thing about Browserify.
And now Parcel is here, gaining steam...
I've been out of Node for like 6 months, wth happened! I give up!!
Without the ability to compose multiple small libraries to form the exact solution that we need, we had little choice but to rely on the One True Framework to solve every problem that we will have.
This means if the One True Framework doesn't serve the exact need you have (and it almost definitely won't, there's a combinatorial number of requirements out there), it's time for a rewrite!
The thing is, even with Linux distros, most of the stuff you want is built-in by the distro. Once you start to add your own stuff, it can get really ugly and you have to be really pro to get anything done. It seems like every time I'm working on updating a Linux image I have to do some really bizarre thing where the package manager doesn't even work right and the instructions or some forum have me doing some mind-blowing workarounds I don't even understand.
So I think you are combining two different topics. I am all-in for libraries over frameworks. But the larger, more heavily curated libraries where you only need minimal customization are just objectively better. Having a large, curated standard library != a framework.
When doing a project that takes only a few weeks, I would probably choose a framework that has everything in the box. But if you are building something that is going to be developed over a period of years, the reduction in complexity achieved by building your own can be life saving.
When I look at your list, most of the things fall into the categories of "Pretty easy to implement" or "Don't want at all". However, there is an advantage for not reinventing the wheel if there is no reason to do so. If there is a nice library that gives me what I want and doesn't impose itself too much on the design, I will use it. But the main advantage for not baking it into a big framework is that I can pick and choose what I want.
As an older programmer, I come from an era where libraries and frameworks cost a lot of money. We built stuff by hand because there were not a lot of other choices. These days, though, virtually every library and framework is free software (not only free of charge, but you get source code too!) It's like living in Candy Land, and I'm not about to complain about it :-) However, I think that programmers today reach too quickly for the pre-built and do not understand the long term advantages of bespoke development. Like most things, there is a balance to be maintained.
If your using 10% of the lib, just implement it yourself.
If the lib is critical, like openssl, bring it in. Other people have solved the hard problems for you.
But yeah, it's a balance.
I normally lose those debates though, and the thing reaches a point where the complexity of the code makes it impenetrable.
- Routing is done in two lines of code: @Controller @RequestMapping("/myroute")
- Cookie and body parsing - no need to write any code to do that, I just have method parameters and all of the data flies in. Whant a validation? Only one keyword on a method parameter - @Valid. Custom validators are supported as well.
- Session management. It just here for me and does the right thing by default. I can replace storages with custom implementations but by default no code is required from me.
- MongoDB integration - Spring Data MongoDB and you only need to define interfaces using naming convention. The code to access the actual database is generated for you.
- Spring Security supports multiple authentication mechanisms and gives you neat DSL to configure it.
- File system access kind obvious thing.
- HTTPS and HTTP2 support provided by Spring MVC as well.
- Compressions support - it just "server.compression.enabled=true" in your config
- Logging - slf4j + logback come with Spring Boot and there plenty custom appenders available to put you logs into logstash/splunk whatever
- Referrer spam filter - not sure about that one but CSRF protection comes OOB and enabled by default.
- Multiple tightly integrated template engines to chose from. Zero configuration code as well.
- Static content comes OOB and enabled by default, just put your stuff into resources/static.
I mean yeah, modern webapp is a complicated thing! So whenever I see somebody trying to do anything "not bloated" it means that I end up writing low level code that has been written multiple times again and again.
The other day I was trying to code a simple thing in Clojure because I love Lisp. Well, it's just embarrassing. I got to simple page showing stuff from Postgres and the boilerplate/business code ratio was at about 70%. Manually configure connection pool, manually start it, manually prepare your component systems, manually mention all of the dependencies for components, manually configure template engine, manually enable static resources support in ring, manually configure and enable session support in ring. Then we come to authentication and don't even try to sell me Friend. EVERYTHING is manual. The only good thing was "environ" which did the right thing but again with "bloated" Spring Boot it comes OOB and I don't need to configure it!
If you don't use something "bloated" it only means that you're writing code yourself, again and again.
To be more concrete, to the original spring bean scanner, we were passing in a set of package names, which it would scan. Spring registers those bran defininitions in yhe order that it scans the beans. My custom scanner (found and registered all the same beans), broke our app because it wouldn't startup anymore due to a circular dependency error. Once I sorted the bean definitions by the original pack path inputs, that startup error went away.
I think we are 4.x
Extra details: I used the fast -classpath-scanner library. I subclassed the annotation cand date componend scanner class (well, something like that), and rewrote a method to load the resources for the string specified, treating the path a a fqcn, not a package path. Then I could feed that class the output from the fast classpath scanner (which was the list of classes with the annotations). Until I sorted the input by the original package paths, my app wouldn't start. Mind you, the method I overwrote simply created bean definitions. But that ordering difference made all the difference.
I can dig up exact class names if you are curious. The scanner of course didn't replicate all spring bean sear check capabilities - just the ones we were using. But it cut the scan time by 60% (several seconds).
It is opinionated and provides libraries and solutions for almost everything you need to do, BUT it always allows you to use your own if required.
I love Spring Boot and I wish there was something even remotely as good and full featured in other languages.
Even a 'simple' web app is a convoluted mess of shit if you are to run a real world production grade system. I'm so sick of all these 'hello world' toy examples.
I'm lucky enough to have been around the industry for a while. I could probably count a dozen or more things that started out "Like X, only without all the BS!" --- only to end up with just as much BS or more than X ever had.
Reduced complexity is often a rallying cry, but I think the root of the phenomenon is in trying to find one's own social and professional standing in the situation where all the prominent positions are already taken and what little is left requires years of hard labor (complexity, certification, corporate review system, etc).
If this situation upsets you consider the alternatives, they might be worse.
Tech is ripe for applied group psychology and anthropology. The social, psychological, and anthropological factors are obvious to casual observers -- but completely invisible to the people they affect the most.
There's a reason for it, and perhaps overall it's a good thing....but that still doesn't mean that it can't be accepted and acknowledged as a facet of the community.
Currently popular music mostly sounds like noise to me, but that's not the point. What are the current generation of musicians supposed to do, be silent and spend their lives listening to the great bands of my youth? It's impossible to match Pink Floyd in the style of Pink Floyd. They need a new style.
Facebook is losing traffic to whatever is the latest trend in social media, not really because people are suddenly paranoid about privacy, but because each generation needs a network where the previous generation is not.
And for as long as humans write programs, there will be a need to invent new languages, not because the old languages were technically inadequate, but because each generation of programmers needs a way to escape the shadow of the previous generation, the way acorns need squirrels to carry them away from the shadow of oak trees.
OTOH, the young always have a fresh perspective and they usually have good ideas based on the times. They should be listened to and mentored. Very few active older (1995+) folks left in IT ,after the method management and purposeful purging methods of the last 10 years, to mentor them. Most of us weren't great at teaching anyway. It was a paycheck.
It's interest to see which parts of our civilization came which side of the divide. Market economy, for example is a great way for a young enterprising person to find their own footing away from the old (hence startups). Academia OTOH went totally the other way (hence grad school).
I try to explain why we do things the way we do and if they still want to try to change things, I make sure it's easy to go back again in case it fails the usual way.
1. X is SO bloated and poorly engineered full of bad legacy decisions.
2. We can totally do better let's invent a new thing, Y!
3. Wow, Y is so clean and fast and understandable.
4. But it doesn't do this thing a bunch of people reasonably really need... let's add it.
(repeat 3 and 4 a few hundred times)
5. Y is so bloated and poorly engineered and full of legacy decisions. We can do better! (Go to 1.)
The grass is always greener, but mature complicated software is _usually_ complicated for... reasons.
Rebuild from scratch but also recreate all the existing functionality in a much better standard library and finally the chain can be broken. But nobody wants to do that.
There are some challenges coming up with design changes to the compiler and C# that might overlap what F# already has but it'll get sorted out.
Great resources for getting started with F# at https://fsharp.org/
My personal preference is generally to install the SDK and use the http://ionide.io/ with VScode as it seems to work most reliably cross platform.
I have .NET Core but the whole thing seems to require Mono and it isn't clear from fsharp.org that you can do without.
MVC5 was never released though, and the changes have been rather minimal from ASP.NET Core v1 to v2 with straightforward migration guides, so it might look messier than it actually is if you were working through all the previews and release candidates instead.
Nevertheless, Microsoft has a long history of having messy v1.0 with most of the stability coming after v2.0, so you can consider the foundation pretty stable now that it's on v2.1 and more.
TypeScript and JS underneath is actually quite malleable - you can escape static typing at any point and revert to simple JS object model when things don't map cleanly in the type system - and then still have types at the boundaries - makes meta-programming trivial in some cases - where it would look like a monstrosity in C#.
F# is interesting and has a lot of advantages over C#, but few people seem to be willing to invest the time to pick it up in the .NET community.
So I don't really view .NET core as a superior alternative, I've worked in JVM land, they are more mature and while Java sucks there are other languages on top of it as well and are decent to use (Kotlin ~ C#, Scala ~ F#)
o_0. Think you need to check yourself mate.
I believe the productiveness of more "expressive" language tends to be undermined by the loss of productivity that occurs when you're compelled to write blog posts or comment on hacker news about how amazingly productive and expressive your language is.
If I need to waste that time sifting trough boilerplate than I'm pretty upset because I get less shit done in that time window.
Chatting on forums is a casual brain teaser and keeping up to date on industry stuff.
> [I find that] using a language with limited expressiveness (C#) is not very productive for me.
Like, I'd figure you can be mad productive in any language (even COBOL?) although I'm only completely cosy in a couple. There's no need to be so dismissive of the tools that others use.
Why is C# not expressive? It has the DLR and `dynamic` keyword which behaves just like JS typing if that's what you want, because it seems like your issue is really with static typing in general. Functional languages are nice but it seems C# with functional its slowly and carefully integrated functional extensions is actually more productive for most developers.
Think about AutoMapper and then compare it to a TS solution using spread operator. How much boilerplate automapper crap do you see in your typical enterprise C# project ?
And that's not even touching on functional features, like you can't even have top level functions in C#, it's "one class per file dogma" + multiple wasted boilperplate lines and scrolling. I recently rewrote a C# program to F# - didn't even modify much in terms of semantics (OK having discriminated unions and pattern matching was a huge win in one case), just by using higher level operators and grouping stuff - line count went down to 1/3 and was grouped in to logical modules. I could read one module as a unit and understand it in it's context instead of having to browse 20 definition files with random 5 line type definitions. I could achieve similar improvements by rewriting to TS or Python.
C# adds overhead all over the place, people are just so used to it they don't even see it as useless overhead but as inherent problems they need solve - like how many of the popular enterprise patterns are workarounds around language limitations ?
When I bring this up people just assume I'm lazy about writing code - but I don't really care about writing the code out - tools mostly generate the boilerplate anyway. Having to read trough that noise is such productivity drain because instead of focusing on the issue at hand I'm focusing on filtering out the bloat from the codebase.
I could rewrite your entire comment in reverse about how I find C# highly expressive and readable while dynamic languages or Kotlin (blech) are a mess of inconsistent whack-a-doodle experimentation.
But my opinion is useless.
The value in any platform is productivity and if any given team can be productive, it doesn't matter if it's COBOL, RPG-3, Pascal, BASIC, or a functional language like F# or plain old JavaScript.
It's more that C# is static typing done poorly IMO - a relatively limited type system that adds overhead compared to dynamic languages or more expressive static languages.
I agree it makes JS better. I agree it's a good tool for its purpose.
But "fascinating" ?
It's hardly the most elegant scripting language down there (Ruby, Python, Kotlin and Dart doesn't have to live with the JS legacy cruft).
It has a very small ecosystem outside of the web.
The syntax is quite verbose for scripting.
It has very few data structures (and an all-in-one one).
Very poor stdlib.
Still inherits of important JS warts like a schizophrenic "this".
Almost no runtime support if you don't transpile it (which means hard to debug and need specific tooling to build).
And it's by no mean the only scripting language having good support for typing (e.g: VSCode has great support for Python, including intellisens and type checking).
What's so fascinating about ?
What's fascinates me is that we are still stuck with a monopoly on JS for the most important platform in the world.
The typing system is what is special though, especially in how seamless it is in adding strict types alongside pure dynamic objects, but also allowing you to choose pretty much anything in the middle of that spectrum depending on your definitions.
You can have a few strong-typed properties mixed with others in a generic type that inherits from something else but can only take a few certain shapes. It's unlikely you need all that in most programs but it's the fact that you can do it which makes it great. In fact, the Typescript type system is actually turing complete.
Perhaps this video on Typescript from Build 2018 would help: https://www.youtube.com/watch?v=hDACN-BGvI8
That's pretty much my point.
> he typing system is what is special though, especially in how seamless it is in adding strict types alongside pure dynamic objects, but also allowing you to choose pretty much anything in the middle of that spectrum depending on your definitions.
> You can have a few strong-typed properties mixed with others in a generic type that inherits from something else but can only take a few certain shapes. It's unlikely you need all that in most programs but it's the fact that you can do it which makes it great. In fact, the Typescript type system is actually turing complete.
Apparently you haven't read my comment because I clearly says it's not special. Others languages do it to.
> Perhaps this video on Typescript from Build 2018 would help: https://www.youtube.com/watch?v=hDACN-BGvI8
Perhaps this article would help: https://www.bernat.tech/the-state-of-type-hints-in-python/
All services I've deployed built on rust pulls in a kitchen sink of deps.
Granted. I get a static binary as my end result, so maybe it's fine.
I think we had a generation of ecosystems with Node, Ruby, Python, that tried to do solve the unapproachable systems around the Java/etc ecosystems and make them more open.
They succeeded, but the next generation seems to have been about solving the plethora of tools that came with those languages. Rust, Go, etc, having first-party tools are trying to improve upon that, and yes I think Rust is by far the best implementation I've seen.
I'm interested to see what the next generation is.
I think Javascript suffers from this quite a bit. ES6 "classes" should never have made it in, for example. Not only did they add an extra level of abstraction for beginners to learn, but the only reason for doing so was "my code doesn't look like it does in other languages"...
As much as I beat the FP drum these days at work, I find the class syntax a much nicer way of organizing solutions to certain, pardon the pun, classes of problems.
Whether or not you find this to be semantic diabetes is a matter of taste, I suppose. I'm curious what, specifically, you find to be the major issue that makes you say they should have been left out.
ES6+ flavors of JS & Typescript really made me take web programming seriously again.
Something about components just fits the class model well.
The main issue with adding classes is that they're very, very complex if you want to make them useful. The initial version was pretty harmless, but it was also almost pointless. Now they need to back fill all of the missing features (eg: private fields), which brings in an enormous amount of complexity. Most of the time, if I need private fields, I can just use symbols (not quite private, but close), or I can do
function() { let private = 123; return { // use private in functions here } }
It adds (there ARE things classes are better add) very little compared to the insane amount of work that has to be put in the language to get it all working. Decorators are in a similar boat, where many decorator usages can be expressed just as easily with a higher order function, so adding the extra syntax is just bloat.
The cost isn't worth the reward.
The biggest thing classes give us that is very difficult to replace in vanilla javascript is a semantic construct that is easy to statically analyzed. In the typed flavors (Flow, TypeScript), I can analyze the interface of plain objects, but not in vanilla JS. Part of why React using ES6 classes can be useful.
Thats a great benefit, but Im not sure it's worth the trouble.
That's a really good point. I was about to disagree with you but then I created a thought experiment.
Thought Experiment:
I wonder what the JS landscape would look like if ES6 Modules were introduced as part of ES5 about 8 years ago? I could definitely see how that would make classes fare less appealing if we already had a great module system (sure CJS existed but browser didn't support it).
Looking at the timeline of when these features were implemented in all major browsers:
* ES6 Class[0]: implemented 2.5 years ago * ES6 Modules[1]: implemented 1 month ago
[0]: https://caniuse.com/#feat=es6-class [1]: https://caniuse.com/#feat=es6-module
The rise of Java and the OOP revolution isn't that far behind us (2 decades seems like a lot in the tech world, but its still within a single generation of humans).
I've found that developing JavaScript in an IDE that reads and validates type information from JSDoc allows me to introduce strong typing while maintaining the flexibility and simplicity of vanilla JavaScript without getting as bogged down as I get with Typescript.
You can do almost everything with jsdoc comments in flow and TS, of course. It's awesome.
Well, I'm really happy that the class-statement got in ES6 though.
Before that, whenever I needed something class-like, I had to search how do you do this in Javascript, and find five different answers, no but really which is the proper one for prototype-based inheritance, waste an hour (or more) and frankly I still don't know, there's just so many ways you can do it, which is the RIGHT one? and Javascript wraps in on itself in so many cool ways, but there were never any definitive answers, just more rabbit holes.
Now, there is the class-statement, and it's one less rabbit hole to get trapped in. I actually get more stuff done now that there is one right way to define a class, or class-like object with a constructor, properties, methods, etc.
Similar thing goes for the function arrow notation. Javascript, and the event-based environments it usually operates in, wants you to use anonymous functions a lot. But the relatively verbose way to define them, still held me back from using them freely as much as I wanted, trying to "optimize" them away if possible.
I've used an excellent one. The Refactoring Browser parser engine in Smalltalk. I've used it to eliminate 2500 of 5000 lambdas used in an in-house ORM with zero errors -- all in a single change request. (Programmers were putting business logic into those lambdas.) Like any power tool, it's not stupid proof. However, it gives you the full syntactic descriptive power of the language. So if you can distinguish a code rewrite with 100% confidence according to syntactic rules, then you can automate that rewrite with 100% confidence.
Here's where it can go wrong: If your language is too large and complicated, there the probability you can run into a corner case that will trip you up. Also, it will always be possible for a given codebase to create something which is too hard to distinguish, even at runtime. (You can embed arbitrary code in a Refactoring Browser Rewrite transformation, so you can even make runtime determinations.)
"Bulletproof" isn't "invulnerable." A vest with an AR500 plate will stop certain bullets hitting you in certain places. It won't protect you from being stabbed in the eye or stepping on a landmine. Despite that, it is still a useful tool.
Think about Java, it solved a class of problems that C was unable to address (e.g. unsafe memory, native threads). Thus enabling a new class of programs. But the new class of programs created opportunities for new platforms to solve with the benefit of a clean slate and fresh design having learned from past successes and failures.
There's a disturbingly low-level of historical knowledge passed along in programming. Some bits and pieces are encountered in a quality Computer Science curriculum, but usually in rarefied, theoretical form, and inevitably balkanized into drips and drabs as part of subject-oriented coursework.
New platforms bring exciting and meaningful evolution often at the cost of what techs like .net and Java have a few decade advantage in. It's also interesting to see what Java devs are innovating with themselves, Scala, Kotlin both have good things happening.
Maybe using one large, inter-syntax friendly world like JVM will help.
When experience is overlooked for youth, we relearn and reimplement the same libraries repeatedly in every new tech to feed some developers needs to build temples to their greatness.
Still, Fitzgeralds quote comes to mind... "So we beat on, boats against the current, borne back ceaselessly into the past." and technology is held back by reinventing the wheel.
That hole I can credit as giving C# the advantage in that tight niche, and stilling the development of the JVM platform in general.
By the time that the rust on JVM improvements were dusted off, all initiative was lost. Java was playing catchup to the competition.
IBM gave up on the first counter proposal, Red-Hat and Google didn't bother to rescue Sun.
So we might even have been left with either Java 6 or being forced to port our applications.
Neovim is an effort to modernize and remove cruft from vim, so they get to keep all the good parts and throw out the backward compatibility. If it works out it can eventually replace vim, not to different to what vim did to vi.
I'd like to see similar stuff done to much of the GNU tools. Make for instance has to worry about backward compatibility and posix compliance that makes it hard to progress. As of today there have been about 12,000 attempts to replace it with something else and I find all of them inferior for one reason or another, they've all reinvented the wheel poorly. If someone had taken the fork and modernize approach we might have something better by now.
It doesn't even have to be a "hostile" fork. The same can be done by the developers of the existing tools.
Loop, as it might seem, doesn't mean there is no progress made in between.
Of course, use cases will still evolve, and your initial understanding is always flawed, there's no magic bullet, designing general purpose software (or a language or platform!) meant to hit a wide swath of use cases flexibly is _hard_.
And then, yeah, like others have said, you need skilled, experienced, and strong leadership. You need someone (or a small group of people) who can say 'no' or 'not yet' or 'not like this' to features -- but also who can accurately assess what features _do_ need to be there to make the thing effective. And architects/designers who can design the lower levels of abstraction solidly to allow expected use cases to be built on top of them cleanly.
But yeah, no magic bullet, it's just _hard_.
As developer-consumers of software, we have to realize that _maturity_ is something to be valued, and a trade-off for immature but "clean" is not always the right trade-off -- and not to assume that the immature new shiny "clean" thing will _necessarily_ evolve to do everything you want and be able to stay that way. (On the other hand, just cause something is _old_ doesn't _always_ mean it's actually _mature_ or effective. Some old things do need to be put down). But resist "grass is always greener" evaluation that focuses on what you _don't_ like about the original thing (it's the pain points that come to our attention), forgetting to take into account what it is doing for you effectively.
For awhile, "Burn the Diskpacks!" was a battle cry of the Squeak Smalltalk community. That sort of policy fights bloat, but leaves old users in the lurch. I think that we are now to the point where a language/environment can trim bloat while not abandoning old users. If the language has enough annotation, and has the infrastructure for powerful syntactic transformation tools, then basic library revisions can be accompanied by automated source rewriting tools. We were pretty close to it in Smalltalk, without the annotations.
To be fair, it's not impossible for some improvement to occur in this process.
> originally node.js was presented as a bloat-free alternative to "enterprise languages"
?
That's when they go instead with the newer system, that didn't exist long enough to have accumulated criticism. Which is backed by enthusiasts still in the honeymoon period.
They owe their success to these people and so the way that they can pay it back is by using their voice as a tool for improving things.
I'm also a long-time user of Dart, so when he brought that up, and compared TypeScript to its shortcomings, I definitely agreed.
That being said, even with the Deno project, I'm not so sure what can come in terms of performance and security from running JavaScript outside of a browser. The choice of V8 also raises concerns for me about build systems. He mentioned the failing of GYP, but anything using the same build toolchain as Chromium always introduces a wealth of complexity for anyone building said software, as not only do you need Google-specific build tools, but also very specific configuration including Python.
It will be interesting to see what comes in the future.
If it were up to me (which I guess it isn't), I'd probably prioritize portability/less complex builds, built-in FFI, a flat module system, and optimizing calls to C from JS.
I'm just now building a node app to filter point clouds, so lots of number crunching. In two days I've got something in javascript that's faster than the C++ solution I've been working on for a week. Mostly because javascript/node makes it trivial to parallelize file IO while doing work in the main thread. This app reads 13 million points from 1400 files (~200mb), filters all points within a clip region, and writes the 12 million resulting points in a 300MB file, all in 1.6 seconds. (File reads were cached by OS and/or disk due to repeated testing, but file writes probably not)
My personal conclussion is that javascript can rival or even exceed the performance of C++, not because it's inherently faster, it's obviously not, but because it makes it much easier to write fast code. For the highest possible performance you'll defenitely want to use C++ but sometimes you'll have to spend multiple times the work to get there.
For example; Typescript is fast to write. and Golang is reasonably quick to write, _and_ execute. Both should have ~15% less bugs than javascript, potentially making them faster to develop (where bugs matter).
https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-q...
If you aren't context switching between your backend code and your frontend code (even when both are JS), you're probably incurring technical debt in your architecture to be paid in even greater numbers of dev hours down the road.
When you are writing an all-JS full-stack app, do you really feel like you're only working on a single app, as opposed to two different apps which happen to share the same repo?
Is it? I was working with a system where server is written in Erlang and client (and another server) is written in Python. No problems with switching back and forth.
> Yes, that's a cost your brain is paying.
What cost? I said I haven't noticed any.
> new devs [...] then need to learn and understand both Python and Elixir.
No, they don't need to learn even a speck of Elixir.
--
What you described is a trusim that one needs to learn two languages to write in two languages. Yes, this is obviously true. What I'd like to hear is an argument that switching between them used in a single system is costly, because I haven't observed that. This is what I discuss with, not with that learning another language has its cost.
There are more things you have to remember. Workflows in both languages. Of course it's more stuff, thus more context. And you have to use both languages constantly to stay fresh in them. The syntax isn't the only problem, just the easiest one.
Is it just as easy to maintain Spanish and English skills than just English?
"I don't notice it" isn't a very strong argument. I bet you don't notice the effects of slight dehydration and your diet and exercise on your output either. But if you were actually experimenting with it, I guarantee you could soon perceive it.
How wouldn't there be a cost of switching between two languages? Normally. You could try, you'd know then. Though the prerequisite is a system that is designed, with clearly designated borders between the parts, not a system that has just emerged.
Proving something's non-existence is a little like proving that you're not a weapons smuggler. How would you expect to even start?
> There are more things you have to remember. Workflows in both languages. Of course it's more stuff, thus more context.
But this is irrelevant to switching between the languages. You have just as much to remember if you write unrelated things, each in its own language.
> Is it just as easy to maintain Spanish and English skills than just English?
"Just as easy than"? Really? In a thread about languages?
You picked wrong analogy. It is just as easy to write prose with every second paragraph in English and Spanish as it would be with just English. The prerequisite, obviously, is that you know both languages.
> And you have to use both languages constantly to stay fresh in them.
For some value of "constantly". It's not like people forget everything about a language when they don't use it for a week or a month.
> "I don't notice it" isn't a very strong argument.
Well, at least it's some argument. On your side is only "how wouldn't there be a cost?", clearly from a position of somebody who doesn't use many languages.
Unless you flush the pages manually, your dirty pages (written files) live on long after your process died. Depending on system and configuration, minutes or even hours can pass before they are flushed to disk.
On writing, the double coordinate values are transformed back into a fixed-precision integer format and stored in the output Buffer object.
I'm not generating an intermediate buffer since that does decrease performance a bit. It's directly from input buffer to output buffer. Output buffer is initially allocated with the same size as the input, and before sending it to the output stream it's cut to the actual size.
One thing I've previously learned and which has shown to be still true is that writing individual bytes to a buffer is faster than writing integers.
So originally I did this:
outBuffer.writeInt32LE(ux, outOffset + 0);
// do once
let tmpBuffer = new ArrayBuffer(4);
let tmpUint32 = new Uint32Array(tmpBuffer);
let tmpUint8 = new Uint8Array(tmpBuffer);
// do many times
tmpUint32[0] = ux;
outBuffer[outOffset + 0] = tmpUint8[0];
outBuffer[outOffset + 1] = tmpUint8[1];
outBuffer[outOffset + 2] = tmpUint8[2];
outBuffer[outOffset + 3] = tmpUint8[3];Likewise, you don't need to copy bytes in your last code sample. You can copy uint32s.
Edit: Also, the stride from one record to the next can be anything, e.g. 15. An Uint32Array would need a stride of 4 or a multiple of 4 to be useful.
Edit2: I could try to create 4 Uint32Arrays with byteOffsets 0 to 3, and with a view length of a multiple of 4, then use the one that works with the attribute I'm currently processing. Not sure if that's really going to be faster but who knows.
Also simplest HTTP request in express is handled in several ms and that's A LOT in my opinion.
Points of interest:
Schedule files to be loaded. This will usually load around 1000 files: https://github.com/potree/PotreeServer/blob/redo/src/Regions...
Whenever a file is loaded, iterate through the points in it, do the filtering and write the results to the output file: https://github.com/potree/PotreeServer/blob/redo/src/Regions...
Wait until all files were loaded and processed: https://github.com/potree/PotreeServer/blob/redo/src/Regions...
Wait until the output was written to disk: https://github.com/potree/PotreeServer/blob/redo/src/Regions...
What happens is that ~1000 files will be loaded in the background, points are filtered in the main thread and even while some files are still being loaded, we already start writing the results to the output file.
> I tried to process simple CSV files in a very straight forward (but async) way and got reading 200mb CSV and just splitting it to columns (with simple split by comma) takes ~10 seconds.
CSV may be much bigger challenge since it's ASCII data. That always tends to take multiple times longer.
Not to take anything away from Ryan Dahl's ability to reflect and be open about what he considers his design mistakes, but it probably also helps a bit that he walked away for quite a while before coming back. A bit of distance helps in these matters.
edit: forgot Scala's SBT, admittedly a builder using Maven repo's but still an excellent example of how bad UX in this area can get.
I thought Ryan did a great job of explaining his regrets without giving the impression that Node was a "mistake", is "inferior", or anything so drastic.
They're ill-informed. GPP is correct that, for example pip is fundamentally inferior to npm [1], and those that insist on throwing shade at npm on HN should be corrected. They're wrong, and insulting a sound, well maintained project, without basis.
Preferrably by giving them better ammunition, since I do see NPM as substandard in quite a few ways, which is inexcusable when there do exist examples to learn from (whether it be a positive or negative influence).
First, it helps to clarify whether we are talking about npm the client or NPM the repository and ecosystem. Client issues are generally easily resolved, just use a different client. For npm, this could be yarn. For cpan, this could be cpanm, or cpanplus, etc.
If it's indeed the repository we are talking about, there are some obvious things that could be done to greatly improve it the NPM module ecosystem. For example, how about automating module tests against different versions of Node to determine whether it's in a good running status now for the current and prior interpreter versions, on the platforms it can be run on? [1] How about a prior version, in case you're trying to figure out if the version you're on has a known problem on the platform combo you're running on? [2] Or perhaps you want to know what the documentation and module structure looked like for a module a long time ago, like20 published versions and over a decade ago, because sometimes you run across old code? [3] Or as an author, the ability to upload a version, even for testing, and getting an automated report a couple days later about how well it runs on that entire version/architecture matrix with any problems you might want to look into?
In case you didn't notice the trend, I'm talking about CPAN here, which has been in existence for over two decades, and many of the features I've noted have been around for at least half that time. All in and for a language that most JS devs probably think isn't in use anymore, and on encountering a professional Perl developer would probably think they just encountered a unicorn or dinosaur.
Sure, NPM isn't all that bad compared to some of the examples that were put forth, but the problem is that those examples are a limited subset of what exists. Given the current popularity of JS and the massive corporate interest and sponsership, I frankly find the current situation somewhat disgusting. The only thing keeping JS from having an amazing module ecosystem is ambition. Sure, NPM might be a sound, well maintained project (points I think are debatable), but it could be so much more, and that's what we should be talking about, not almost annual fuckups[4] they seem content with dealing with.
1: http://matrix.cpantesters.org/?dist=DBIx-Class+0.082841
2: http://matrix.cpantesters.org/?dist=DBIx-Class+0.08271
3: https://metacpan.org/pod/release/MSTROUT/DBIx-Class-0.08000/...
4: https://hn.algolia.com/?query=npm&sort=byPopularity&prefix=f...
That was all I was responding to.
I definitely learned some cool stuff from your comment, and appreciate that, but my point was simply that the all the drive-by FUD that npm gets on HN is unwarranted.
> I frankly find the current situation somewhat disgusting.
This feels so hyperbolic though. The things you mention are cool 'nice-to-haves', to say not having them is 'disgusting' is a huge stretch in my opinion.
What I find somewhat disgusting is the massive amount of mistakes they've made over the years, and the time they've had to take to fix them, that could have been mitigated or entirely avoided by surveying best practices from other package management systems that have gone through the same pains.
2018-05-28 - ERR! 418 I'm a teapot (this is not a joke)
https://github.com/npm/npm/issues/20791
https://news.ycombinator.com/item?id=17175960
2018-02-21 - Critical Linux filesystem permissions are being changed by latest version
https://github.com/npm/npm/issues/19883
https://news.ycombinator.com/item?id=16435305
2017-08-01 - Typosquatting package names
https://twitter.com/o_cee/status/892306836199800836
https://news.ycombinator.com/item?id=14905675
(a little obtuse, but moderated package namespaces with
trusted maintainers can mitigate this, and spread load
from levenshtein distance checks.)
2017-11-03 - Visual Studio Code 1.7 overloaded npmjs.org, release reverted
https://news.ycombinator.com/item?id=12860806
(10% increase in NPM load, specifically to 404 pages,
causes NPM to fall over due to naive 404 handling and
apparently, poor ability to scale. Good thing they
caught it at 10% instead of the 200% it would have
reached...).
2016-03-29 - changes to npm’s unpublish policy
https://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy
https://news.ycombinator.com/item?id=11382885
2014-02-28 - npm’s Self-Signed Certificate is No More
https://blog.npmjs.org/post/78085451721/npms-self-signed-certificate-is-no-more
https://news.ycombinator.com/item?id=7320833
2012-03-08 - npm (Node's package manager) leaks all user password hashes and salts
https://gist.github.com/jashkenas/2001456
https://news.ycombinator.com/item?id=3679996
I mean, I would cut them a little slack if they seemed to have plans for making stuff better and a roadmap and it was just a matter of time, effort and resources they were lacking, but it seems to continuously be a case of them waiting until the shit hits the fan and they're forced to first take a look and see how to fix this new problem they've never envisioned, and then figure out their solution. Sure, it can sound hyperbolic initially, but I think that's just because people haven't really stopped to take stock of what's really going on here, and how it's not really getting better in any useful way. In the midst of emergency fixes is not how you should plan your new features. :/
As with much of programming language design and implementation over the last 3+ decades?
below> that could have been mitigated or entirely avoided by surveying best practices
Yes, people who would spend their time working on language designs and implementations, should at least be familiar with the many surveys of best practices. Surveys of repos, type systems, memory layout, parallelism, and so much else. Language choices are intertwined and subtle, and adhocery has enormous downstream costs to the field and to society. The programming language design and implementation wiki exists for that reason. To accessibly distill our collective experience. Not using these resources is negligent - a disregard of our profession's responsibilities to society.
Oh, wait. Our field can't be bothered to create surveys of best practices. Or a wiki. Knowledge is inaccessibly dispersed among balkanized and siloed human communities, assorted academic papers, and scattered code.
Shall we continue to blame pervasive failure on individual language developers and tools? For how many more decades? At what point do we start addressing it as a systemic problem?
:) So I agree with your observation, but suggest the problem extends far beyond package management systems.
I'd put it par with rubygems, ahead of pip, gradle, maven, a little bit behind mix, and far behind cargo. Not a bad spot to be by any means.
For the purposes of this discussion, it is useful to note that cargo was written by Yehuda Katz (wycats), who had previously written Bundler, and so actually had some concept of what mistakes he had made before and experience specifically in this area, in order to apparently (I haven't used it yet, but I have heard lots of good things) finally have built something truly great.
* The maintainers have pushed several breaking updates by mistake (I'm a teapot recently).
* There have been a few cascading failures due to the ecosystem (leftpad).
* node-gyp (alluded to in the talk) break cryptically on install in different operating system/package combinations. It also obscures the actual package contents.
* The lack of package signatures and things like namespace squatting significantly hurt the overall security of npm.
And let's not forget how terrible things were pre-yarn with the nested folder structure of node_modules and no lock file.
Compare that to NuGet where I've literally never had any of these problems.
Exactly reproduce a build at a later date
Part of it is technological (npm didn't have package-lock.json until very recently), part of it is organizational (the npm repository is surprisingly fluid), and part of it is cultural (the JS community likes zillions of tiny constantly-changing libraries). The net result is that I cannot walk away from a JS build for three weeks without something breaking. It breaks all the time. UGH.
It's still my favorite tech talk, very fun to watch.
Then a few minutes later says “I thought being able to specify URLs in import statements would be cute..”
Uhh...Houston, we have a problem with this one.
It sounds like he doesn't want me runtime to handle dependencies at all.
You can always reference urls with version numbers present in them.
I could go on but you get the idea. A package manager is not simple and requires A LOT of choices. The best one I’ve seen is old CPAN.
Obviously it's not what you should do if you are publishing a library for others to use, but for local use, and the kinds of exploratory scientific computing he was talking about, it sounds perfect.
* Much-beloved TypeScript
* Much-beloved VSCode
* Much-beloved GitHub
If they hire Ryan to flesh out his vision for deno we'd probably need a new acronym, MAFANG
On a more serious note, I wonder if deno could support a lower level construct like Observables. As much as Promises are perceived to be an improvement over callbacks, they still have major flaws (only one of which is mentioned in this talk), this is something that Observables can address
Just leaving it here, given Azure Sphere and WSL.
My suggestion is just the distribution I would expect Microsoft to acquire, if they would decide to go shopping instead of pursuing their own.
[0] https://www.theregister.co.uk/2001/06/02/ballmer_linux_is_a_...
I totally recognise the IO problem in our connected world and NodeJS really does solve the problem around the "many simultaneously persistent connections", something that would be really hard to do without something like NodeJS. In essence (and in my humble opinion) NodeJS is basically a programmable socket server and it's main feature is "websockets".
Writing software applications in NodeJS is the most awkward experience due to it's async nature. Business logic is inherently sync, not async. Most of NodeJS's existence has been trying to find an elegant solution to make this async behaviour look like it's sync, from callbacks, to promises and now actual language features added to JavaScript itself (async/await).
The problem with Javascript on the server is not Javascript, but the runtime in which it's executed.
I think it would be interesting to have an sync version of NodeJS that acts more like traditional Ruby and Python next to the async variant that we currently have. Both types could then be used along side each other, each solving the problem for which it is best suited.
Even little things like "make these two database requests in parallel and wait on them both" or "process these urls but only have 8 requests in flight at a time."
What's the state of the art solution to this in node.js?
const results = await Promise.map(urls, url => process(url), { concurrency: 8 })
For example, off the top of my head this would probably take me 40 lines of ungeneral wait group code in Go.
The first thing I did for a latest project in node was a sync =>{done()} queue for ws requests. There is a room for fine-graining, but it is a routinely job of determining parallel cases that don’t hit db “ci” in acid much. Not a big deal. The choice for node was obvious in that it has most straightforward installation and maintaining on windows boxes (I’m language and OS agnostic [except js allergy], but peers aren’t). I think that async is yet another hipster bullet, since our to-profit businesses rarely need much traffic and those that need are barely fit for 10kloc of js.
It’s awkward personally not so because it’s async, but because it’s too low level and npm doesn’t feel like a well-documented and thorough module set. It lacks unixlike formality and strictness in phrasing, oriented on beginners, but not on -pedantic needs. It’s hackerish and cool until you meet borders that are required beyond the “move fast” thing — it slows you down. And I didn’t meet a good abstraction over ws and db layer yet. It feels like a bicycle and a shovel when you have to dig a career (pun intended).
What part of it is really hard to do without NodeJS? You can do this easily in Go, Elixir, C++, Rust, etc.
In other platforms developers need discipline to choose non-blocking APIs over blocking ones, or to wrap their blocking calls in async execution contexts like threads or coroutines.
Here's a nice WebSocket server in C++:
https://github.com/uNetworking/uWebSockets/blob/master/READM...
Of the languages mentioned I believe Go is actually closest to all io being non blocking by default; with the caveat that you are using multiple go routines.
import { test } from "https://unpkg.com/deno_testing@0.0.5/testing.ts"
There is so many issues with that I don't even no where to begin.
* It becomes a more painful process to upgrade dependencies (have to find/replace across your codebase).
* Many versions of the same library get pulled in. If you depend on package@1.5.3 and a dependency of yours depends on package@1.5.4, that's twice the dependency size as compared to both just using 1.5.4. This matters more in the case of web app bundle size though than running local programs.
And also, golang tried no centralized package management by using git repo. It didn't end with people go getting from moving masters. Of course they did.
And in an area of finally accepting lock files, do you really want to go back in middle age ? Lock files are not a constraint. They are a god saver. You want lock files. You don't want to have either vague dependencies or pin pointed ones. You need both to stay sane.
And of course somebody will do something dynamic that will open a security issue.
And of course typo squatting is going to be so much easier.
And removing a bad lib ? From npm you signal the admin. When it's on it's own domain ? Good luck.
And then searching for libs is going to be fun. And naming, naming will be amazing.
And having a quick glance at the dependency of a lib ? So much fun.
And wait for the search/replace in the code that will change your entire run time by mistake.
And no possible alternative package manager. Hope their dependency resolver never sucks cause you will be stuck till you can install the next node... if they ever fix it.
Ah, and the git blames to see what dependencies have changed are going to be just peachy.
Oh, and your juniors copy/pasting code from the net is going to get extra crispy.
But wait, running the code on conditional imports means your project may install something at ANY moment in its life cycle. And could be changed by somebody editing the code by mistake without really asking to change a depandancy. You know, like a ctrl + D on "0.1" to replace all those floats quickly.
Also cool URLs don't change. Until they do. Tiny URLS dependencies are going to be hilarious.
I could go on and on and on and on...
I can't understand how you can be intelligent enough to code freaking node JS and not see THAT elephant in the doll room.
What? There's no fundamental problem with it. It will continue. What do you mean?
Yet we did. And we replaced it with "if" or"while", to avoid repeating history.
This is a prophecy: this dep managing concept, if kept in this form, will cause something terrible. It will.
Good luck.
Possible issues:
- trust
Not an issue since if you trust an author’s packages on npmjs you should also trust them from unpkg, github, etc... (Aka there is no trust in npm land except trust in the author)
- Versioning
Put the version in the url instead of some json, same difference.
- Lack of ^ for automatic upgrades
That’s a feature. Avoids the need for lock files.
- Repetitive
Have a dependencies.ts to group external imports, same thing as package.json
Any other issues?
You will always have built time dependencies with npm/yarn.
My experience with npm issues were usually that some dependency had its own build process.
There are so many inconsistencies in how JS libraries are packaged and downloaded. I think scripting in general leaves the whole dependency thing out in the wind.
Now. If we could turn packages into testable objects, that would be cool.
Did you mean to say something else?
For libraries or professional projects, ship your dependencies or use a build system. But for one-off projects, being able to pull in a utility library without needing any tool but the interpreter you're using or some extra build step seems like a big win.
https://youtu.be/M3BM9TB-8yA?t=19m55s
ps: I am a junior developer. I am taking up Dart to learn Mobile apps development using Flutter.
That being said, the Dart project has long abandoned the goal of replacing JavaScript in the browser, and instead provides a VM, Flutter, and a to-JS compiler.
Many people do not know this, and have written the language off entirely. Hence a lot of the negative reaction to Flutter.
(Though, if I'm quite honest, this is a bit off-topic from the OP)
Syntactically, Swift is almost identical (dart being older) aside from GC vs ref counting. I think it has more of a future in the Java managed language area (but with better typing, better syntax, and first class functions with closures)
They have change their mind many times about the design/future of the language, the churn is still experienced today, see Dart 2.0 (not released as stable yet), which some would say its a new language from Dart 1.0.
Regardless of how good a language it is, typescript fits in a similar space and is much more popular, has a more vibrant ecosystem, and is easier to migrate to.
Google never sent me kickbacks to provide for foolish and doomed DartVM-in-Firefox work, so I laugh once again from Skull Island at such idiocy.
They gave it another life with Flutter and it looks like it can be a big deal this time.
1. Get a native Dart VM into Chrome and eventually other browsers.
2. Get a significant number of client-side web developers that were using JavaScript to move to Dart.
It's probably not obvious, but these goals are in tension with each other. In order to motivate adding a giant new VM to a browser, you need to make the language pretty different from JS. Likewise, you need to make your implementation much faster than JS.
Both of those push you down a path where interop with JS is difficult. You don't want your language's semantics too close to JS because that reduces the value proposition of the language. And you don't want JS interop requirements to limit how you implement the VM around things like garbage collection.
But for (2), to get people to move, you need the absolute smoothest migration path you can get. You'll make all sorts of compromises and edge cases in your new language to reduce friction when getting developers to migrate to yours and you'll do anything to make interop seamless to support heterogeneous projects. (For example, TypeScript pokes quite large holes in its type system in order to play nicer with JS idioms.)
The Dart leads prioritized (1) over (2). The idea was that the VM would be so great users would flock to it giving us (2). That didn't work out, unfortunately. In practice, I think it's very hard to create a language implementation so much better that it trumps the value of existing code. So you really do need to win at (2) at all costs, if you want to a successful web-only client-side language.
That's the approach TypeScript has taken, and they did a fantastic job at it. Having one of the world's best language designers doesn't hurt.
In the past couple of years, in response to this and other changes in the landscape, we pivoted Dart. We now aim to be a multi-platform client-side language. In particular, we're the application language of Flutter, a cross-platform mobile framework.
Flutter is a very different platform than the web -- there isn't an existing entrenched corpus of billions of lines of code. Performance and memory usage matters more. You can't JIT on all platforms. Developers coming to Flutter are equally likely to be coming from Android (Java) and iOS (Objective-C, Swift) as they are the web.
Those different constraints play well to Dart's strengths. And, in particular, they align nicely with Dart's move to a full, sound static type system. Dart 2 is more "C# with less boilerplate" than "JS with more types".
Dart is still also a web language, and the better static type system really helps with static compilation to JS, but it's not our only path to success.
The Dart VM itself is great for server-side, however Google is focusing on the mobile and web tooling and support. While they do develop server-side packages e.g. for AppEngine-, gRPC-, or Memcache-support, connecting to databases like Postgresql is through a community-supported package, and sometimes it is hard to find an actively developed one.
Considering these limits, there are still good server-side frameworks, and there exists couple of big full-stack Dart applications. I've created a HackerNews-crawler twitter-bot (@DartHype) in a matter of hours in Dart, and it is running almost unchanged since then. Not that it is a big feat, but it was an easy thing to implement given the ecosystem.
If you have a fresh project, and you can select your database and other parts of your stack, Dart can be a good choice. Depending on the domain, the performance is close to the one in Go, or in Java VM, and it is much easier for beginner to pick up than other languages, while the tooling provides more safety than JavaScript or TypeScript.
However, if you need to connect to Sybase, it may not be the best choice.
The language, its consistent API, the IDE and tooling support with the static analysis is just great for beginners and advanced developers alike.
People like to hate Dart because it threatened to take away their beloved JavaScript. For those who have actually tried in the past few years, I only hear they wish their IT stack could be migrated to Dart. If you start a new project, choose wisely :)
I think it's easy to over-estimate how much "positioning" Google actually does with projects like this. We are a very big company and different parts work fairly independently of each other.
Your description is how I think of the two languages, but you might get different answers from different people. I like classes and object-oriented programming in general, and I think it's a fantastic fit for UI applications. So I think Dart is an easier fit for that domain.
Meanwhile, Go's concurrency model and nice standard library seem to be a good fit for servers.
Given the breadth of software people write today, I think there's plenty of room in the world for lots of languages.
My previous question in 2016 was "Reconcile Dart and Go", the best answer I got was: """This isn't true; they're both general purpose programming languages with strong static type systems and decent async I/O stories. You can write a web app or a server in either language, though Dart has a better ecosystem for frontend development, and Go has a better ecosystem for backend development.""" https://news.ycombinator.com/item?id=12131397
It would be great if you could give a new comparison (post-Dart-pivot) as to how you differentiate between Dart and Haxe when both (apparently) have similar goals.
In general, I dislike existential questions about programming languages. No one seems to do that for other product categories. "Why should Monet and Renoir both exist?" "We already have Castlevania. Why do we need Metroid?"
I understand languages are a little different because each requires an ecosystem that is fueled by programmer attention. There is an opportunity cost that time developer X spends writing a library for language Y is time not spent writing a library for language Z.
But, overall, I try not to get too hung up on that. I find thinking of things in zero-sum terms stressful and often not very useful. There are enough programmers to support a wide variety of languages. If we're worried about not having enough total nerd capital to support all of those ecosystems, I think there are plenty of other inefficiencies we could focus one. Eliminating the 7,000+ different Node libraries for doing asynchrony would probably free up a few person-decades of effort.
Does your team plan to release any solution like Flutter for native Desktop GUI apps?
That's what I remember hearing when Dart was just getting off the ground. So what happened? Script tags can specify text/javascript or text/dart. Did the the Chrome team just veto any integration? Did a prototype into chromium ever even exist? Google put out experimental quic stuff out before so why not a new interpreter?
I'm convinced if it was out there we'd have seen some use and some hype! I'm a little disappointed to hear Dart pivoted away. Such high hopes!
Thanks Miley Cyrus!
This is actually why I, as a developer, decided not to use Dart: because it would give Chrome a competitive advantage and more control over the other browsers. Chrome has enough advantages as it is, and I like having competition in the browser market.
In many ways, this is similar the path that asm.js/WASM took. First it's polyfilled to JS in all browsers. Then some browsers get faster native support while still polyfilling the others. Then eventually all browsers support it natively.
The initial asm.js design was a little different because as a subset of JS, the polyfill was a no-op. But the binary form of WASM, I think, required a JS polyfill on browsers that didn't support it natively.
Dart could have taken a similar path, but we didn't get the level of user excitement required to motivate browsers to follow along.
Meanwhile Dartium could not land in WebKit pre-Blink, because two GCs impose a cycle collector “super-GC” with inevitable extra cost. See https://lists.webkit.org/pipermail/webkit-dev/2011-December/... (I found this link from my HN post at https://news.ycombinator.com/item?id=12773857).
DartVM like PNaCl was trying for too much in Chrome, exceeding what other browsers could afford to embrace at high direct and opportunity costs in a competitive post-Chrome browser market. A spec would take many years and multiple competing implementations to forge. Code is spec with such big single-company-grown projects.
The Dart community is ridiculously small compared to anything to the competition, despite the heavy investment in it.
I lost track how many times I've heard my friends cursing JS and even TypeScript, and telling me they wish they could use a sane language instead. Well, you actually can, and I believe should.
Obviously it failed at that as was rescued by the Google AdWords team that adopted it.
Now they are trying the 2nd coming of Dart via Flutter, it remains to be seen if it will ever take off, or how much Google is committed to release an productive version of Fuchsia.
Personally I am not bothering with it, until I see Flutter listed here https://developer.android.com/guide/components/fundamentals
He also mentions he only knew MS made it, but not who specifically, which lets his choice sound more profound.
I set "transpileOnly" for tsc loader and set different more strict command like "npm typecheck" to "tsc --no-emit ./src/index.ts". I usually add only strict null checks and it's enough for me. I also disable most of tslint as it's way too heavy in standard CRA app for example.
This way you get hints from tsc in editor so you see what is wrong but at the same time you can run your app without fixing everything upfront.
One huge strength about Node (&Deno) is having the same language and tools on the front end and back end. Its a huge benefit to have a team on one language, even if it might not be the optimal choice. I'm not sure if that is the problem he had with Go though.
To your point, though, you'd still use basically the same language on the server and client, since deno runs typescript and typescript can compile to run in a browser. The Go side would take the place of Node's native modules, which currently have a C++ interface.
The interfaces between server- and client-side code should be well-defined and language-independent. You don't want the same people writing both, because it's harder to check that your API is working to spec if it doesn't get fully independent testing.
There's also a lot of useful server-side optimisation and security management that Node - or a high-level replacement - can't handle.
V8 may improve things, but it's going to have to improve performance a lot to be competitive.
https://www.toptal.com/back-end/server-side-io-performance-n...
They dont have to be and that is the point. If you write JS/TS front and back you can make everything much simpler with the associated benefits.
Whether I'm using React or Angular or Ember or Vue or jquery or vanilla JS on the frontend doesn't have any impact on the code I'm writing on the backend. Because one is concerned primarily with handling UI, and one is primarily for data access. There's just naturally not a lot of overlap between those.
If I'm trying to get things done as efficiently as possible, I'm not using Node because the ecosystem is pretty terrible compared to other languages like Ruby and Python. Sure there are lots of libraries available in npm, but I've found that a lot of them are janky or hard to use, missing features, poorly documented, incompatible with each other, etc.
I really don't see how python or ruby is any better than nodejs (its more a matter of taste and familiarity and you don't sound that familiar with nodejs/js in general), if anything, nodejs because of v8 is light years ahead of the ruby and python official implementations, and all three communities have lots of bad and good libraries.
> > The interfaces between server- and client-side code should be well-defined and language-independent.
> They dont have to be and that is the point. If you write JS/TS front and back you can make everything much simpler with the associated benefits.
That post implies that if you're writing in the same language on the front and back ends, it not only makes writing that language easier, but also somehow makes it easier to marry the two. Since most web services operate through some sort of HTTP interface, which is language agnostic, I can only infer that that poster somehow meant code reuse.
So there's that, plus I've seen countless times before over the years where people explicitly tout code reuse as the primary benefit to using JS on the backend. I'm not buying any of it.
Quite a bit of the server & client share code. Card handling logic, deck code handling in etgutil, eventually I'd like to move the game engine to being serverside, user management (easy optimistic protocol handling), svg rendering was used on both until recently due to Chrome having a buggy svg renderer
Source: used quite a bunch of golang + gopherjs, ruby + opal, js + node
Good luck rendering React server-side in Go or Ruby.
Whatever the case, if people write clients in react to talk to Ruby backends, can't they do the same to get "server-side react clients" that run in front of their backend app? Or does that go against the programming model somehow? Would it enforce too much of a barrier between "the view" and the model than is traditional these days?
An SSR React app that uses Redux would probably create a unique Redux store instance for each new client, dispatch just enough actions to fill in whatever data is needed for the initial render, and then serialize the Redux store state to the host HTML page so that it can be used to re-hydrate the Redux store on the client.
For more info, see the "React Server Rendering" [0] section of my React/Redux links list, and the "Server Rendering" recipe [1] in the Redux docs.
[0] https://github.com/markerikson/react-redux-links/blob/master...
My current hobby project is a kind of REPL/programming language that does all of its compute in the client, and being able to hit a button and export a "serverless" microservice is lovely side-effect of being able to run JS on the server. It isn't exactly "the backend" to my application, more "a headless client exposed to the net," though, I guess.
(As someone new to JS, the frigging Node ".mjs" fiasco almost derailed the whole thing, though... What were they thinking?)
Edit: oh, and "optimistic rendering" is also a big win -- if you can show the result of an action before it has persisted (or preview it before the user has done it) shared code can also come in handy. Though perhaps we're too scared of network round-trip times.
I'll probably start webpacking the backend soon instead, but I'm sure as hell not going to rename every single file in my codebase.
For example, I had a standard RPG style combat system. Being able to share all of the code that does calculations for combat allows you to put those calculations into the UI as much as you want.
I think standard CRUD web apps are different because there's much more asymmetry in the responsibilities of the back end and front end.
You’d be surprised at how well screwdrivers work on those funny-looking nails with the threads and the slotted heads.
Snark aside: learning a second language is not really that hard, and will make you a better user of the first language in ways that you couldn’t possibly have predicted.
For my part, I don't want it to go anywhere. Javascript powers some of the most interesting and exciting things in the world of software right now - but that doesn't mean that it's the best solution for every problem.
Most of the "infrastructure" stuff around deployment and, obviously, the JS ecosystem with all of its build tools and dependency explosions is still a mess, but it's important to keep eyes on the goal of getting rid of incidental complexity. Early Node did that well, modern typical Node development is of course another matter.
Once you get rid of as much accidental complexity in the environment as you can, having one less language to use is a huge win.
Ruby? Laughable. PHP has a zillion dependencies. Go with all of its system paths. Java VM, no thank you.
I've liked protobufs the little that I've used them. Maybe that could be a solution, but I don't think it'll happen.
A huge portion of my pain in web development is how amorphous and ill-defined the client-server interface inevitably is.
Would you ever want this vs an explicit “build” step that downloaded all the required resources ahead of time ...?
Does deno walk the program source and download everything that could be required upfront or do the resource loading on demand as it’s executed ...?
I just don't see how this workaround is the correct solution. Just saying that's it's generally a bad idea is not enough for me. At some point there will come a time when you want to treat a file as a function, and when that day comes you will want async/await at the top level.
I'm also a fan of using env vars for configuration and locating dependencies, as mentioned in the talk. Much simpler and easily extensible compared to e.g. vendored directories (requires messing with the contents of the source directory, which requires write access and breaks hashes, etc.), hard-coded system paths like /usr or ~/.some-default-location (causes conflicts when running multiple incompatible versions), etc. A simple env var of paths, e.g. colon-separated, maybe with some sane quoting convention, can be used for all of those if desired, whilst making it super easy to extend-with or restrict-to any other location(s) instead. It's also trivial to "bake in" an env var to an application, just call `PACKAGE_PATH=foo:bar my_app` rather than `my_app` (or make a one-line wrapper script).
I agree with others that importing from URLs seems like a bad idea: network I/O is one of the least reliable actions we can take, which would make importing far more complicated than necessary (what if we're offline? should we follow redirects? should we check for proxy settings? how should we report errors? etc.). All of this complexity and the inescapable problems of network failures are completely avoidable by just downloading things up-front. Package managers/build tools can fetch whatever they like (URLs, git repos, etc.), however they like (with proxies, caches, etc.), to wherever they like (one big cache, project-specific vendor dirs, whatever), and just stick the resulting directory (possibly of symlinks) in the program's environment (e.g. via a wrapper script, as above).
O please just let things be configured by a single JSON value, built from smaller JSON values if necessary.
What I'm saying is to use env vars. You could put your JSON straight into an env var, or if you want a persistent JSON file on disk then put the path to that file in an env var.
It was just a cool project and he moved on to the next cool thing.
Although one of the more interesting "talks" I ever heard him give was a discussion on promises. I was lucky enough to be in (the ridiculously long) line behind him waiting for food at NodeConf in 2012 and he and an engineer from Microsoft had a pretty spirited discussion that explored the subject in way more depth than I had previously thought possible. Ironically, given that he now considers the removal of promises to be a mistake, it was the engineer from Microsoft who took the pro-promises side of that argument.
JavaScript's async/await is essentially taken from C# (with "promises" instead of "tasks"), so that could explain it.
I remember the very early version of Node.js that had them (looks like they were added in v0.1.0 and removed in v0.1.30), although they weren't true chainable promises we have now.
https://github.com/nodejs/node/pull/18297
It's of course not a full promisification of every built in, but it looks like theyre at least trying it out.
require("fs").promises.readdir(".").then(console.log) require("fs").readdir(".").then(console.log)In practice you'd probably do:
const fs = require("fs").promises;
// ...
fs.readdir(".").then(console.log);There are legacy code that assume that calling `readdir` will yield undefined, and will have just passed that result to a function, that alters its behaviour based on whether a parameter is undefined.
Promises add hidden state, corner cases and incompatible libraries for very little gain.
Really the best case I found against Promises comes from, believe it or not, the pro promise chapter of a book:
https://github.com/getify/You-Dont-Know-JS/blob/master/async...
I really don't understand how the author managed to view the things he described as positives.
Just the length of the chapter is a testament to how overly complex promises are. On top of bringing in a ton of jargon such as "thenable", "rejection", "fullfilment", "future value" (I think the author means you are guaranteed a nextTick whooptidoo), "uninversion of control", "revealing constructor", it admits that promises do not solve most of the issues mentioned about callbacks in the previous chapter and often makes things fail in more subtle ways.
It describes how you have to manually call things like Promise.race() to solve some of the problems hardly making things more automatic then callbacks.
He talks about "Thenable Duck Typing" saying horrifying things such as
"Given that Promises are constructed by the new Promise(..) syntax, you might think that p instanceof Promise would be an acceptable check. But unfortunately, there are a number of reasons that's not totally sufficient.
Mainly, you can receive a Promise value from another browser window (iframe, etc.), which would have its own Promise different from the one in the current window/frame, and that check would fail to identify the Promise instance.
Moreover, a library or framework may choose to vend its own Promises and not use the native ES6 Promise implementation to do so. "
and
"The standards decision to hijack the previously nonreserved -- and completely general-purpose sounding -- then property name means that no value (or any of its delegates), either past, present, or future, can have a then(..) function present, either on purpose or by accident, or that value will be confused for a thenable in Promises systems, which will probably create bugs that are really hard to track down."
He praises immutability but ignores the fact that promises add mutable hidden state between your call and callback.
He mentions that promises can silently swallow errors and calls it the "Pit of Despair". There is much more. Read the chapter.
Pit of despair indeed.
Callbacks are so much simpler and beautiful:
https://medium.com/@b.essiambre/continuation-passing-style-p...
[1]: https://en.wikipedia.org/wiki/List_of_server-side_JavaScript...
https://www.mappingthejourney.com/single-post/2017/08/31/epi...
Apparently these days,
"Ryan: Yeah, I think it’s… for a particular class of application, which is like, if you’re building a server, I can’t imagine using anything other than Go"
!!
Ryan has always had, IMHO, excellent taste and good instincts and guts to make things simpler. A lot of the accidental complexity in our industry persists simply because people tolerate it, and it's great to see that he hasn't lost his fire in that area. Looking forward to see more about Deno.
The point about promises was an interesting one for me personally since I was one of the people at the time who argued in whatever small way for taking promises out, in the hopes that the language community would come up with something better.
I have mixed opinions on the topic now, but it's interesting to speculate about what might have been. The Node.js ecosystem was weakened by having different ways of handling the async question, and by a lot of developers not knowing the best practices in using callbacks effectively and leading to "callback hell", which is totally unnecessary. It's possible that having promises baked into an early Node would have constrained or even fragmented what we ended up with in the language, and that would have been worse. I'm still a little disappointed that we didn't end up with anything more elegant than promises in the language itself.
It's interesting to compare package management in Node and Go. NPM got the early adoption and became the de-facto package manager at a time when JS had no such thing. In Go, the package manager question has been unsettled for a much longer time and there are more chances to experiment. Package management is simply difficult, and it seems impossible to design a good programming language and resolve the package management questions at the same time. It's sad to see some of the criticism against NPM... it's much easier to criticize than to build a better system.
Some parts of npm are much better than with most package systems, some really suck.
Maybe if npm weren't included so deeply into Node, it would make something like Yarn emerge sooner and replace npm without much hassle.
I don't agree with those complaints, but I do agree with you that "some parts" are really good. node really figured out the right search strategy for an unscoped import. (e.g. "require 'foo';" rather than "require './bar/foo';") Just look for a directory with the name "node_modules". If you don't find it, go up one directory and try again. So simple! So predictable! So complete! It works so well, all manner of "left-pad" abominations can be supported. Any other system should think very carefully before using a different import search strategy.
[0] with the exception of those related to path depth: I think those are resolved now? I wouldn't know because I stay on an OS that doesn't go out of its way to frustrate me.
To save on disk space, use pnpm, best of both worlds!
Go is a well designed language that is productive, has a very tolerable learning curve, good documentation and does not need a JavaScript facade.
If kids in the 80s without the Internet could learn programming, you can learn Go in 2018.
Expect another video in 10 years: Things I regret about Deno, probably having advice very similar to this.
It sounds like Ryan is still interested in making V8 work so I have to ask: why do you want to be writing server code on a client browser engine?
Why do you think JS became so much faster compared to the anemic octopus is was before ?
And can you imagine the perf of Ruby or Python if a 10th of those resources were allocated to those ?
This seems like an odd dependency to me, and it seems like it adds no value.
I'm going off of what I've seen myself, and what I've seen in the xi editor:
https://github.com/google/xi-editor
"The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty of the libraries available for the other ones."
Low-friction interfaces for developers are great but I'm not sure I agree with the comparison here.
"Performance" in a server also means throughput. There's a reason why protobufs exist and it is all about performance (and type safety which is related) and there's no way JSON serialize/deserialize overhead, in both directions every time, is going to be a good choice here.
Ryan; kill that feature now. It’s not needed. It’s just cute.
He mostly regrets the way Promises work (edit: that the async programming APIs don't work with them), security/sandboxing, and the module system.
Then he introduces "Deno", a successor which is under construction and which will fix all these problems. It exposes a language based on TypeScript. Internally, some parts of Deno are written in GoLang.
Uhhh, Ryan, there's this thing called Lisp...
Edit: so two space indents weren’t a consequence of “callback hell” ?
for example, a lot of ruby i've seen (admittedly it's been a while) used two spaces, and i don't recall it being any more prone to deeply nested code than other languages.
that said js code tends to be rendered in a variety of other places like browser dev tools, which may influence this as well.
By the way, was i the only person to notice that “Deno” is an amalgamation of “Node”?
Javascript has a lot going for it from async-everything to conveniences like destructuring. And it works in the browser and has things like Typescript.
I have to find good reasons to use another dynamically-typed language over Javascript.
With async/await is so nice now.
await step1() await step2()
That's it.
Python's scoping is easy, it's just also very shallow. What are your problems with it?
Seems pretty classless to rage quit the community, brag about how much better Go is, then mark your return at a JS conference by shitting on Node and hard-forking the server-side JS community.
Though, it's posted in a confrontational way so the conversation deteriorates.
Is being a dick pretty common in open source discussion?
> I do not need any help from you thanks. This is aimed towards Ryan
Yikes.
Also, calling everything a horrible miscarriage is not a discussion.
Pointing out the problems with something is easy. Anyone can do it. What's hard is evaluating problems in relation to the trade-offs that were made. Only amateurs think everything is "horrible" as if there are never trade-offs.
I agree with my sibling: it's only good example of worthless Github issue "conversation".
Seriously why are we as a community still entertaining those outlandish statements and attitudes? If you heard that sentence as a non-tech person, you'd swear modules is a computer virus or something, that it's so inherently bad that it's not even worth discussing.
Props to Ryan for his amazing work, but come on let's try to be civil and respectful here.
when he said "this project is done now" my takeaway was that he didn't anticipate the popularity of node. and "so wrong" was him acknowledging that he was very obviously mistaken.
Thanks for providing context.