When Node.js is the wrong tool for the job
medium.com
medium.com
That, and nose.js developers seem to repeat the claim that node.js makes delivery faster… but is it really any faster than ASP.NET, Rails, Django, or Go stdlib? Those frameworks are so fast for prototyping and delivering bread-and-butter apps as it is (and some of them let you do multithreading to boot).
I'm also really not interested in how things work for "typical CRUD apps" because those are so trivial to write in any decent environment.
I'm worried that node.js articles are the same kind of echo chamber that Rails articles were 10 ago.
If you are a developer then you should invest your time and be able to learn and use the best tool for the job.
If however you are something like a Product manager then learning JS is much more valuable time investment. It is the one language that will, currently, run anywhere ... Backend, Frontend, Side end, whatever.
You can quickly get good at JS to the point of being able to prototype the full app stack and come up with something that works, goto market and fail. Should you find your self in the unlikely position that your product meets its success, then you can go ahead and hire a Go/Elxir/WebASM whatever guy to optimize this and needs larger throughput.
You have 24 hours in day, 6 to sleep, 3 to breath, 1 to eat. How do you want to spend the remaining 14 ? You get good at what you practice :)
I tend to agree.
> That, and nose.js developers seem to repeat the claim that node.js makes delivery faster…
The qualification was "if your team already knows JavaScript". OK... If the team already knows Java... building something like this in Java would be faster to deliver, and wouldn't hit some of those issues the author brought up (4 vs 1, single vs threaded, etc).
And the whole "oh, you know language X" - means almost nothing when the project is more than trivial hello world. Every project I've been brought in on, the qualifier was "must know tech X". And almost always (there were exceptions) I knew more about tech X and software dev in general than the original developers, and the crunch was not "how do I do XYZ in this tech?" it was "how do I get the other developers to actually use version control?" or "use version control sanely?" or "document anything?" or "write tests" or "have test data" or "have a repeatable build process"?
Knowing ASP or PHP or Ruby or whatever, but going in to a project without repeatable build process, tests, documentation, requirements or version control, is a recipe for disaster. Stressing the "deliver quickly" aspect of any language, if you're actually trying to deliver for a business or with a team of people, is extremely destructive short term thinking. And yes, sometimes, in rare cases, it may be a necessary evil, but I think it's become the norm with words like "agile" being thrown around as synonyms for "don't have to write anything down".
One of the things you'll learn if you hang around Hacker News/online programming communities is that a lot of the people in those communities are at the top end of the spectrum (there is a degree of self-selection that means a higher proportion I think).
It's pockets of relative sanity in a sea of disaster.
Nobody ever set up source control for the code I had to work on and nobody but me ever worked on it, the others did have source control as far as I know and at least one had 2 people working at a time. We did try to get my using it but it was clunky and didn't last long since it was just me using it.
It seems so strange in hindsight but that whole place was so so different from where I am now.
If you're just out of university or read tech blogs or hackernews, you'd think everyone these days is doing TDD, pair programming and agile. The reality is that a lot of major businesses do none of that.
4 years of no version control, no test environment for at least a year (it was shut down then never turned on again), no written docs, no written tests (and for periods of time, no backups). This software manages and controls 9 figures of USD annually.
The 'bottom line' was/is always "i need a form that does X" and "I need a form/report to show Y". Yes, the immediate short term goals of the business worked.
The entirety of the codebase has been hardcoded to work with library calls that were known tech-dead-ends years ago.
The business now needs someone else to come in and manage things, because the original guy left. The business is now up a proverbial creek, because everything was in someone else's head. TDD/testing/docs externalizes some of that, and does have real business value, but it requires someone to be able to look more than 6 months in to the future with respect to tech decisions.
Their bottom line will be hurt over the next year because they will be needing to pay far more to repair the tech debt, and it will mean fewer new things get done (or.. they'll pay even more for that). So... "business value" needs to be defined as to how you're measuring it, and 'short term' is always easier than 'long term'.
I'm ignoring the approach that says: Let's just spend a LOT of time writing mocks that stub out 90% of everything that matters, and pretend that the tests actually show something that matters.
However, note I'm talking about something way more basic: many business don't do any kind of testing at all that isn't done by people manually trying the system. In the case of internal tools, this "testing" is often done directly by internal users trying to use the tool. This is way more common than one would suspect from reading tech blogs (unless one reads TheDailyWTF, of course).
It's important to recognize them as bad. It's also important to try to understand what lead to their adoption; often there are ways to work around that and apply a best practice in your area, even if other areas still suffer without it.
The "purchased" ones are usually worse than the open source ones (e.g. - Git, SVN et al).
The "ENTERPRISE!!!" devs don't use the system effectively, usually: haul things off to a corner for weeks before doing a single check-in/commit; may or may not know how to branch; bumble with breakage during merge/pull operations; end-run the system by releasing non-compiled files directly to production without saving them in source control.
Source control use is like the other comment about programming language: the main issue is not whether you have used brand X, but whether or not you have a coherent mental model of what the hell you are doing in the first place. That said, there are some BAD commercial source control systems out there, preying upon the "you need this, even/especially if you don't really know why" crowd. These are usually sold to The Management in terms of the control they bring to source control - they will have a trigger to force entry of a trouble/enhancement ticket to go with a check-in/commit. That fact that they suck at source management is of no consequence (e.g. - painfully slow, little to no command line interface for automation, poor research/discovery features, poor branch/merge support, unreliable / data corruption)
I've seen this so many times on consulting gigs and with the occasional new employer. I'm hired because I'm familiar with the technologies they're trying to use, but the first major value I bring to the organization is simply getting some kind of sane process: working version control, automatic builds, test suites, one-button deploys, and so on.
Same, and it's getting really fatiguing. Probably the #1 driver for me to get a side-project off the ground is to finally actually write code and not just play janitor for someone else's mess.
Unfortunately, this is pervasive in the "real world", especially with regard to client-driven (agency) work. Deadlines beat developers almost without fail. I agree that it is destructive short term thinking. Entire weeks of programming can be destroyed in a 15-minute phone call that (re)highlights a limitation clients either didn't account for in their specification (if you can call it that) or just chose to ignore.
This isn't to say that using a language or framework a developer is familiar with is baseless, but I've heard this advice before and see it reinforced often:
1 - Hire people not skills.
2 - Success is 90% preparation.
Instead of writing "a quick Python script" for personal use, I now write "a quick node script" because there's almost no thinking required on my part. Just require the packages and throw them together.
Not even Python has this package-completeness.
While npm has a lot of packages, Python has a lot of fully matured, well-tested and well-documented packages.
It's apparent to me that there are a lot of low quality, duplicate and otherwise unstable npm packages.
Don't get me wrong, I actually like npm too but I don't think the Python comparison is even in the same ballpark.
Take the case of Node and JavaScript itself. The ecosystem is huge. The tooling is complex. A newbie starting out will not even be able to get many examples to run without a mesh of transpilers and build tools. There are roadblocks everywhere; what's the type annotation there? What's with the strange JSX syntax? What's a transpiler? Add to that, EcmaScript itself is a moving target, more so than any other language.
Similar issues exist in other languages. C# is a simple language to master, .Net is a huge ecosystem. There's not going to be a lot of resistance moving from C# to F#. But for the newbie, the challenges are daunting. Should you use Microsoft's preferred ORM which ships with .Net or write your own SQL? Which UI tooling should I use? There is plenty of advice on the web, yet it needs an experienced developer to make those decisions.
Consider this. if you chose SilverLight a few years back, your product is soup today.
Picking an ecosystem is not easy. Has never been.
The amazing part about JS is that most of the code written for browsers a decade ago still works today. It is extremely unlikely that we will see a company outright ban JS in their web browsers in the same way that iDevices don't support flash.
Yes, that is mostly true, but it glosses over two big problems with the JS ecosystem.
The first is that longevity seems to be a dirty word. I have seen professional JS developers say, without a hint of irony, that their code only needs to last a year, maybe two, so it doesn't need to be designed for long-term maintenance or ongoing development. That same attitude exists in much of the library and tool development, which means while your code might still run in 2 or 5 or 10 years, the dependencies for your code might not have kept up. Even if they still run, they're stuck at whatever level of features and security and flexibility they had when they were abandoned, which might well have been only six months or a year after you added them to your project. Now you have a perma-brake applied to everything you do.
The other big problem with the JS ecosystem is that JS itself is a moving target. Thanks to evergreen browsers, Node, and those controlling them, standardisation and stability also seem to be dirty words now. Your code from 10 years ago, using only basic JS language features, might still run. Unfortunately, your code from 2 years ago, using whatever alpha/beta quality JS features supported promises or observers or local storage at that time, might not. And the same goes for code in dependencies you introduced 2 years ago that are already abandonware.
Multiply those two factors and you have a recipe for instability and frequent unexpected failures unlike anything I have ever seen on any other platform.
"Nothing we did had any shelf life" was one of the core problems identified by Gene Kranz in his "tough and competent" speech to his mission control team after the fire. I think it's a salient point.
When all the parts are constantly churning, fires are to be expected. Progress is messy if we're not cautious, and I'm not sure if we are being cautious in the right places. We could learn from Mission Control. (Lots of room for debate, here! And I'm being purposefully vague.)
class Shape {
constructor (id, x, y) {
this.id = id
this.move(x, y)
}
move (x, y) {
this.x = x
this.y = y
}
fixed_offset(off_x,off_y) {
return () => {
console.log(`offsets x: ${off_x} y: ${off_y}`)
return [this.x + off_x, this.y + off_y]
};
}
}
var Shape = function (id, x, y) {
this.id = id;
this.move(x, y);
};
Shape.prototype.move = function (x, y) {
this.x = x;
this.y = y;
};
Shape.prototype.fixed_offset(off_x,off_y) {
var self = this;
return function() {
console.log("offsets x: " + off_x + " y: " + off_y`)
return [self.x + off_x, self.y + off_y]
}
} import defaultMember from "module-name";
import * as name from "module-name";
import { member } from "module-name";
import { member as alias } from "module-name";
import { member1 , member2 } from "module-name";
import { member1 , member2 as alias2 , [...] } from "module-name";
import defaultMember, { member [ , [...] ] } from "module-name";
import defaultMember, * as name from "module-name";
import "module-name";
The only people who care about ES6 classes are the ones who couldn't be bothered to learn how to use duck typed prototypes and were always trying to shoehorn in polymorphism. Prototypes were one of the few things Javascript had going for it, but adding classes means now there are two OO systems that don't interoperate. This is a new problem, not an improvement. There's a similar problem with callbacks and promises.
Other problems added in ES6: imports don't give an error when the thing I'm importing, continuing the JS tradition of handling errors by pretending everything is okay.
And this is in addition to the rotten core of JavaScript. There remains no integer type. Basic operators still take arguments of all types and return something regardless of whether it makes any sense. "this" still has little to do with this.
In any other language I'd complain about the lack of threading, but given JS can't handle easy stuff like implementing a sane comparison operator, it's probably better that they don't try anything legitimately difficult like threading.
You mean you're not interested in what a huge part of the industry is doing? Well then that explains why you don't see the benefit.
There's tons of work creating user interfaces, transform some business logic and shove it back in a DB. No one said it was going to be a life-fulfilling experience. It's just work.
Now many of these shops want a native ui, mobile ui, responsive design and a back-end developer all in one. Many times they hire for specific frameworks so that you're ready to be productive day 1.
If I were staffing up a team to do a CRUD app, or even more complex, I'd staff up with JS developers.
It's 2017, We've had desktop computers for 35 years and most of the people I know working in various industries still manage a huge chunk of their work in email and excel because it's hard for their management to understand the benefits of automation or why they'd make an investment in the longer term with good software, I know of one company thats in the FTSE250 that manages it's entire pallet and tray inventory by using excel spreadsheets on a shared drive and sending people from the office to sites to look whats there (they order 100K+ Trays at a time).
Otherwise, yeah, trivial stuff probably is done best with spreadsheets, wikis and ad-hoc emails.
Keeping track of some office supplies ordering, excel is fine.
Keeping track of 100K pallets where delays because they got 'lost' knock on right down the supply chain not so much.
Yes, you chopped off the important half of the sentence, and responded as if I hadn't written it. I said that I wasn't interesting because CRUD apps are already so easy to write in many different frameworks.
Let's assume that what you're saying is true—that most shops want all of those skills rolled into one person. The people I've met that are good at both front-end and back-end development are comfortable in multiple languages to begin with. The developers who only know JavaScript and are not comfortable in other languages are not good full stack developers.
But I don't believe that this is true to begin with, most places I've seen or the places where my friends work have people who focus on front-end and people who focus on back-end. There's just too much of a crazy front-end landscape to expect magic from someone who doesn't live in that world, and back-end is its own world.
It doesn't matter if CRUD apps are easy to write, the point is there's market demand. And for some reason you write them off as trivial, but I've seen some business logic/requirements that'll make your head spin.
> The people I've met that are good at both front-end and back-end development are comfortable in multiple languages to begin with.
This type of position is becoming way more common. But I imagine many of them would prefer to work with one language anyways (I do). It keeps things simple. One language to be an expert on, that can be interviewed for, one testing library, one debugger, one set of coding standards. It's kind of a no brainer. I'd argue that almost any expert front-end dev can be a decent back-end developer.
You can reuse code / modules.
You Don't suffer from context switching inherent with jumping between languages.
Also, people act like JavaScript is a bad language purely because it is not strictly typed. It's an incredibly fast and robust language with a vibrant ecosystem -- something I'd argue is more important than pure technical bullet points.
You will find support for the most random and amazing things in JS. It is truly mindblowing.
I recently had a client request to run a process involving extracting data from an Excel file. Originally wrote it in python using openpyxl http://openpyxl.readthedocs.io/en/default/, then it turned out the client also needed XLSB support. openpyxl and xlrd don't support XLSB, but the only thing that did work was a JS library http://oss.sheetjs.com/js-xlsx/ and node module
E.g. - concurrent operations in a single VM, but using actors exchanging immutable messages, NOT shared data and locks.
If only it were a better one.
Maybe something that was designed from the beginning to be a general-purpose programming language, and not a hacked-together-in-a-weekend scripting language intended to add low-grade interactivity to HTML, that has bloated and twisted into the mess it has become.
I'm eagerly awaiting the advent of the WebAssembly era, and honest-to-goodness real compilers. Flying Spaghetti Monster be praised, then we may be able to leave some of this nonsense behind.
This is also an excuse for NIH ( never invent here ) where people try to shoehorn an OSS product even if it doesn't fit their needs .
If you are a web shop you have javascript. Using node.js allows you to eliminate one extra language.
Is it perfect? Definitely not. Is it best for everything? Probably not the best for anything. But, it's good enough for most stuff. It doesn't make sense to incur the burden of another language for a one off task.
And why not use a shared memory server?
> Operations started adding rules with 100,000s of domains, which caused a single set of rules to be about 10mb large ... If we weren’t using node.js, we could cut this bandwidth by 4 as there would only be one connection to the Redis cluster retrieving rule sets.
Maybe a 10mb json string isn't the best design decision.....
Or you know, you could have one node process connect to the Redis server, and have the local processes read from a shared memory server.. Or you could not store your rules as a 10mb friggin JSON string..
> When rule sets were 10mb JSON strings, each node.js process would need to JSON.parse() the string every 30 seconds. We found that this actually blocked the event loop quite drastically
Well then do it in another thread and save it to shared memory. Maybe, just maybe, JSON strings aren't the tool for the job here.
As the above comment clearly points out, there are workarounds to make things work. Still, when I think about the amount of time and effort that goes into debugging systems when they hit limits like the author mentioned, I imagine most of the the benefits gained by using a language like Python or JavaScript go away.
It has happened to me before. I design something, then I run it in multiple CPUS and it doesn't make sense anymore...
Or realize that if try to scale the current design doesn't hold.
Node.js running in a single thread is a good thing if you know how to work with it. Or if you really need multiple threads, use another PL.
Or you could write it in C++ and bind it to Node.js. And run it as a background process. Or spin up a worker? there seems to be multiple solutions here...
That said, I can imagine other languages (like Java or Go) could still end up being more efficient than Node.
Similarly, the memory issue isn't so much that a single copy of the table takes a lot of space, but rather that they need to store 4 copies of the table -- because they're running 4 different processes in order to utilize multiple cores.
Both of these issues are specific to nodejs.
There's not enough detail to be sure, but this sounds more like "when a relational database would be a better idea than redis."
Edit: That is, pushing the evaluation of the rules down...rather than pulling a kv and walking 10MB (of JSON?) to get to the small number of rules that apply for the transaction.
Unless you know for sure what limits you will hit - it makes sense to iterate quickly and find out. Then, if the service is actually hitting limits (and probably not the ones you thought) - re-write it in a multi-threaded concurrent language like go, elixr etc - or a language designed to solve the actual problems the service is hitting (which might be disk i/o or other infrastructure level things not language choice)
We go great length to figure out which attributes to fetch and add limits to all our sql queries. These are best practices but with node they are must.
-Having a hard time taking anything this guy says seriously
Actually, that's its fundamental design. They reduced everything down to a very small core with very little features, so things would be obvious and you don't have to learn or remember much. It's refreshingly simple!
With that said, I'd say they reduced it too far down. There's no generics so you end up using `interface{}` everywhere which often leads to issues due to its late binding. Or you end up just using codegen tools, IIRC.
Also since there are no exceptions, you end up with constant checks for error code returns, which end up usually just being strings and not much else. Not saying exceptions are the best way to approach error handling, but they do allow you to reverse through an entire function call stack and clean up any state along the way, along with adding more granular error information you can write handlers to react to. Go reminds me of the pain that was C error handling and juggling error codes.
This is completely standard and the only way to do node in-memory caching. Think of each worker as a completely independent node process, which is only bound to the cluster by a master process which has the ability spawn and kill child cluster processes.
This isn't accurate you can use shared memory. There are a few modules that implement this. In addition, you can offload the JSON.parse to the dedicated "caching" process that updates the shared memory.
Ok nevermind, google is my friend: https://github.com/PaquitoSoft/memored
I can see where this would come in handy. But at 240MB total resident memory per CPU across 4 node workers that OP describes, I wouldn't hassle with it.
I don't think I understood your point. But is not Node.js supposed to not block long lasting tasks?
Node.js is garbage (well, good for simple crud apps) and so is JS.
That being said, I do like how C# handles it more than the way JS has handled it, but both are fantastic.
You can say JS is "garbage" all you want, but it's close to the most popular language on the planet and many many people are writing great things because of it.
My general (perhaps wrong) impression is that other languages commonly used in backends are moving toward async io, usually through maturing libraries.
Many of the applications people think of for node.js end up being "glue", much like python, and can live within this constraint for a very long time, where the I/O optimization is a nice benefit.
It's true that other languages add async models (or emphasize those that they already have), but eg. in the case of Java you're sitting on 20 years of synchronous library and custom code, and it's not clear moving to async is worth it at this pont.
Rate limiting is not hard either way, but single-threaded async is slightly easier because your counter is automatically thread safe.
The larger and more complex your system gets the harder it gets to keep it thread-safe. For me this is the big advantage of single-threaded async, and I like node because this single-threaded async is idiomatic in JavaScript.
There were probably opportunities for the author to architect the system in ways that were better suited to node (given that was the chosen platform), but the architecture choices were not unreasonable by design. These are some good things to consider when architecting a system, and considering node as the platform.
I'm not sure I agree that node is the "perfect for simple CRUD apps" though