Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]
rust-lang.org
rust-lang.org
It’s actually one of npm’s web services that was rewritten.
FTA: “Java was excluded from consideration because of the requirement of deploying the JVM and associated libraries along with any program to their production servers. This was an amount of operational complexity and resource overhead that was as undesirable as the unsafety of C or C++.”
I find this comparison a bit odd. Even if not using containers, the JVM isn’t hard to deploy as distro package managers include it. Unless a team is managing servers manually rather than an automated tool this doesn’t seem that complex. Am I missing something here?
Can you give examples of languages(compilers or VMs with bad defaults in your opinion?
My opinion of a bad default would include a language toolchain which defaults to dynamic linking and requires one to opt into static linking (Java, Python, JS, etc, etc, etc). Worse than that is an anorexic toolchain that has no defaults whatsoever and requires you to pass every little detail directly to the compiler--bonus points if your language has a massive ecosystem of competing tools which are meant to manage these sorts of details for you but utterly and uniformly fail to do so (looking at you, C/C++).
If they had instead made the default to act like almost everything else it would have worked with standard process monitors and limits with no effort required and a substantial fraction of the downtime I’ve seen for Java applications would never have happened.
Similarly, if developers who deployed apps knew enough to avoid this, we’d know by now because it wouldn’t happen so frequently. It does highlight who failed to do real health-checks (e.g. years ago most Hadoop services needed huge amounts of RAM to start without crashing but they’d be listed as okay) but it’s the kind of thing sysadmins have been cleaning up for decades.
As i've moved onto 11, i've slashed our apps' JAVA_OPTS down to almost nothing - max heap size, some GC logging flags, that's it.
The default GC from IBM, Azul, and others wasn't the same one.
In fact Azul has made a business of selling JVMs with GC that can handle hundreds of data GBs.
Modern OpenJDK versions also have such GCs now.
Regression tests are required for any toolchain.
All of these must have been tested against the Java TCK. So, the answer is "whichever you/your company decided is okay" for the first part and for the second part "newest update of the Java version your software runs on". Doesn't sound very complex to me.
New java versions do break existing libraries or apps, and need to be tested thoroughly. When the company hasn't budgeted for that expense, it becomes difficult to update.
Often an architect or software team will insist on using the Oracle JVM rather than the included openjvm. That adds extra steps to download, store as an artifact, distribute, verify, etc etc.
The people who wrote the build pipeline have since been laid off, and an updated set of libraries requires a lot of work to trace back through poorly documented and understood code to make changes.
(Not to disagree with you here, it's more that I'm trying to illustrate how, with poor foresight, Java dependencies can get difficult to manage)
Have you used the Rust compiler? My experience tells me that any Github Rust project not updated in the last two years doesn't work with my Rust compiler. Whereas Java apps written a decade ago still compile and run on OpenJDK/Oracle often with zero or near zero changes.
> Often an architect or software team will insist on using the Oracle JVM rather than the included openjvm.
Yes, you could choose this. But if you have the ability to choose an entirely different language, I guarantee you have to ability to choose OpenJDK if that's what you really want.
> build pipeline , an updated set of libraries
Doesn't Rust have libraries?
In fact, if anything, Java libraries have a longer shelf life and greater compatibility because they don't require integration of build systems. Java libraries can (and do) use 5-year-old compilers, postprocess bytecode for perf (e.g. Hikari) or size (e.g. Proguard), and even use entire other languages like Scala and Kotlin. But the only thing you as the library user need is the JVM bytecode, which is still high-level enough to maintain runtime interoperability but sufficiently low-level enough to achieve strong build-time interoperability.
---
Much of the "organizational overhead" seems to come down to "I don't like managing the runtime". And that's not wrong; static binaries are nice.
But how much harder is it really to manage/isolate/version your JVM runtime for your server deployment, than manage/isolate/version your Rust compiler for your CI pipeline?
I don't think this is correct? Rust introduced an "editions"-based system in 2015, with the very aim of ensuring forward compatibility on a crate-by-crate basis. The aim is definitely that any crate written to be compatible with some "edition of Rust", whether 2015, 2018 or whatever, can be made to compile on a future version of the compiler.
So, yes buyer beware. But when the whole ecosystem does it...
Please file bugs against the Rust compiler then, because that would be a serious violation of the compatibility rules!
The 0.3.0 checkout fails because of a change in Cargo. Specifically, I think it used to define OUT_DIR while compiling the build script, but this was not intentional. It was only supposed to be set when running the build script. ripgrep 0.3.0 got caught here in the time after when the bug was introduced but before it was fixed. See: https://github.com/rust-lang/cargo/issues/3368 In this case, cargo update doesn't solve this. You cannot build ripgrep 0.3.0 with any recent version of the compiler. You need to go back to a version of the Rust distribution which ships a Cargo version that sets OUT_DIR when compiling build scripts. (Or, more likely, patch build.rs since it's a trivial fix.)
Of course, both of these things are fine on their own. But strictly speaking, from a user experience point of view, code does break from time to time. With that said, ripgrep 0.4.0 and newer all compile on Rust 1.33, and ripgrep 0.4.0 was released over two years ago. So, strictly speaking, it does show the GP is wrong in at least some circumstances. :-)
But as a possible example, typemap [1] which is a library featured on Awesome Rust [2] ("curated list of Rust code and resources").
Project uses cargo. Has a lock file with two deps. Last code commit was May 2017. And I can't get it to compile.
But moreover, you stumbled upon one of the worst cases, a soundness problem that has yielded a lot of discussion because of that crate specifically: https://github.com/rust-lang/rust/issues/50781
The issue here is that the typemap crate was found to be relying on a compiler bug. The compiler bug could be exploited to write transmute in safe code: in other words, all the safety guarantees of Rust go out the window unless we break that crate. It's a bad situation, but I don't know what we could have done differently. I think everyone agrees it's not worth sacrificing all the safety properties of Rust to keep typemap building.
Sadly, some of my projects still used an older version of the openssl crate somewhere inside their crate tree.
The openssl-sys crate author chose to check that the native library version it was compiled against was below a certain version, so it broke. All requests by users to fix the legacy bug were deflected with the mantra "go update the openssl crate the fixed version is out since a year already"...
My understanding was that by virtue of my project not having the edition 2018 key I would have been isolated from such changes to the language.
I'm also not condoning the decisions made in the project I describe. In fact, those ancient decisions are responsible for a significant amount of stress and work for me right now.
I've absolutely had experience of good JVM based projects that had loose dependencies on libraries and jdk versions such that it ran just fine under the openjvm, and ran just fine when that package was updated.
Oh, then that makes more sense.
It's always possible improve. But Java is pretty much king when it comes to compatibility and maintainability. (Not so much in other areas, like verbosity, language features, type system, memory overhead, etc.)
Node gets a bad rap mostly for the fact that it has tons and tons of inexperienced engineers using it (probably as one of their first programming languages.) Same reason PHP got a bad rap back when. You can build solid software in both, by finding and absorbing engineering best-practices from people who have already had hard-fought battles to learn them; but an engineer will have to be burned at least once on building/scaling+SLAing something before they start looking for those. You might say that the Node ecosystem has a lot of programmers that are “engineering virgins”—they’ve never been forced to contend with the real problems of engineering software. But they’d be “engineering virgins” no matter what language they’re using; that’s not an indictment of the language, just a consequence of its popularity and approachability.
EDIT: I did miss that. Sorry for the noise.
> I just like that someone at npm would avoid something because it has lots of dependencies and overhead. The irony is strong with this one.
For this to be "ironic", having lots and lots of dependencies would actually have to be a Node.js "best practice." But it's not.
It seems like a best practice to outsiders, for the same reason that setting `register_globals` seemed like a "best practice" in PHP back in PHP3/4. Because it was extremely common, one might assume that it was endorsed as a canonical approach. And so you do it yourself, and write sophomoric tutorials suggesting others do the same, perpetuating the problem.
In reality, the "best practice" followed by experienced software engineers (for Node.js or any other language) is to carefully consider your dependencies, and to try to avoid dependencies that cause an explosion of sub-dependencies. The NPM maintainers are experienced engineers, and so they follow this best practice.
There is no irony here. It is not "the Node.js way" to use tons and tons of dependencies, such that the NPM maintainers are going against the grain somehow. It's just the way of programmers inexperienced in engineering to not care about dependency proliferation; and then, further, to make a large set of their own tiny libraries (with already-exploded deps trees) because they aren't yet at a stage of programming expertise where they see that code as trivial to bang out whenever they need it (see: the left-pad package) that then further encourages others to depend on them. It's the "copying and pasting a solution together from bad code in five StackOverflow posts" phase of one's programming career, except instead of having to copy-and-paste, all the snippets are symbolically linked together into a big tree and you refer to them by name. (Again, that's not an indictment of Node.js—there's nothing you can do to stop a bunch of inexperienced engineers from doing this to your package ecosystem as well, if they happen to be drawn into your community.)
From people that seem plenty intelligent and experienced.
You will probably hear that you should use Laravel or similar. I'd argue it's a pretty big hammer, so don't reach for it if you don't need it.
The biggest issue is probably still the breadth and inconsistency of the standard library. Too many ways to do the same thing. Also, the general issues of a dynamically typed language, sprinkled in with things like == vs ===.
I like the books this guy writes: https://github.com/codeguy
Don't get me wrong, if you need to do one small thing, like for example I had a desktop app and the user could submit feedback directly from the app, then a single PHP file was enough for this case(no dependencies, no frameworks), you get the submitted data, clean it and put it in the database or submit it to a third-party API that can handle it.
Not to mention that the Web is now full of discussions around this, with official posts from Oracle, Red-Hat, Amazon, IBM, Azul, Microsoft explaining how to go forward.
But assuming that Oracle support is actually desired, here is what you are asking for.
https://www.oracle.com/technetwork/java/java-se-support-road...
https://www.oracle.com/java/java-se-subscription.html
https://shop.oracle.com/apex/f?p=DSTORE:2:::NO:RIR,RP,2:PROD...
Old Java dog here.
But lets just forget about the efforts from AdoptaJDK, Red-Hat, Amazon, Azul and bash Oracle, it is more fun.
None of that helps you with what the charge is when you're elastic, or using multiples of hyperthreads that don't add up to a integer number of cores. I agree that OpenJDK is a better idea.
Enterprises that care about AOT Java have been using them almost since Java exists.
Either way the runtime is baked into the app and gets deployed and tested with it as a core component. Runtime upgrades then become vanilla deployments.
Very few people would choose to install libraries (JARs) used by their code via their OS package manager for instance.
However in that case, this whitepaper (and many others) are damning in how little they actually state and why so many technical decisions go wrong.
One are I felt was missing is some data here:
"It keeps resource usage low without the possibility of compromising memory safety. "
How did the resource usage compare with the Go and node rewrites? What metrics were used under which workload? Benchmarks are never perfect but I think a CTO-level person would like to see a table of results like that.
Note that only one service (authentication) was rewritten from node to Rust.
There’s also a second service we know about in Rust, and that’s the one that renders package README pages.
Just so everyone here is aware, this is by now an outdated complaint against Java.
https://vertx.io/blog/eclipse-vert-x-goes-native/
I'm choosing vertx as an example since it competes already with rust and c based applications over at https://www.techempower.com/benchmarks but you ought to be able to compile general programs ahead of time.
The below videos show GraalVM based app loading up Spring framework, and flowable process engine and making a rest call to an external service all in 13 ms!!!
Checkout
Beyond that, enterprises using PTC, Aicas, IBM, ExcelsiorJET JDKs have had the option to deploy AOT native code in production for the last 20 years or so.
https://github.com/oracle/graal/blob/master/substratevm/LIMI...
Vert.x encountered some issues with reflection while Rust simply does not support the sort of dynamic reflection that Java running on the JVM can achieve. SubstrateVM forces you to have compile time reflection, and Rust can also support this to some extent. If Rust is a feasible alternative to Java for you, then you are not going to encounter this limitation if you choose to go with Java. Plus, if you ever decide you do need this power in your application, if you go with Java you can pay the cost of the increased operational expense and install a JVM.
The amount of time it takes to learn something is often indicative of its power. Anyone who has learned a foreign language or a musical instrument knows that the time spent investing up front pays huge dividends down the road when you have the skills and tools to richly express yourself. The reason that Go takes two days to learn is because it artificially limits the amount of up front investment at the cost of limiting expressiveness over the lifetime of your use of the language.
What happens is that if you are already proficient in one of them, one of the others takes way less time (specially in the case of C++, given it forces you to learn almost all paradigms).
In addition, writing a small program does not mean you have learnt C, C++ or Rust.
I would argue that something that takes even an experienced engineer a mere hour to write is very small and has little complexity (especially if unit tests are counted towards the hour it took them to re-write it). This means it's difficult to gauge how much Rust was 'learned' during that week.
The language is nuts. It’s true what they say cargo is even better than the language, it’s just so easy to add packages to your project or to split your project into packages.
Cargo is an amazing investment as this will help people write non duplicated code. Like how many string implementations are there across c code bases. Each c project has so much code that’s the most boring, repetitive shit you can imagine. Cargo let’s you concentrate on writing your code without hassle.
I have experience with a lot of package managers, gems, go, cocoapods, sbt, cabal, pip, spm, npm, you name it but cargo is on a different plane of existence. Cargo makes the whole internet your standard library.
I also like cargo workspaces. Modern development needs a workflow where you pull in a dependency, and work on it in tandem with your code. Achieving a good workflow for this is surprisingly hard.
Well put! I've struggled with this exact situation, and although I figured out a setup that works for me, it's still not ideal. In my experience, npm the package manager doesn't enable such workflows reliably (yet).
In one of projects I needed foo::string that can share data with foo::variant without copying the data each time. So foo::string was implemented as a COW string - smart pointer for foo::string_data. std::string simply does not work in such requirements.
So I am not sure I understand how cargo will help in this case. Either Cargo source repository will contain implementations of any possible permutation of requirements of strings or people will just use standard std::string.
The only feature that I need in C/C++ is unified ability to include libraries in code:
#define PNG_APNG_REQUIRED
#include source "libs/png/png-amalgamated.c"
In any case decision to include library to a product requires quite a lot of reasonings and architectural investigations.
For typical web front-end projects NPM or Cargo probably make sense. But for, say, NodeJs or Cargo itself they should not use any such automatic downloader.
Basically, lifetimes are allocators as a first-class language construct. You can reason about what happens if values are stack or heap allocated and specialize your code based on that. Lifetimes are definitely an advanced feature and I think you can do some crazy optimizations with them. But your use case is not hard to implement. If you show me the C++, I'll show you how to achieve the same semantics.
Cargo helps because Rust lets you build cleaner abstractions and abstractions that compose nicer. So integration is super easy.
> For typical web front-end projects NPM or Cargo probably make sense.
I might misunderstand your sentence, but neither Rust nor cargo are for front-end?
If their complaints are about GC tuning, is it not the same thing as tuning the GC in Js/Go? Java still had arguably the more mature GC of any language
> The process of deploying the new Rust service was straight-forward, and soon they were able to forget about the Rust service because it caused so few operational issues. At npm, the usual experience of deploying a JavaScript service to production was that the service would need extensive monitoring for errors and excessive resource usage necessitating debugging and restarts.
Is this satire?
I think the satire here is that internet got so centralized lately that even a simple piece of code in JavaScript requires such a huge behemoth of an org running and maintaining all this monstrous infrastructure.
I think a good first pass would come from studying analytics from npm. What are the most used packages? The most stable? I know lodash makes a lot of sense but there’s also underscore. I think the biggest hurdles are really political over technology as everyone has been so entrenched now that a one-size-fits-all stdlib would be hard. Not impossible, just hard. I do wish someone were working on it and I hate to say it but Google probably has the most skin in the game with v8 and Chrome yet I don’t really trust them not to abandon it. So who else is there? It wouldn’t be a very ‘sexy project’ either but still seems worth it to at least try.
I think it would also make a lot of sense to look at what's in the Python and Ruby standard libraries.
Not having a standard library on par with other scripting languages just added overhead and incompatibility for years as people invented ad hoc alternatives, often buggy. The accelerated core language growth has been hugely helpful for that but you still have issues with things as basic as the module system which exist for understandable reasons but are just a waste of developer time.
The current human population of earth is about 7.7 billion, so that number should probably be closer to 1.17 npm packages per week per human being. That is still quite a lot, though
And perhaps the CI server folks would want this anyway because it would be vastly faster.
- NPM is intentionally suited to lots of small libraries that do one thing and do it (hopefully) well, and composing those libraries in useful ways. Whereas systems like Debian have scaling limits with large numbers of packages, NPM tries hard to avoid this so that one hundred ten-line packages are as reasonable as a single thousand-line package.
- CI systems aim for reproducibility by deploying from source and having declarative configurations, in much the way that most distro package builds happen in a clean-room environment.
is the actual quote.
And: it is incredible easy to build and deploy.
I think whether Rust is useful or not depends entirely on the application. If you need high confidence in what the thing is doing, it should run parallel and fast and you are familiar with the concepts Rust is using – it isn't a bad choice. For me it replaced Python in nearly every domain except one-use scripts and scientific stuff.
It can be hard for advanced programmers to abandon certain patterns they bring from other languages though. In the first months I tried too much to use OOP, which doesn't make any sense and leads to convoluted code. If you work more in a compositional and data oriented way while making use of Types and Traits, you will end up with much simpler solutions that work incredibly well.
Katherine West's RustConf 2018 Talk on ECS Systems describes this incredibly well and might be even good to watch if you are never intending to use Rust at all, because the patterns discussed are quite universal: https://www.youtube.com/watch?v=aKLntZcp27M
Oh, well, you don't need Rust for that! :P
The other part is that component oriented programming is actually a branch of OOP, from CS point of view, with books published on the subject at the beginning of the century.
"Component Software: Beyond Object-Oriented Programming"
https://www.amazon.com/Component-Software-Beyond-Object-Orie...
First edition uses Component Pascal, Java and C++, with the 2nd edition replacing Component Pascal for C#.
>At npm, the usual experience of deploying a JavaScript service to production was that the service would need extensive monitoring for errors and excessive resource usage necessitating debugging and restarts.
But if you factor in over the life time of the program, Where Node saves you a week times at the initial implementation and you paid back in extensive monitoring, it is probably safe to say Rust's TCO is much lower.
Not Sure how Rust will flare against Go. But I think there is a high probability that Rust is better in the longer run.
I guess it will be domain dependent. Go uses a highly-developed concurrent GC, which is going to make it a lot more convenient for certain specialized workloads that involve graph-like or network-like structures. (That's the actual use case for tracing GC, after all. It's not a coincidence that garbage collection was first developed in connection with LISP. And yes, you could do the same kind of thing in Rust by using an ECS pattern, but it's not really idiomatic to the language.)
I'm really skeptical of this unless it's just a wrapper for a thing that happens to already exist. It would be interesting to have comparative LOC numbers.
> At npm, the usual experience of deploying a JavaScript service to production was that the service would need extensive monitoring for errors and excessive resource usage necessitating debugging and restarts
So, they deployed it after an hour, but it wasn't finished until they stopped having to debug it in production?
deploy anyways
Sure, but when you consider the drastically reduced operational cost that they're talking about there... that week is absolutely peanuts in comparison, and that was also a week including getting to grips with the language sufficient to produce the component. You really don't want to have to pay attention to production. You want to be able to concentrate on getting stuff done, not losing time keeping what you've already got just ticking along.
How so?
And that the operational improvement once written in Rust was notable enough to write a paper.
Imagine the K8S team porting from Go to some other language for similar reasons.
I don't see how any of this is critical of JavaScript, which isn't even really discussed in the paper and still runs the rest of the infrastructure. If anything the paper is more damning of C, C++, and Java but I still think damning is far too extreme to describe what the paper said.
I don't see it as damning on either of these. On C++ it says "we didn't want to learn it." Which is fine. Maybe after learning they would have decided different, or not. On Java they said "we didn't want to learn how to operate it" as they feared the complexity of an Java application server for a single small service, which they can create in a way which hooks into their monitoring infrastructure. No damning their either.
However their company's purpose is to push Javascript and they are saying "operating JavaScript is hard, doing this in Rust is easy" which directly goes against their business.
Use the right tool for the job. The PHP website is running PHP, but it isn’t running a web server written in PHP. Web servers are system software; PHP isn’t for writing system software. Same thing applies here. Node does fine with business logic, but isn’t really architecturally suited to serving the “data layer” when you have control/data separation.
Also, I would assume any crypto in v8 is already written in C, with JS calling into it.
The rust implementation was more efficient than the JS one. A CPU bound service of course is bottlenecked at the CPU, and this benefits from efficiency.
At scale, it makes sense to replace this with Rust. Javascript did the job, but did not provide the same efficiency as Rust.
Except the things that end up being memory-bound instead, but the NPM database isn't large enough for that.
Not in a relative sense. If authorization is 5% of the work, scaling it leaves it at 5% of the work, and it's never a bottleneck. Authorization was being a significant bottleneck, not a tiny percent, and that is somewhat surprising.
Perhaps the new worker threads may alleviate this, but I'm not sure (it's still an experimental API).
Are you sure? I'm not familiar with the PHP.net architecture, and there may be less gains from how PHP has traditionally tied itself as a module to web servers in the past, but Rails (and any number of other dynamic language frameworks) are actually web servers implemented in that language, with an optional separate web server such as NGINX or Apache you can run in front to handle the stuff they aren't as good at (static file serving, etc).
Now, that is a framework, and not the language proper, but I wouldn't be all that surprised to find python.org running on top of a Python framework.
"NOTE: Some of our maintainers prefer to use web servers other than Apache, such as Nginx. While this is permitted (as long as everything ultimately works as directed), we do not officially support these setups at this time"
k8s was originally written in Java, so they have already done this once.
There is a FOSDEM 2019 talk about it.
"The clusterfuck hidden in the Kubernetes code base" https://fosdem.org/2019/schedule/event/kubernetesclusterfuck...
> We look at what it would take to begin undoing the spaghetti code that is the various Kubernetes binaries
Well at least the developers are being frank about it I guess.
> The audience walks away feeling empathetic that they aren’t alone in their journey to writing idiomatic Go and is now equipped with strong refactoring techniques developed by some of the world’s top engineers for the Kuberentes project.
As an occasional user of kubernetes, minikube etc. it's not something I would have guessed to have been developed by the word's top engineers.
I mean kubernetes tries to, and probably manages to provide an useful abstraction, but at a few million LOC and a few man-month of full time senior engineering effort to run anything in production it's not exactly a epitome of elegant and efficient engineering.
https://adtmag.com/articles/2019/01/17/project-marble.aspx
If you want to have some reading fun, check /r/androiddev/ every time there is a "stable" release.
The interesting thing with Kubernetes is that it's basically a re-imagining of Borg, which one assumes was not a few million lines of code when it was already running all of Googles infra more than a decade ago. It's obviously not solving the exactly same problem (e.g. Google correctly recognized that DNS isn't so hot and wrote their own replacement protocol, BNS which wouldn't fly for external adoption etc.). But I'd be curious to know how Borg's code quality and size back when it became the standard way to run stuff at Google maybe 12 years ago compares to Kubernetes today.
Every projects of that scale are going to have issues.
Is this maybe parsing data structures itself?
Publishing is a relatively rare event compared with reading, but in a world of private packages, orgs, and teams, the "can {user} read {object}" gets more complicated. It probably wouldn't be CPU bound if not for the sheer scale we're dealing with, but once all the IO bottlenecks are resolved, you still have to check to make sure that a login token is valid, then get the user associated with it, then check the teams/orgs/users with access to a thing (which might be public, in which case, the problem is a lot simpler, but you still have to get that info and check it), and then whether the user is in any of those groups. So there's a straightforward but relevant bit of CPU work to be done, and that's where Rust shines.
Resource usage is similar though not exactly aligned e.g. Haskell has significant ability to statically enforce invariants and handle error conditions, but the complex runtime and default laziness can make resource usage difficult to predict.
I'd guess OCaml would also have done well in the comparison as it too combines an extensive type to system which is difficult to bypass with an eager execution model.
It will take a fair amount of time to be proficient in Haskell to the point where it is not a (potential) big problem.
Which creates a new barebones rust project called "funkyproject". Every dependency specified in it's Cargo.toml will be automatically downloaded at build (if there is a new version).
When a build is sucessful the versions of said dependency will be saved into a Cargo.lock file. This means if it compiles for you, it should compile on every other machine too.
A cargo.toml allows you also to use (public or private) repositories as a sorce for a library, specify wildcard version numbers to only select e.g. versions newer than 1.0.3 and older than 1.0.7 etc.
Because the compiler will show you unused dependencies you never really end up including anything you don't use. In practice this system does not only work incredibly well, but is also very comfortable to use and isolates it self from the system it is running on quite well.
I really wish Python also had something like this. Pipenv is sort of going into that direction, but it is nowhere near cargo in functional terms.
EDIT: I misunderstood and thought you were talking about installing a package. If you're running `npm install` to just reinstall dependencies then yes the lockfile should not be modified. However it seems like that is indeed the case and you may be talking about a prior bug with NPM.
For the passers-by, the only way to make npm behave expectedly in this specific case is to use "npm ci" instead of "npm install". If you do not do this, npm will assume you want to update the packages to the latest version at all times, at all costs, even if you have a lock file in place, and even if you have your package file and lock file locked to exact versions. (i.e. 2.0.0 exact, not ^2.0.0)
This is a new addition, and it has been added a couple months ago. Before that, you had to check your dependencies into your source control. That might still be the best practice, and likely the only trustable way to get reproducible builds consistently over a longer time horizon.
Wait what? Are you sure about that part? That's a violation of npm's semver constraints https://semver.npmjs.com
(I agree with you that "npm ci" should be the default behavior, and "npm install" should be called something different, like update-and-install)
As a result, when you do a npm install in your oblivious and happy life, npm naturally assumes you want to summon Cthulhu. If you didn't want to summon Cthulhu, why did you call the command that summons Cthulhu? Yes, the default command summons Cthulhu because we believe in agile Cthulhu. If you don't want to summon Cthulhu, try this undocumented command with a misleading name we've added silently a few weeks ago for weird people like you who don't want to summon Cthulhu when they want to do a npm install. But seriously, why do you not want to summon Cthulhu?
Unfortunately, this was the impression I've gotten of the position of npm folks when I read a few threads about this. I've moved to npm ci for now and moved on. Npm's package lock is many things, however, none of the things it is, is a package lock.
`npm ci` is optimized for a cold start, like on a CI server, where it's expected that `node_modules` will not be present. So, it doesn't bother looking in `node_modules` to see what's already installed. So, _in that cold start case_, it's faster, but if you have a mostly-full and up to date `node_modules` folder, then `npm install` may be faster, because it won't download things unnecessarily.
Another difference is that `npm ci` also won't work _without_ a `package-lock.json` file, which means it doesn't even bother to look at your `package.json` dependencies.
For example, why did this person experience the changing lockfile? https://github.com/npm/npm/issues/17101
Or why do these docs say?
> Whenever you run npm install, npm generates or updates your package lock https://docs.npmjs.com/files/package-locks
Oh, this seems like what I experienced: https://stackoverflow.com/a/45566871/283398
It does appear that npm works somewhat differently than the “obvious” way we would expect package managers to work vis a vis lockfiles :(
At least npm ci gets the job done for my use case :)
In the bug linked, they wanted to install a specific package (not matching what was in the lockfile), without updating the lockfile. That's what `--no-save` will do.
The SO link is from almost 2 years ago, and a whole major version back. So I honestly don't know. Maybe a bug that was fixed? If this is still a problem for you on the latest release, maybe take it up on https://npm.community or a GitHub issue?
This is not correct. `npm install` will update your dependencies, not install them, disregarding the package versions defined in the lock file.
It feels like you are not getting the point of having a lock file in the first place. It should be obvious that you can't do an install (which npm calls ci) if you don't have a lock file.
The lock file represents your actual dependencies. Package.json should only be used to explicitly update said dependencies.
$ json dependencies.esm < package.json
^3.2.5
# package.json would allow any esm 3.x >=3.2.5
$ npm ls esm
tap@12.5.3 /Users/isaacs/dev/js/tap
└── esm@3.2.5
# currently have 3.2.5 installed
$ npm view esm version
3.2.10
# latest version on the registry is 3.2.10
$ npm install
audited 590 packages in 1.515s
found 0 vulnerabilities
# npm install runs the audit, but updates nothing
# already matches package-lock.json
$ npm ls esm
tap@12.5.3 /Users/isaacs/dev/js/tap
└── esm@3.2.5
# esm is still 3.2.5
$ rm -rf node_modules/esm/
# remove it from node_modules
$ npm i
added 1 package from 1 contributor and audited 590 packages in 1.647s
found 0 vulnerabilities
# it updated one package this time
$ npm ls esm
tap@12.5.3 /Users/isaacs/dev/js/tap
└── esm@3.2.5
# oh look, matches package-lock.json! what do you know.
I often don't know what I'm talking about in general, but I do usually know what I'm talking about re npm.
Why do people want this? The builds are no longer reproducible, security and edge case issues can come out of nowhere, api changes from an irresponsible maintainer can break things, network and resource failure can break the build, it's just a terrible idea.
The proper use of a semvar system is entirely optional and unenforceable and seen people been bitten countless times by some developer breaking their package and having everyone complaining ... If the tool didn't do the stupid thing of just randomly downloading crap from the internet none of this would be a problem.
I presume all my dependences are buggy...I just know that the current ones don't have bugs that I have to deal with now. You swap out new code and who the heck knows, it becomes my job again. It's more work because of a policy that doesn't make sense.
Newer code isn't always better. People try new ideas that have greater potential but for a while the product is worse. That's fine, I do it all the time. But I sure as hell don't want software automatically force updating dependency code to the latest lab experiment.
Cities, power plants, defence systems, satellites, and airplanes run on software from the 80s; they don't break because a new version of some library had bugs and it automatically updated, no. They fucking work.
There's a giant inherent huge irreplaceable value in predictability and this approach ignores all those lessons.
Rust is no worse here than say Haskell with cabal or stack or Swift with whatever they were using I forgot or go for that matter.
If you update the versions in your Cargo.lock are ignored and updated if the build is sucessful.
If you add a dependency only that depndency is downloaded, the rest is kept as you had it.
If you clean it is as if you cloned that project fresh with git and you will have to download all dependencies. If there is a lockfile the exact versions from it will be used.
To me this is extremely flexible and works very well AND you get precise control over versions if you want it. By the way it is also possible to clone all dependencies and keep a local copy of them, so you are really 100% sure that nothing could ever change with them. Although I am quite sure crates.io doesn't allow changes without version number change, which means you should be save as long as you rely on the version number.
It allows specifying dependencies with much of the same freedom you mentioned, in an environment.yaml file and other config files, you can provide arbitrary build instructions via shell scripts, use a community led repository of feeds for stable and repeatable cross-platform builds of all libraries [1], generate the boilerplate automatically for many types of packages (not just Python) [2], compiled version specifics with build variants / "features", and you can use pip as the package installer inside a pip section in the conda yaml config file.
[0]: https://github.com/conda/conda [1]: http://conda-forge.org/#about [2]: https://conda.io/projects/conda-build/en/latest/source/resou...
The article is hosted at rust-lang.org so one would be wise to take their words with a grain of salt. And Rust isn't even a tiny fraction as popular as JavaScript and so when you use Rust you're choosing from a small set of packages written by experts and the kind of people who use languages that nobody else really uses. Meanwhile, JS has millions of packages written by everybody for various platforms (since JS can run in all sorts of environments where nobody would ever want to run Rust.)
Also there's the anecdotal, yet easily empirical evidence that just about any developer who uses JS can tell you about: I deploy new JavaScript services all the time without any of those problems.
So, I wonder if you're actually asking this question or if you have some other agenda.
What kind of load do your node services get? I’d be willing to bet the npm registry has more. That plays into this kind of thing.
Rust definitely has some benefits as well as tradeoffs when compared to JavaScript which they discuss. Learning curve is higher but the end product is probably devoid of a number of errors and operational issues over the lifetime of the service. While in theory possible to get similar results with JavaScript the level of consistent discipline it requires is in practice impossible.
These are facts not opinions.
I would say Rust is both a strongly typed language and a statically typed language. The static type checking happens at compile time, and in general the types in use are strict and strongly typed at runtime.
But, even Rust allows you to cast types from one to another and use dynamic types determined at runtime.
Yes, most people would say that static typing is the primary advantage you get from the compiler in Rust.
There's a great writeup by one of the C# people (I want to say it was Erik Meijer, but I'm having a hard time finding it atm) about the distinctions we're discussing here, their relevance to correctness, and the impact on ergonomics. My takeaway from it was that occasionally you will encounter problems that are easier to solve with some freedom and that's why strong/static languages like C#/Rust include pragmatic escape hatches like the dynamic object and the Any trait (respectively).
If you do find that writeup, I'd be interested in reading it.
I'm not sure why you think this has changed today, or what you mean. It appears to me that many programmers don't realize that the two are orthogonal, so I find it an important, not pedantic, distinction (it just happens that languages generally improve on both fronts over time, hence asking for one also gives you the other, but it's because of correlation, not causation). I'll make an attempt at describing it here, please tell if I'm missing something.
Strong typing means that types describe data in a way that the data won't accidentally be mistreated as something else than what it represents. E.g.
(a) take bytes representing data of one type and interpret it as data of another type (weak: C; strong: most other languages)
(b) take a string and interpret it as a number without explicitly requesting it (weak: shells, Perl; strong: Python, JavaScript, Ruby)
(c) structs / objects / other kinds of buckets, (strong: C if type wasn't casted; weak: using arrays or hash map without also using a separate type on them that is enforced, the norm in many scripting languages although usually strengthened via using accessor methods which are automatically dispatched via some kind of type tag; also, duck typing is weaker than explicit interfaces)
(d) describe data not just as bare strings or numbers, but wrap (or tag / typedef etc.) those in a type that describes what it represents (this depends on the programmer, not the language)
(e) a request of an element that is not part of an array / list / map etc. is treated as an error (similar to or same as a type error (e.g. length can be treated as being part of the type)) instead of returning wrong data (unrelated memory, or a null value which can be conflated with valid value)
These type (or data) checks can happen at runtime ("dynamically") or compiletime ("statically"). The better a static type system is, the more of these checks can be done at compile time.
For security and correctness, having strong typing is enough in principle: enforcing type checks at runtime just means getting a failure (denial of service), and systems should be designed not to become insecure or incorrect when such failures happen (fail closed), which of course might be done incorrectly [1]. Testing can make the potential for such failures obvious early (especially randomized tests in the style of quickcheck).
Static typing makes the potential for such failures obvious at compile time. It's thus a feature that ensures freedom of denial of service even in the absense of exhaustive testing. It can also be a productivity feature (static inspection/changes via IDE), and it can enable more extensive use of typing as there is no cost at run time.
[1] note that given that static typesystems usually still allow out of memory failures at run time, there's usually really no way around designing systems to fail closed anyway.
I personally don't fret about it unless we get into specific details about these notions. I do especially like your (d), which many people often overlook when designing programs. An example would be to use a String as the Id in a DB, but not wrap the String in a stronger type to represent the Id, thus not getting the advantage of static type checking by the compiler. So there are definitely areas where this conversation can lead to better advantages of different languages.
For example, in Rust declaring a type to be a `struct Id(String);` would cause no overhead to be associated with the Id in terms of memory allocation to that of just a String. Not all languages can say that, thus we could also get into a fun conversation about the overhead associated with the type system itself. All fun topics.
Perhaps Rust's name is apropos: your code will be so reliable that you won't need to look at it again until it has collected rust on its thick iron framework.
Edit: this seems like I’m suggesting rust makes shoddy results. Didn’t mean to imply that. I’m actually very excited to use Rust in prod soon.
The JS dependencies:
"gulp"
"gulp-clean-css"
"gulp-postcss"
"gulp-uglify"
"autoprefixer"
"postcss-uncss"
"uncss"
So I'm not at all surprised that this might create surprises when deploying JS services in production.
I wrote trumped.com and deployed it prior to the last presidential election. The frontend and assets have been redeployed, but the core rust service for speech generation hasn't been touched. I've never had a service this reliable, and it took so little effort!
Rust is the best language I've ever used, bar none, period. And I've used a countless many of them.
The only places where I won't write Rust are for small one-off scripts and frontend web code. (Even with wasm, Typescript would be tough to beat.)
Scalability is an architectural concern which cannot be ignored by system developers. This is because scalability is not about speed or performance, it's all about figuring out which workloads can be split up and executed in parallel; in order to do this, you need to understand the real-world problem which the software is trying to solve; this is not something that you can delegate to a compiler.
The best that a language can offer in terms of scalability is to make it easier to reason about parallel workloads and make the difference between serial and parallel workloads as explicit as possible. Whenever a language tries to hide the complexity of parallelization behind thread pools, they're not solving any real scalability issue; they're just delaying them some more.
True.
> A language cannot solve scalability issues; all they can do is push the needle a tiny little bit further in terms of performance but this is completely meaningless.
> Scalability is an architectural concern which cannot be ignored by system developers.
A language can prevent or delay such architectural concerns from being addressed by not offering sufficient capabilities.
They have spoken about this publicly before; it was in their newsletter when they first deployed it, and there’s been a couple of conference talks.
> Most of the operations npm performs are network-bound and JavaScript is able to underpin an implementation that meets the performance goals. However, looking at the authorization service that determines whether a user is allowed to, say, publish a particular package, they saw a CPUbound task that was projected to become a performance bottleneck.
Oh, Really ???
So essentially Authorization service and I doubt the security algorithms computation are the main cause.
What i dont understand here is why is it not possible to write lower level JS or asm code to craft a well optimized code which V8 can totally nail to minimum CPU instructions required?
The server was hammered at maybe 500 signatures a second, greatly slowing down token generation.
I'm guessing they dropped to Rust so they could use native C libraries for signature generation.
This is actually one of the reasons that I don't think Go is a very good language for most people/organizations. It was conceived with a specific set of guidelines that Google needed; easy to learn, performant, etc. Go is also designed to be used by teams of thousands, so it's much easier to adopt misc packages into the fold and maintain them than it is to manage links to outside requirements.
> npm called out the Rust community as a positive factor in the decision-making process. Particular aspects they find valuable are the Rust community’s inclusivity, friendliness, and solid processes for making difficult technical decisions. These aspects made learning Rust and developing the Rust solution easier, and assured them that the language will continue to improve in a healthy, sustainable fashion.