Show HN: Auto install npm dependencies as you code | Hacker News Reader

Show HN: Auto install npm dependencies as you code

npmjs.com

60 pointssiddharthkp77 comments

I mean, wow. I've seen half-baked ideas before, but this takes the cake. Typos now equal remote code execution, and it's even easier to enter dependancy hell, because deps are now implicit.

The whole node ecosystem is a mess. Shrinkwraping is a pain, easily bottoms out due to node's small default stack limits, and often the only solution is to `rm -r node_modules` and rebuild the dependency graph from scratch.

Typos with npm have always allowed RCE. You can have typos while doing $npm install ....

Right, but when you are running npm install you know you are installing and so you know you should be careful, and only hit return if you are sure it is right.

With this, you have to be careful the WHOLE time you are typing. You normally don't worry about hitting return in your editor causing RCE.

Good point, any ideas on handling this better?

Opened an issue here: https://github.com/siddharthkp/auto-install/issues/2

I don't really know if there is a way to fix this issue... unless you had some sort of whitelist of acceptable packages, and a prompt if you try to install a non-whitelisted package.

One other possibility is to have a delay, so that it waits an amount of time before installing, to give a chance to catch and fix the typo.

Added a --secure flag which will install popular modules only (>10k downloads last month)

Good idea!

The idea is inherently flawed. depandancy installation should be done as needed by people who are conciously aware of what they are doing. A better solution would be integrating NPM into your editor, so that package installation is trivial, but still explicit. M-x install-from-npm-region, anyone?

  (defun install-from-npm-region (start end)
    (interactive "r")
    (call-process 
      (concat "npm install " 
              (shell-quote-argument 
                (filter-buffer-substring start end)))
      nil
      (get-buffer-create "*NPM Output*")))

That function MIGHT install random things through NPM if your mark is unset, but it is, at least, explicit. And the random thing in question is your last region, so it's far more likely that NPM will just throw its hands and shout "what the hell?" Than actually install anthing...

Also note that that code hasn't been tested. I wouldn't copy this into my .emacs just yet, if I were you. Run it by somebody who actually knows elisp, first.

If I understand correctly, this doesn't happen while you type, it happens after you save the file. Not much difference in care needed between saving a file or hitting enter IMO, if you are aware of the consequences. Also, its not like you don't know the thing is running - you'll subconsciously double check you typed the repo correctly before you save, knowing it will invoke an install.

Unless your IDE, like many, auto-saves files.

Even Emacs'll do that.

Indeed. If there's one thing we learned about node dependencies this year it's that nothing can go wrong if you take dependencies without thinking about them.

Exactly how is any of this different than installing via the command line?

Because it installs based on what you type, if you accidentally type something like `expres`, which has some kind of vulnerability or is malicious, you won't have time to notice the typo before it installs it. It could happen when doing it manually, but it's less likely because it doesn't install based on looking at your code but based on your `npm install` command.

I'm not buying the argument. I often hit [npm install] 20 times a day, and sometimes you get distracted, or just sloppy at the end of a long week. A nice feature of this, is in vim I'd have a dict of common modules, and then have the autocomplete autocomplete their names for me. That way you get in a routine where if the library isn't in the autocomplete, you need to take a second and think about it. So nothing wrong with the tool, it's just a tool. Use a flamethrower carelessly, you're going to have a bad time.

I actually looked at the code for 5 minutes. This module is vulnerable to arbitrary code execution simply by having a malicious JavaScript file in a directory or subdirectory underneath where `auto-install` is run, without the user even needing to make a typo due to its design.

The documentation currently just says 'avoid typos'.

siddharthkp: please give me a way to contact you. see my contact info on my profile.

>The documentation currently just says 'avoid typos'.

So their security model is basically the same as with C: "do it perfectly the first time."

They have an email address on their profile page.

Really? I don't see it.

https://news.ycombinator.com/user?id=siddharthkp doesn't return an email address, and I don't see one on the GitHub profile page either.

https://www.npmjs.com/~siddharthkp

Thanks. Reaching out.

DM on twitter? I'm @siddharthkp

Your DMs are closed.

Sorry about that, fixed.

Lots of people addressing security and remote code execution by typo.

Yes. You're right. If it scares you, don't use it.

It's always possible to run malicious code by typo, and this is only a little different from installing dependencies from the terminal. Even when you do spell a package's name correctly, you still don't know for sure what you're installing.

The guy just made a cool thing - it seems a little out-of-scope to freak out over security when npm was never really there.

Thanks for the support :)

The concerns are fair though, just added a --secure flag which will install popular modules only (>10k downloads last month)

Regular expressions are a very naive way of detecting calls to 'require'. For correctness you're better off recursively walking the AST.

I've successfully used Detective in a couple of my personal projects to find all require statements.

Relevant issue on Detective: https://github.com/substack/node-detective/issues/8

The AST? What AST is that? You don't get access to an AST.

The AST you get from any of a wide variety of parsers, of which esprima may still be only the most popular of many.

It has to open and read a .js file already, it can certainly turn that into the representative AST for said file and then use the data from that. It will be slower, but it will also be more accurate and less likely to turn up false positives or miss things.

It has to parse a javascript file, which isn't trivial. The reason they use regular expression is because implementing a javascript parser isn't an easy problem to solve fast, even though the grammar is available.

You downloading megabytes over network and performing disk IO with it. What's the problem with 100 milliseconds of file parsing?

One certainly does not need to implement a parser, just use one of the many available. As the sibling here pointed out the time it would take to parse and walk the AST is negligible compared to the downloads happening.

Tools like acorn[1] and Esprima[2] can parse the JavaScript source and output an ESTree compliant syntax tree.

Then you can traverse it and modify it like any other AST.

[1] https://github.com/ternjs/acorn [2] http://esprima.org/ [3] https://github.com/estree/estree

In practice regex gets all the require calls. And in many situations the very small percent of errors is outweighed by the huge advantage in performance that a simple regex provides.

So i wouldn't call regex a 'very naive' choice, since people making it are aware of tradeoffs and pick regex intentionally.

It literally takes about 50ms to parse libraries like jQuery (mobile), angular and React with acorn[1].

Wether the choice was deliberate or not is debatable, but given the speed of these parsers I'd reason that there isn't any advantage to using regexes.

Reducing the amount of false positives is also one step closer to making this tool somewhat more secure, though certainly doesn't address any of the previous comments in this thread.

[1] http://esprima.org/test/compare.html

Good point, let me look into that.

Very cool! The Webpack equivalent: https://github.com/ericclemmons/npm-install-webpack-plugin

Sweet!

So I can make a malicious module called expres and another one called expresss and screw with ppls machines?

This.

Installing modules from npm is dangerous enough. Nice for education or playing around, unsuitable for a serious developers' workstation.

This.

I love that the Node community enjoys innovating for convenience, but ideas like this one are less than half-baked from a security perspective. Just make a few typo'd popular packages, and use npm install scripts [1] and you have a very easy remote code execution vector on developer workstations.

The bigger problem I see is that npm is a circus. No package signing and a ridiculous debate on why that's been going on for a year and a half [2]. Credentials leaks of popular modules. [3]

When everything is a module and everyone is supposed to include modules vs. writing their own very simple functionality for things even like isArray polyfills [4] (24MM downloads a month!), you end up with the same attack surface that gives WordPress such a shitty reputation for security. It's not usually core, it's all the plugins by authors of unknown provenance and skill. WordPress gets pwned because there are a lot of plugins hastily written by new developers and used without audit by mom-and-pop web app shops and/or those that trust the code because they aren't capable of auditing it meaningfully.

When you use an npm dependency, you are taking on all of their dependencies. You are trusting they don't leak creds, that npm has not been compromised, and that the chain underneath has been audited for malicious behavior. In reality this is impractical: go npm install express and see just how deep the dependency chain goes. Things like Snyk are required to just understand what might be vulnerable.

[1] https://docs.npmjs.com/misc/scripts

[2] https://github.com/node-forward/discussions/issues/29

[3] https://github.com/ChALkeR/notes/blob/master/Do-not-underest...

[4] https://www.npmjs.com/package/isarray

EDIT: Ironically, this module itself is vulnerable to code injection.

> It's not usually core, it's all the plugins by authors of unknown provenance and skill.

This is otherwise known as an active developer community and is a good thing. In any open library ecosystem, it's ultimately up to the developer to carefully choose and vet third-party modules. There isn't any substitute for that.

The alternative is a tightly controlled standard library, but that isn't npm's stated goal. Such a controlled, curated, audited standard library is, however, something that could be built on top of npm, but obviously not vice versa.

So npm being a circus is, in the grander scheme of things, a good thing. Novice programmers will necessarily produce novice code.

edit: if it wasn't clear, I completely agree about the security risks of this project.

What kind of vetting is occurring when you're automatically installing npm modules as you code?

But, "Novice programmers will necessarily produce novice code."

Having had my own variety of experience with CPAN, PEAR, Tcl's package thing, C by way of building my own RPMs and DEBs, and now watching the ongoing trainwreck that is npm, yes. You're right about that.

> The alternative is a tightly controlled standard library, but that isn't npm's stated goal.

What npm says it is doing on paper and in its charter is not necessarily what npm gets used for. At this point in the ecosystem's maturity, npm developers are doing their users a massive disservice and opening them up to a lot of risk. Maintaining this line of a vibrant active community, and "developers should be responsible for their own security", rings hollow.

As I see it, npm appears to be acting like there are a lot of unsolved problems in this realm, and in doing so are endangering a developer community that is absolutely full of amateurs.

I would argue that JavaScript is the most accessible language of the current era, and Node.js is being pushed by a lot of third parties as the new easy way into programming. These new guys don't know security and want to contribute to the community, and the npm gatekeepers seem to be doing little to stop naïveté from causing virus-like consequences in their dependency chains. When I was a teenager writing code I had no idea what security issues I needed to patch against. I wasn't educated on it and didn't care. I just wanted to make cool stuff. The problem with npm is that the cost of entry of your "cool stuff" into the hands of a thousand trusting others is too low; there is no delineation between what is authoritative and what isn't.

Debian solved this problem years ago. Restrict npm defaults to vetted packages, and have people add repositories as need be (e.g. multiverse, Ubuntu PPA, etc.) for packages that aren't audited or by trusted parties.

The Node user experience often ignores most security issues for ease of use; this is OK when you are guarding people against the most likely mistakes. This is a problem of setting insecure defaults and expecting the Internet to play nice. The user experience, the marketing message, and the community at large defends its openness to the death. I'm all for openness, but at some point senior developers should be attempting to shepherd their new developers into making secure decisions, and thinking in a way that is somewhat security minded.

It's taboo in npm culture (as in, the actual npm employees) to stop anyone at the gate. In fact, they actively encourage experiments of all kinds into it.

> As I see it, npm appears to be acting like there are a lot of unsolved problems in this realm, and in doing so are endangering a developer community that is absolutely full of amateurs. > The problem with npm is that the cost of entry of your "cool stuff" into the hands of a thousand trusting others is too low; there is no delineation between what is authoritative and what isn't.

I agree that npm has been a bit slow with a bunch of important features like package signing, sandboxing post-install scripts, etc. but as a counterpoint to the authoritativeness issue, I would argue that vetting and defining "authoritative" packages is a difficult problem. I'm not aware of any open/semi-open package ecosystem that has solved this problem (please do correct me if I'm wrong).

As an example in the JS world, which of lodash/ramda/underscore/functionaljs should be the/an authoritative javascript FP library? Should they all be marked authoritative? If so, what is the criteria for a new library to also be authoritative? What happens when a library is abandoned? How do you even define abandoned in an open ecosystem?

These are solvable problems, but not easy ones to reach consensus on.

The Redhat-like alternative is to have a central entity employ/pay contributors to audit and maintain libraries, but it's debatable whether npm would have grown to its current size with that model.

Lodash is not a "FP" library—it's just a utility belt. And yes, it should be the authoritative IMO as it has the most support, users, and is worked on almost full time. It can be modular and each method can be installed separately, which is awesome.

Ramda is a utility belt that sticks to pure functional practices wherever it can, something the JS community doesn't do, so it shouldn't be the authoritative.

Underscore is dead and was replaced by Lodash.

While it's hard to do the above with all kinds of libraries, there are some where it's easy.

Thank you, I knew I wasn't the only who thought this.

This is one of the benefits of using a container-based system like the one we designed for https://tonicdev.com . On Tonic, you get the same conveniences displayed here (actually better since all the packages already exist, so no install-wait time), but the entire development process happens in a container. Once you're ready to move on, you can download the project with shrink-wrap to get identical behavior, but until then you can more freely try whatever you'd like.

I wonder what https://tonicdev.com/ does.

Can you turn off the uninstall feature? I envision, "Nooooo, I was just refactoring! The network here is so slow noooooo please!"

Maven knows I've made the mistake of triggering a big download on a bad network.

Sure!

Opened up an issue https://github.com/siddharthkp/auto-install/issues/1

Given that in npm v3 the order of installation determines where things are installed, this seems like it might cause some interesting bugs that never happen on the developer's machine. :)

I'm not sure how this could happen as one shouldn't be depending on the directory structure in the node_modules/ folder anyway. Do you have an example of where this might cause issues?

Well, one shouldn't... :)

I don't actually know that there's any problem that `rm node_modules; npm install` won't solve, since that would make things the same as a fresh install.

All of the scenarios I could come up with were about long-running installs, rather than new deploys, so more likely to happen on dev than prod.

ETA: actually wrong, see below.

I can't think of any scenario when one would be relying on a specific structure being in the node_modules/ directory. And if there is such a need, then it's likely the code is doing something very strange or very special in which case this kind of package would not even be considered during creation (and I still think wouldn't cause problems).

Even in long-running installs I still don't see a potential issue here. Do you have a small specific example you can think of?

Package A depends on lodash: ^4.6.1

Package B depends on lodash: ^4.14.2

... you know what? I just installed semver to check my beliefs, and it turns out I'm just spreading FUD. Caret doesn't work the way I thought it did.

Please disregard.

Wow, I'll have to look that up. Thanks!

Dependency installation should be a bit frictional...

That's interesting, tell me more?

A typo should not start installing random software on my machine.

I could be wrong, but I think he might mean that people should take care not to add too many dependencies to their project - if it's too easy, then it might result in unnecessary dependencies and brittle code.

However, I'm not sure I agree with the statement - you could use this tool and still have the discipline not to pull in random packages.

The idea is that people in general go down the currently-easier way, which is "add the dependency", leading to microdependencies and left-pad idiocy in npm case.

If there is a friction, the balance is changed a little against pulling dependencies, at least those most trivial.

This!

In any application with a lifecycle longer than your average "TodoMVC example" dependencies become a liability: You have to make or be sure than they are going to be around and still compatible with your current and future code. If you blindly add dependencies this problem gets more complicated.

As the other comments here have alluded,

a) a typo can run unintended code on your box

b) a bad/old/hacked project (correctly typed) can run unintended code on your box

c) both of the above conditions are considerably more severe because they can be triggered with near-zero friction (unlike a gemfile, which are manually pulled-down and code is run from a separate command. Gemfile results are also generally small enough (1-2 pages) that one can visually spot typos if you are monitoring the output

d) complex dependency systems can often turn into dependency hell

e) (IMHO) instantly-available dependencies potentially reduce a given programmer's likelihood to attempt to solve their problems with the language itself (i.e. don't use Cassandra when a Dictionary will do) ... the nodejs community seems especially susceptible to this

That's all I can think of right now...

Thanks for pointing out the obvious risks due to typos.

Added a --secure flag which will install popular modules only (>10k downloads last month)

`auto-install --secure`

That's one fast response time! Awesome!

;)

I am very interested in this for projects that I'm just starting and don't expect to run on anything but my machine. It's an annoying break of flow to be writing code and say "Crap, I need request or lodash" and have to stop, npm install --save, require, then go back to what I'm doing. Yes I still have to require it in but for side projects/one-offs I find this pretty cool.

This might just be a question of workflow optimization. Have you considered waiting to do the installs until you've paused in writing code anyway, or using your editor's shell command execution capability (if any) to fire off these installs and let them run in background while you continue to work?

You feel me, brother! That's exactly what happened when I decided to create this :)

Belongs in the Acme:: namespace.

Oh wait, npm doesn't even have namespaces...

Remind me again why everyone is using this shitheap?

Just when you thought dependency hell couldn't get any worse, it becomes automatic.

no.

Reply on news.ycombinator.com