Npm Private Modules
npmjs.com
npmjs.com
That kinda rubs me the wrong way; it feels like they're trying to force this monetization as they've had plenty of us from the community wanting better GIT support for this very reason.
Honestly npm isn't a very complex piece of software, you could even replace it with bower if you want git with versions. I wonder if any competitors are going to spring up who can simply iterate faster. I won't even get started on their login system, ugh.
You can even point to a specific commit hash. But no, it doesn't read any of the versioning. What if two different branches have the same version number? The publish step is pretty important, because ...
> If they updated the use of GIT you could completely eliminate the need for this feature.
... You'd still miss out on the prepublish hook - one of the major reasons I've looked into using NPM Enterprise.
Sometimes your package needs to do some housekeeping _before_ you publish, and the files generated by that should be distributed -- but not checked into source control.
The semver mechanism would be based on the tag, so it would not work when pointing to a specific commit hash (and you wouldn't want it to).
> ... You'd still miss out on the prepublish hook - one of the major reasons I've looked into using NPM Enterprise. Sometimes your package needs to do some housekeeping _before_ you publish, and the files generated by that should be distributed -- but not checked into source control.
That's a nice feature and all but not required to have proper git support.
some-user/some-module#1.0.0
Now this is not very robust, as it does not manually incriment semver. That being said a module it not a git repo. There are pre-publish hooks that can be used to execute scripts before publishing. Further, tracking off of git introduces an entirely new set of problems from tracking tar balls. I can see why the team would not be particularly interested in investing time in it.
That was the parent's criticism. Other systems do increment semver.
What would the "proper" behavior be? How would npm know which commit is the one you intended to be the version change? Scan the history of package.json and only grab the commit when the version number changed?
Isn't that entirely the point of using tags?
Nonsense; this is a very quick operation in git. Even in enterprise situations you're probably not going to have a huge amount of private repositories.
Currently, private packages are only available for individual users, but support for organization accounts is coming soon. Feel free to create a user for your organization in the meantime, and we can upgrade it to an organization when support is here.
* NPM is unable to keep track of the version. This means no semver and that `npm install` re-installs the module every time. * Git sub-modules do not work. This can sometimes be a deal breaker. * This requires either your username password or private key to install. This gets especially messy if you are trying to do some type of CI build (e.g., build a docker container) and don't want your keys to leak into the build artifact.
Imagine if RubyGem/PythonPackage/NuGet goes with this stragtegy.
No other language does this. you have all the same capability for propretary package with RubyGem/Python/Nuget, even Ubuntu allow you to add third party package repository.
It gives me the clue that npm may no longer improve, or many improvement will be "premium" feature.
I hope someone fork npm and add ability to parse semver from tag for github repo.
I'm a little surprised they didn't go with a model similar to GitHub: - Payment plans with X number of private modules, or $1/month per private module - Unlimited paying/non-paying collaborators for private modules (perhaps only read access for non-paying)
With npm's model all my collaborators will have to pay for npm private modules as well.
The way that npm has structured things, programmers pay once for membership in a commercial tier, where presumably money is already changing hands to work on private code. I think this makes much more sense and won't bias the code itself in a negative direction.
I hadn't thought of it from the point of encouraging/incentivizing the authoring of more public packages.
But couldn't the current model then discourage the authoring of public packages, and lead people to start primarily publishing private packages.
I assume whatever reason people are authoring public packages today won't change because they get a paid account. Let's at least hope that's the case.
https://docs.npmjs.com/misc/scope#associating-a-scope-with-a...
Some of the other commercial npm hosts and open source offerings already support scoped packages.
Ok in that case I hereby officially really like the way npm Inc are working on monetization.
https://books.sonatype.com/nexus-book/reference/npm-configur...
Source code: https://github.com/sonatype/nexus-oss
I suppose if you are not using Maven, though, a custom tool integrated with the Node toolchain would be more comfortable.
We ended up going with NPM Enterprise instead.
NPM Enterprise' github authorization module requires an access token in the user's .npmrc file, but no modification to the actual running app to add/remove new users: they just have to configure their .npmrc correctly (by default, if a user has read access to the github repository referenced by the repository field in the package.json, they can install the package - if they have write access, they can publish it too). No fiddling with sending someone's NPM credentials around the network, inserting them in a YAML file and restarting the registry application.
It was six months ago that we looked into it, though, so it may have improved since then.
We also looked at using nodejitsu, but their administrative portal was abysmal, their billing buggy, and reliability unimpressive. All of which explains the godaddy purchase ;)
The auth token you need locally in your own .npmrc is added automatically when you do "npm login".
We wanted something available "now," not "in a bit, after we write an auth plugin for it."
https://www.npmjs.com/package/npm-registry-couchapp
npm's open-source project which lets you run your own registry on CouchDB.
pip install --find-links http://dist.example.com/packages/
I wish I could do the same with npm.
Here's a question: people with read-only access (to my private packages) have to be paid users too?
> Give read access or read-write access for those packages to any other paid user
So in the end is $7 per user, you can't have just one paying user publishing for everyone else. I guess I'll have to wait for organizations accounts for now.
I would hate if NPM went the way of wordpress plugins, where every stupid little plugin costs $5 to access.
1. It doesn't support threads (facilitating structural sharing of large data-structures between parallel tasks, which cannot be done using ordinary processes).
2. The module-loading mechanism ("require()") natively doesn't support delayed loading, which is needed when loading from within a browser. Yes, there is the "browserify" package, but, come on, something as basic like this should be supported out of the box. Especially considering the fact that there is a "http" module hardwired inside nodejs (why isn't this a separate npm module, btw?)
3. To make my own privately held modules and install them properly, I have to run a npm server? This seems like an awful lot of work for something as basic as this. Ok, so now I can use the cloud for this, but come on, I should be able to do this just from within the filesystem, like e.g. git does it.
For people interested, one can use the package "sinopia" for hosting your own private modules. It seems to be a pretty decent package, but be aware that the authentication settings out of the box are completely insecure.
"dependencies": {
"private-module": "git+ssh://..."
}
"dependencies": {
"private-module": "git+https://<user>:<password>@..."
}
"dependencies": {
"private-module": "git+https://<token>:x-oauth-basic@github.com/<account>/<repo>.git"
}
2. require() is part of the CommonJS spec, and how it physically works is dependent on the implementation. You point out that Node's implementation doesn't work well in the browser, but Node itself does not work in the browser so that point is moot. I agree that it might be interesting to load remote modules in Node, but keeping that operation synchronous does simplify the language quite a bit.
3. You can also map modules to public or private git repositories in the package.json, as long as the private key used during npm install has access. If the git repo has tags, a tag can be specified in the git uri as well. Private npm repos are the superior way to distribute private modules with wider access, but I think this is handled fairly cleanly already.
1. Node.js is a tool for building servers. On a server you generally cannot afford to have the event loop blocked by a computational intensive task. You need threads.
2. It would only require a "promise" to make the module-loading asynchronous. Leaving that out is not what I would call "quite a bit of a simplification", especially if using asynchronous callbacks is the "modus operandi" of programming on the Node.js platform itself.
3. Okay, I stand corrected. I remember that I waded through the documentation quite a bit though, trying to figure this out.
But my biggest problem with workers is that they don't have an event-loop, so I can't share asynchronous code between the main thread and the workers.
The way it works is you open a download stream from S3, pipe it into a Node.js transform stream, and then pipe that stream into an upload stream that uploads the data back to S3 using the multipart upload API.
The Node.js design is very much like using Unix pipes. You can pipe a huge multi TB file through grep without blocking anything. The data just streams from disk into the grep process, grep filters it down to things that match, and then streams the results onto the screen.
Computation on huge streams in Node.js works the same way. Your event loop remains unblocked even when operating on a stream TB's in size because you are only ever touching a portion of the dataset at a time. Additionally if you do it properly your overall memory usage remains low as you are exporting the data back out of the machine as fast as it comes in. I've used this technique to process streaming data many GB in size while keeping the node process under 200 MB of memory used from the system perspective.
Recommended reading: https://nodejs.org/api/stream.html
Here is an example of an upload stream that I created for the use case of processing a large multi TB data set and piping the result up to Amazon S3: https://www.npmjs.com/package/s3-upload-stream
But basically, what I wanted to do, is implement a module that works as an index between threads (e.g., a search-tree for fast lookup). However, since in Node.js all threads are in a separate process, it is (afaict) impossible to make this efficient, as processes do not share data.
Basically Node.js is designed around the concept of microservices and separation of concerns. Rather than doing everything in one giant, multithreaded monolithic process you break your service up into loosely coupled components that talk to each other via messaging and share common datastores. Some people really like this pattern (I'm a strong advocate of it myself) because it scales really, really well.
You are not supposed to use your main event loop for computational intensive tasks.
Offload those tasks to separate workers and use queues.
That's node's basic knowledge. Its a trade off that you're supposed to be aware of when using node.
EDIT: I mean workers which run in a thread (as opposed to in a process). An example is given by the webworker-threads npm module. Threads allow one to structurally share large data-structures, so one does not have to serialize them when calling a worker (serializing large structures would block the main thread).
Here is how a typical node stack works:
Nginx load balancer talks to a cluster of node server processes, one per core. The server processes handle all incoming web requests that won't block the event loop. On a typical REST server this is 99% of your tasks, and each node process can handle thousands of concurrent requests due to the way that the event loop works.
If there is a heavy, blocking task like processing an image or PDF file, (although even these things should be able to be done in a nonblocking stream manner) the server processes send a message through a background queue such as RabbitMQ, or Amazon SQS or the like to a background process which has the sole purpose of processing heavy tasks pulled from that queue. Fundamentally if you are using Node.js properly you don't need multiple threads. Instead you use multiple processes, and the processes are essentially "threads" that can talk to each other either using parent/child processes communication, HTTP, redis pubsub, or any other mechanism you want.
But there is no reason why anything should block a Node.js process if it is written properly. I've even done heavy video transcoding in a streaming manner in a Node.js process without blocking the event loop.
I think most people here misread the line "facilitating structural sharing of large data-structures between parallel tasks, which cannot be done using ordinary processes" in my first post.
And by large data-structures, I don't necessarily mean structures which can be "naturally" streamed. I'm thinking more of a large index, for example, which can be used for fast lookup, and be used from several threads at the same time.
Having processes (here named workers) is a nice feature, but doesn't cut it when you want to share large amounts of data between threads (serializing that data would completely block the main thread). In my view, it is unfortunate that the designers of Node.js didn't opt for having multiple threads as opposed to putting every thread in a separate process.
It is[0]. It's a dependency of browserify.
If you don't use it, part of the concat+minification step prunes dead code.
> To make my own privately held modules and install them properly, I have to run a npm server? This seems like an awful lot of work for something as basic as this. Ok, so now I can use the cloud for this, but come on, I should be able to do this just from within the filesystem, like e.g. git does it.
If you don't care about having your package published somewhere other than your local machine, you can use npm link[1], or `npm install <directory>`[2].
0. https://github.com/substack/http-browserify/