PM2: Production Process Manager with a Built-In Load Balancer | Hacker News Reader

PM2: Production Process Manager with a Built-In Load Balancer

github.com

88 pointsthunderbong83 comments

Tangentially:

Writing this at 1:30am, might not be very insightful.

We actually are phasing out the use of pm2 in our org since we are already running within kubernetes. And running pm2 within k8s seemed redundant (and is likely a relict of previous deployment strategies).

It also made certain debugging tasks trickier as we very rarely like to attach debuggers to our staging environment and the cluster/fork behavior of pm2 made it hard to effectively target indivual executions/processes.

Further we had some issues running nextjs after upgrading to the latest version within pm2 when using custom configuration files (we did not end up finding the root cause but decided to drop pm2, least path of resistance etc).

One other thing that also gave us headaches was the fact that pm2 allows specifying different env variable values for different environments but no other configuration (such as cluster count etc.)

We were successful entirely replacing it with docker-compose configs locally that are ultimately closer to how our services are deployed in production.

Overall I think pm2 is suitable for local dev and prototyping small to medium complexity services.

At work, we have kubernetes and pm2 running on a pet server outside of kubernetes. I cannot wait to get rid of pm2 and the pet server. For smaller projects, I'd rather just work directly with systemd or better yet, use the podman-systemd integrations.

It seems strange that you would even consider using pm2 still if you have all of that other stuff.

I use it to avoid the complexity of those things since my operational needs are usually very modest.

It may well be that I did not learn to use it properly, but - to provide a counter opinion - pm2 left a sour taste with me. Tried to use it few years back and by default it does not rotate the logs, did not handle machine reboots well, and at times I was confused what is running and what is not. I settled on docker for small projects and kubernetes for larger ones and these things made more sense to me. I admit it could be a matter of habit.

I came to the opposite conclusion for the same reasons.

Docker/kubernetes is fine for things that make money, but for small personal projects it is far too much complexity and I dumped it. pm2 gets the job done with less effort overall.

I’ve found pm2 more complicated than docker-compose for local/self hosted small projects.

Why did you prefer pm2?

Honestly, I hate using docker. It’s a pain to deal with in your dev environment, especially on a Mac. It’s not lightweight for your dev machine. It requires a good internet connection and it uses a lot of data. It’s all very “enterprise”.

If you’re sat in an office, you’re on a work computer, you need reproducible dev environments, probably none of that bothers you.

However, if I’m working by myself, I don’t want to deal with any of that Rube Goldberg shit. I use shell scripts, ssh, rsync and pretend it’s the 00s again, and oh my god I’m so much happier in that universe.

Maybe it’s different on Mac, maybe I’m biased, but I’ve been using docker since it was a small project and I’ve never felt that enterprise cruft.

It saved me from the Rube Goldberg esque pile of bash scripts and puppeteer configs of the 00s.

Funny how perception changes.

It’s so much worse on a Mac. Wrong OS, wrong CPU architecture. Never had any issue with it in deployment.

The other problem for me is, let’s say you’re in a hotel with crappy wifi and docker wants to download a large quantity of images, you’re in for a rough time.

There’s a very sharp downwards trajectory of user experience once you leave the happy path.

Again, no idea about osx, but if you have any project with lots of deps, it’s going to take time to download.

A well crafted image may add an extra 500MBs on top of that. By no means minimal, but not really that bad. (Assuming you have no base images like python or Ubuntu already on your system)

And idk what you mean by “happy path” for docker. Every dockerfile is a snowflake. It’s not a framework like rails.

If you have a good understanding of the tool you shouldn’t have any trouble.

It uses a VM on Macs, it's usually slow and requires too much thinkering to get it to work decently. I gave up development with Linux tools on Macs, very annoying and not worth the hassle. I think it even works better in Windows since WSL.

I'm using runit instead of pm2. Systemd should also do fine. It's over engineered.

Ironically WSL uses a VM as well

Of course it does, but at least it works quite decently. And you're running an actual distribution under it, so no hassles with MacPorts, Homebrew and other third party solutions for installing dependencies. If development on Snow Leopard was quite annoying, I can't imagine what it is now with the newer Apple sandboxes in place.

https://learn.microsoft.com/en-us/windows/wsl/compare-versio...

I’m guessing you meant puppet (iac) not puppeteer (e2e testing)

Yes puppet!

It’s been a long time.

pm2 gets past involving containers

I get to rsync my code up to a server, & pm2 restart it. The server process runs like any other, no container issues to consider, just a json file saying how I'd run it myself if I wanted to

tbf I'd rather replace pm2 with systemd, since I'm using it to run a Rust program, but if I was using node their cluster mode looks neat

My experience starting out with PM2 was the same. You need a plugin for the log rotation, saving the default environment variables is confusing, and stale processes cause weird behavior.

You may already have your own in-app log rotation before adding PM2 so may want to avoid printing to stdout/stderr. For usage with docker you want pm2-runtime.

As others mentioned - logrotate is available as an option, since the system one is more preferable for many. As per reboot handing - it's available via "pm2 startup" script.

I'm with you. I think PM2 is half-baked software. It tries to do too much and none of them well. It has two source-of-truths, the dump file and the ecosystem files. It doesn't have a way to stagger restarts.

For small projects, there is now a way to run docker containers or pods (through podman) via systemd units, so if you are using a systemd enabled distro, then that may be a better alternative than pm2, especially if you already have proficiency in docker and kubernetes.

PM2 has worked very well in my experience. I have about a dozen services (APIs and full-stack applications) across a number of servers, remote and local. Mostly Node, these days trying with Bun. Some of them have been running for years without a hitch. It's been totally reliable.

I'm sure there are alternatives, lighter-weight and closer to the metal, but I haven't had the need. PM2 does the job.

PM2 is very good and I really wonder why it's still somewhat unknown.

People complaining that Node.js only use one CPU core, or benchmarks with low scores for Node.js. There are solutions for this! PM2 is one of them and is very good. Use it.

How do you feel about using pm2 inside of a container?

At work, most of our app is docker-composed in one VPS (early B2B product with few customers). I would like to have the multiple-process benefits of PM2, but I don’t want to spin up a whole load balancer for it, or split the node service from the other services. Everything is so simple right now and I’d like to keep it that way lol.

Last time I googled this I found conflicting opinions, but if I can just stuff PM2 into my node container, I’d be a happy dev!

We use PM2 in a docker container since we got a memory leak and PM2 has a memory limiter option that allows us to kill and restart a process when it uses too much memory. If we let it grow the response time starts to drop due to GC taking longer and longer.

Node.js cluster https://nodejs.org/api/cluster.html or worker_threads

You have to do so many things just to use more than a single CPU core. There are other language platforms and runtimes where using up the entire available CPU cores is the default option, and taking advantage of multiple cores within the app is easy by design.

I can see why people like pm2, similarly to why I can see why people like Nodejs. But I think pm2 is half-baked, and Nodejs's greatest design feature (single-threaded async execution) is flawed by design.

PM2 is a life saver for sidestepping Kubernetes imho

pm2 is the de facto process manager for Node.js. I don't think it's "somewhat unknown."

Many don’t take the time to do some research to figure out “what’s the right way to be doing this? What do i need to search for?”

You write nodemon or npm start- and so it's put at the end of the Dockerfile as such.

I know someone that discovered pm2 because copilot autocompleted their Dockerfile that way.

Just spawn more processes. Great solution! /s

Would you rather scale with more containers? That's heavier.

Properly written node apps are bound by I/O throughput. CPU and memory cost for additional processes is minimal, but latency is significantly decreased with a worker pool.

> Properly written node apps are bound by I/O throughput

Does that mean that node apps shouldn’t do any computation on data, just move bytes between a file and a network socket? If I need my program to think, I wrote it wrong or in the wrong language?

I would rather have shared memory multithreading be a usable feature in my program’s runtime. SharedArrayBuffer exists but 98% of existing code uses objects and arrays so can’t be easily shared between processes without copying, and the cost of copying objects with structuredClone puts a huge optimization barrier in between most code and parallel processing.

> Does that mean that node apps shouldn’t do any computation on data, just move bytes between a file and a network socket? If I need my program to think, I wrote it wrong or in the wrong language?

This is a very uncharitable interpretation of Node's primary use case. I think it would be more accurate to say don't use Node if all of the following apply:

- Your task is CPU bound rather than IO bound (e.g. you're not waiting to insert or retrieve data from the database).

- Your task is highly parallel (some computations are sequential and linear).

You can't use worker threads because:

- You have too much data to bear the cost of message passing via the structured clone algorithm.

- You can't share memory and avoid structured cloning because your data can't be represented using an array buffer.

This is a far cry from the claim that Node can't handle problems that require "thinking". When you're on the JIT compiler's happy path Node can think pretty quickly actually. In practice, many high volume web servers are handling enough requests that there's more than enough jobs for each Node process to have exclusivity over its jobs, and you don't need to share work between processes. Each process or worker thread can be largely independent. Not every task is going to fall within those boundaries, but that's ok, Node doesn't have to fulfill every use case.

No, just write your javascript to do more I/O than compute. Apps should do app things not low level tasks that slow things down.

If you need to compute something complicated, write it in a compiled language and make it a node module. Many popular npm libraries are exactly this.

Or… don’t write Node at all? I’d choose Elixir over Node any day.

I mean... yes

Is systemd capable of doing this?

I thought process managers like supervisor and pm and the others had all been superseded by systemd, unless you’re in a non systemd distro of course.

Systemd also gives you cpu/memory limiting, file system isolation, locked down capabilities for the process, CPU affinity, can use different Node versions for different apps, etc. It removes the need for Docker for the vast majority of cases.

Systemd won’t do load balancing, but you should have nginx/caddy in front of Node anyway, and either will handle load balancing just fine.

What about creating new processes?

Yep, systemd is an excellent and stable process runner - it can start your processes whenever, with dependencies (eg redis/pg), or on a flexible schedule etc.

I created a Node.js binding for sd-notify to cover at least some of the use case here: https://github.com/systemd/node-sd-notify

I don't think systemd has a built-in load balancer. Also, running multiple copies of the same service isn't typical PID-1 stuff, so maybe doing that with systemd involves some friction.

Running multiple "copies" of a service is very much in systemd's wheelhouse; it's called templated units.

I know about templates, but I don't believe they are designed for fungible copies. You can run a bunch of my-app@a-random-port.service, but it's not the same UX as `pm2 start my-app -i number-of-copies`.

They work very well for "worker" processes. The default ssh package on Debian uses systemd template units for the user process; I've used them extensively for job queue workers, in combination with max execution time and a separate timed job to monitor queue load (essentially automatic scale up/down of queue workers).

Realistically if your prod environment relies on you manually running `pm2... ` or whatever, you're doing something wrong. User Experience should have about 0% impact on how your service daemons run.

systemd can run Node.js apps easily, but unless your applocation is doing something special for multithreading that is single threaded. PM2 makes it trivial to run an application in cluster mode.

Had good experiences with PM2, however be cognizant of it's copy left license. We have found it redundant with the likes of Kubernetes.

> be cognizant of it's copy left license

why?

Some organisations forbid copy left licenses

PM2 was really nice until I moved to using Docker Swarm Mode which is even better in so many ways!

For some reason I thought Docker Swarm was abandoned. From the Docker docs,

> Docker Swarm mode is built into the Docker Engine. Do not confuse Docker Swarm mode with Docker Classic Swarm which is no longer actively developed.

the change from docker classic swarm into current docker swarm happened years ago. from what I can tell, usage-wise there are not much difference between the two.

however the sad thing is currently swarm is held by other company (mirantis) after being sold by docker inc. and the new company doesn't seem that invested in improving swarm. you could say its in life support now save for a few community contribution

more explanation by dockerswarm.rocks website: https://dockerswarm.rocks/swarm-or-kubernetes/

Just used PM2 on a recent project for deploying a nodeapp on a single server. Very easy to use and just works.

Does anyone have an alternative to pm2 that's reliable? That's my biggest issue with it: it gets slower and slower to do commands like `pm2 list` over time, and eventually, pm2 itself crashes—taking everything it runs with it.

Also, `pm2 flush` doesn't actually truncate all logs, just the logs of whatever processes pm2 happens to be running at that moment.

I workaround this by killing pm2 entirely and `rm -Rf .pm2` once a month or so.

(This is all on Linux BTW.)

I've had a lot of luck with dokku. It uses docker so it's more heavyweight, but it's super simple and scales really well

Dokku maintainer here.

Dokku itself doesn't run anything in resident (there is a small go binary that listens to the docker event stream and restarts apps as necessary but thats it), so there shouldn't be overhead there.

On the Docker side, there is a _slight_ amount of overhead in cpu/networking, but its so negligible that its sometimes difficult to benchmark. While I can't really make changes to Docker to alleviate issues, I'd be interested in knowing what parts of Docker are heavyweight so we can better surface any such information to Dokku users.

I used this to run an entire startup for 4 years (complex data needs, not CRUD). It was a stopgap that was good enough to stay. I always wanted rid of it, but there were always more pressing problems. Used the load balance feature extensively, which worked but starts to falter when you get to big scale. It's not a stable foundation, it's a thing that's been hacked around enough to the point where the problems are mostly gone.

If I did it all over again, I'd pick something proper like systemd or k8s. I would say this is more a surprisingly capable dev tool than a real service manager.

Take a look at the code. E.g. https://github.com/Unitech/pm2/blob/master/lib/API.js

I don't want this thing running my production environment if I can help it.

Horrible piece of software, it uses a stateful(!) configuration. The daemon updates it's state at runtime and there is absolutely no guarantee it will properly restore that state when it restarts. It's extremely unreliable/flaky, you really don't want that from such a critical service.

It's a fascinating project, and esp for from before docker+k8s normalized and nodejs was reinventing the universe

Nowadays, docker compose or k8s ecosystems do a lot of this stuff in a more team-friendly & ecosystem-supported manner. I always liked pm2 bc the dynamic API layers... except docker & k8s also have APIs, and you get all sorts of extras with them by being a layer below

I wonder if there is a name for this waller garden phenomena. The pm2 team built a wonderful universe and it keeps improving afaict, but tech ecosystems also have been improving around them and at faster & bigger levels...

I am running a 18 node SOA since 4 years with pm2 and I am very happy. Very reliable. No need for the complexity of Kubernetes or even Mesosphere DCOS back in the day. I do not even consider to change anything.

All on a single host, or is this spread across multiple servers? Curious about how people run it with the latter approach if they do

Actually on 3 hosts, but only the primary is at the edge, one other is a hot-standby at another geographic location and the third is for incremental backup purposes via ZFS. The database sync connection are tunneled via Wireguard. Failover to the hot-standby is done via DNS (ClouDNS). Also I run the setup in a Linux Container (LXC). Deployment is done via SCP and Tar. :-D This setup is very reliable to me and at the other hand cheap!

I like PM2 and it's successfully been keeping my APIs alive quite well, but how does the future look given the popularity of Kubernetes? What is the overlap like and what's not overlapping?

I think Kubernetes is unnecessary for the vast majority of deployments. Simple apps run perfectly fine with things like PM2, Monit and init.d

Be that as it may - if you already have Kubernetes up and running, what would be the additional value of having both PM2 running alongside Kubernetes?

Fwiw, last time I asked myself this question my conclusion was that there was no additional value, and we opted to not use PM2. This was a long time ago however, so take it with a grain of salt.

I agree with you on that

PM2 is great if you have an app composed of multiple services that need to share an environment. You may want to reduce communication overhead between these node processes or avoid involving kubernetes to share resources. Implementation details would drive this.

Also, kubernetes cluster workers can fail. How do you want your workload to behave in that case? When your container is killed, should these services fail together? Do you want a better separation of concerns between your app dev team and your devops/middleware team (think disaster recovery and cluster maintenance)? Where does the app dev end and the devops begin? For other languages such as Java or C++, it would be out of the question to let these details matter to anyone but app devs. Their entire apps are in one docker image. Kubernetes should only manage instances of the app and the obvious dependencies that should be externalized (redis, memcached, postgres, mysql, etc.)

In my experience it's really a question of failure modes in your architecture and squeezing out small gains in performance that matter at scale.

I use PM2 with BrowserBox and it's been great. Reliable, scalable, performant. PM2 has been with us since the early days. We've never looked for an alternative.

I don't understand why people think k8s is a good replacement for pm2. The resource requirements to run containers over standard processes is tremendously different in practice.

Can you make Kubernetes deployments easier than this?

    cd myapp
    # Ignore multer.
    NODE_ENV=production PORT=80 HTTPS_PORT=443 pm2 start bin/www --name myapp --watch --ignore-watch="\.git node_modules uploads"
    pm2 startup
    pm2 save

I've experienced some strange behavior with pm2 on my projects where a reboot would completely wipe the configs and I'd have to rebuild the processes (even after a pm2 save)

If it happens again I'll likely look into doing it the old fashioned way with systemd.

Skill issue? Most certainly. It still sucks to think I've figured it out (to the point of testing with manual reboots) only to one day have nothing running and have no clue why.

Could some one please explain to me if this is still gonna be useful if we already have load balancing on the API gateway?

No, it won't. It's basically just a NodeJS application server. On the plus side, you get a dashboard with statistics like latency, memory usages etc. But from your description it seems that it will be a redundant solution.

how does a normal load balancer make sure the nodejs server utilizes all cores?

Is pm2 claiming that the cluster module is a load balancer? Because I don’t really consider round robin to be a full load balancer. Maybe 25 years ago but not today.

The node cluster module needs a way to provide your own balancing algorithm. Then I wouldn’t feel the need to put a reverse proxy in front of it.

PM2 fits the job perfectly when you want to start managed processes from the CLI without a config like you would with something SystemD or SupervisorD). It has some nice interfaces for observability.

Its more lightweight than spinning up Docker containers for each process

Be warned. It’s AGPL. I’ve never been able to reach the developers to get commercial licensing.

It has a paid versions: plus and enterprise. But actually, I couldn't find any information on their site about what the support covers.

It's just something that's not needed.

You can come up with a Unix solution by using bash, piping to logs, daemonizing with &.

And as a bonus, it works for any program, not just with node stuff.

Remember, use the tools available in your foundation before adding more blocks to the pyramid

Is there a robust generic solution already existing? To launch multiples processes, load balance and automatically restart if one crash..

Maybe systemd? I just always find the unit files so cumbersome to deal with (actually I feed pm2 to ecosystem files to ChatGPT and let it deal with the conversion)

Despite a few shortcomings, such as the lack of automatic scaling, it is definitely more stable than, nginx unit, which we had a lot of problems with, due to segfaults and very "tricky" nodejs support.

I set up my first nodejs backend project on an ec2 instance with pm2 almost 9 years ago. I've also been toying with an instant deploy pipeline that takes advantage of its watch and restart.

I used pm2 to run my adhoc Python jobs/bot. Tried with Django and celery and docker, ended up being a pain in the ass since I have to modify on the fly.

Pm2 is neat, but the monitor solution is pricey.

Thankfully since I started hosting my Node apps on Fly.io I don't have to care about PM2.

Reply on news.ycombinator.com