Hacker News Reader

Comment by seiji | Hacker News Reader

Would you fix a software bug by editing the code on a running server and tell yourself that you will add it to the repository later?

Absolutely yes in the right circumstances (have you seen how this website works?).

Not every service needs to be written as "scale out to a billion nodes" architecture with eight layers of checks and balances between idea to production.

We can't globally say "everybody must use a 16 step verifiable app development pipeline" when some people run 3 servers and others run 3 million.

(Plus, not all services are stateless! Redeploying stateful servers is painful—you'll be killing user experience. Updating code live on a running server while maintaining internal continuity of state can maintain sessions/caches/game-state without annoying users by kicking out of their current flows.)

View on HN

I agree with parts of this, sort of.

I maintain a personal server that hosts a variety of VMs for things like mail, static site hosting, nameservice, etc. Last time I rebuilt it, I thought, hey, I'll try doing it with these interesting tools. The immutable-infrastructure approach turns out to be pretty heavy in that case. A lot of times it feels like I have to perform a triple bank shot just to make a minor configuration change.

That said, I think the one-off server, carefully maintained by one person, is becoming very rare. When I build things that other people will work on, I think immutable infrastructure and the cattle-not-pets approach is the only responsible way to go. Even if traffic volume won't be huge, I think the clarity and ease of debugging you get is vital when it doesn't all live in one wizard's head.

I agree that stateful servers are a challenge in this context. But they're a challenge regardless. Having that one thing we're afraid to upgrade or restart can cause all sorts of development and business process issues. That might have been a worthwhile tradeoff 10 years ago when you had to buy all your hardware and you needed ops people to do a lot of stuff manually. Servers go down, and we might as well accept that from the beginning.

That's a false dilemma. There are more options than "edit code on running server" and "use a 16-step verifiable app deployment pipeline".

For example: update the code in your source control repository, and then build the new version of the software.

> That's a false dilemma

It's not, and that's coming from someone who does devops for thousands of virtual machines in production. I don't have an hour to burn sometimes to update a repo, build new AMIs, roll them into production, roll the old ones out, watch logs to ensure canaries pass acceptance tests and that production traffic to new instances aren't erring out, all for minor changes (example: nginx mime type change).

Try something like Heroku. Commit the code to Git, push the repo to the Heroku repo and it deploys it to the servers for you. Easy.

You mean the same Heroku that has the uptime of a server under the dev's desk? No thank you.

Or, since we're talking about adding tooling for better efficiency - just update the source code, and let the tooling build and deploy! :)

Automatic (and safe, obviously) production cutover is still kind of a holy grail, but it's definitely doable, and being done as we speak by a number of pretty neat companies.

Reply on news.ycombinator.com