Ask HN: Reducing the maintenance surface area of hosting a small internal app | Hacker News Reader

Ask HN: Reducing the maintenance surface area of hosting a small internal app

12 pointsgaze13 comments

I work in a small research group in a university where I'm the only one coming in with a background that leads one to care about maintenance and upkeep of infrastructure software. That means that whatever I set up should be trivially maintainable by those coming after. Ideally, it should be able to just be hands off for the next 10 years, but I know this is a bit unrealistic, unless the only application is a static site or the application is hosted on an air gapped machine.

That said, what's the best way to reduce or offload the maintenance surface area in a way that's most trivial to budget? I've been away from infrastructure myself for about 10-15 years now, and am just barely familiar with what Docker, k8s, and all this other stuff is.

Is there some way to box up the application code and libraries and let it run on a remote machine that someone else maintains? Amazon Fargate seems close but I'm terrified about misusing their services and ending up with a huge bill. Heroku might work too if I can slot everything I'm doing into their services. I really would like to have a timeseries DB like Influx or Timescale, but the load is so tiny that Sqlite or Postgres might do just fine. The cloud offerings from Influx and Timescale are just too expensive -- what we're doing would run just fine on an Rpi4 with a USB3 attached drive, but now we're back to a machine that needs to be maintained.

One thing I'm quite weary of is pay per request. I would much rather pay a fixed cost per month and make users wait, rather than having the price per month just be a question mark. Again it's an internal app -- I'm not trying to hit any kind of latency target, and I can build clients with some degree of robustness to services going down or timing out.

The most important things to me are hands-offness and fixed cost. Again, I would love to stick an Rpi in the corner and forget about it if this were not such an irresponsible thing to do.

Any thoughts?

I'm going to approach this slightly differently to everyone else so far.

Simplicity is the best for long term maintenance. I work with a legacy app bundled into docker containers. We simply have to at this point or deploying to a new host or dev box would be impossibly complex. Instead we have bundled that complexity into the docker build. It's still there and still requires a lot of maintenance. Don't do this if you can avoid it.

If you want your app to live for a long time you simply need support. You need engineers to keep it running. This is true of a lot of infrastructure from buildings to cars.

If you're ina university can you get support from the I.T. team. I know research teams probably don't want to deal with the politics of getting them involved but of you can make it their responsibility you will have the best chance of keeping it going for 10+ years.

Most universities will have internal servers for hosting this sort of thing.

I agree with pretty much everything here. I don’t think there’s a no-maintenance solution out there that will just keep running hands-off (maybe 25 years ago in a uni environment, but certainly not for the last 10 years).

You could probably automate a simple(?) cloud stack using a declarative & open-ended template that can be rerun every couple of years (or months) to pick up security changes, but someone’s going to have to set up & pay for sufficiently secure networking from the cloud provider to your internal network - this is a surprisingly large amount of work when you include security, user access, backups of systems & data, etc.

I work in academic IT and this is the sort of thing that we deal with frequently, so I’d hope that your internal IT would be similar - when you request things, tell them the whole story, rather than say request 2 Linux servers, so you can leverage their enterprise security & backup, and they can set up things so that the internal IT can effectively support the system (We see a bunch of issues in systems where we provided bare boxes, the research/academic team set something up, then disbanded or moved on to something else, leaving cruft and security vulnerabilities all over the place).

I have many systems that have been running for 10 years with few or indeed no changes, the same deb applies now on Ubuntu 2004 as it did on 804

I don’t use the latest fancy technology and frameworks that work today but have breaking changes every few minutes. I care about the output of the code, not the feelings of the person writing it. LAMP worked 20 years ago, and there’s been little update. Use Perl instead and there’s been even less. I’d be surprised if rhey didn’t work in 10 years. Meanwhile I accidentally upgraded some version of nodejs and it broke some “fancy” code that somebody had written. We ended up rewriting the entire code in php rather than try to unpick the mess that had been left.

You're not wrong about Perl. I hate working with it but if you're looking for longevity it's near the top of the list for good choices.

There are some things that run you into trouble. When SSL versions change you might have to update that part of your stack. When things like the log4j vulnerability hit you have to update that sort of thing. If you are working with external systems sometimes they change interfaces and you're forced to update. If you're working with government compliance you no doubt have to change every 3 years for no good reason.

I don’t use cpan modules directly, just OS bundled modules. Apt dist upgrade handles the required security issues (and besides these are internal apps, the threat surface is much reduced as you have to be an authenticated user/machine to access it in the first place)

Docker-compose, not k8s. Set up a cron job to run a script to update the OS, pull all your containers and reboot after hours once a week or once a day. Make sure the script specifies non interactive. Set up alerting for low disk space, see https://netdata.cloud or use your own tool. Set up firewall. Use SSL if you can, wireguard if you can't. Even internal networks should use http.

I have something very similar set up to run a few Game Servers and provide a file and container management user interface.

It's been running for a few years whenever I hear about a security incident I go check and it's always updated this far.

Theoretically you should be able to do something very similar by using aws cloud formation. Every time you go to update you would create a new image, install your dependencies, save it, put an instance of the new image up, divert new traffic to the new instance, shut down and delete the old instance.

>I would love to stick an Rpi in the corner and forget about it if this were not such an irresponsible thing to do.

If you can swing a say 10usd pm budget then you can get something of comparable scale from any of the VPS providers.

Simple linux firewall in front of it & have a think about backups and that should be it

If you can stick the service behind a free cloudflare to hide the IP and filter out a bit of the internet nastiness

Whether you docker it or not is personal preference. Def wouldn't do K8S if a rpi in a corner is your ideal vision

Heroku would be the easiest to use, and it has fixed costs attached (unless you use autoscaling, which scales worker based on latency.)

AWS AppRunner (built on Fargate technology) would be another option, but using AWS is definitely more involved than Heroku. With the exception of network traffic which is pay-as-yo-go, you can calculate max pricing for CPU/memory depending on the app configuration.

I use both of them at $dayjob. Unless there are good reasons to choose AWS, I would always recommend to start with Heroku.

Agree that Heroku, while possibly more expensive, is the most straightforward to get going and maintain over time. You're paying them to figure out all the VM/container/database/etc. stuff, and that's what you want. I've had several apps running pretty much untouched for years on Heroku (other than updating libraries and the Java runtime).

Depending on the language, there may be other services that work just as well, but AWS/Google Cloud/Azure are either more expensive or will require more time to setup and maintain.

What is your app's stack? AWS Fargate or Beanstalk is probably the only thing which does what you want (for a limited set of environments), but the problem which remains is updating the libraries your project uses. I don't think there is anything to cover that, unless you make somefully automated CI/CD system which will run nightly builds to update them and push new build into whatever cloud solution you chose.

I would recommend heroku. I just got notices from apps 5+ years old that there was a database upgrade needed. Didn't need to lift a finger; they upgraded them.

Build on the heroku 20 stack and you are good for 5 years, till 2025: https://devcenter.heroku.com/articles/stack

Oh, and include deployment scripts in the repo, as well as some documentation. That way, if you aren't available, someone else with approximately your skill set can help out.

I can't help thinking the usage model you want is more like a router or network appliance than a normal web server.

Have you considered running your software on a synology box or similar high end NAS?

Not sure it would work well if exposed to the internet, but if it's a small group and limited network access it could be quite a simple solution.

Reply on news.ycombinator.com