Hacker News Reader

Comment by kstenerud | Hacker News Reader

This is definitely cool, but it also highlights a huge problem we have with software nowadays.

There are 19 stacks in this repository. 19 pieces of software that require their own maintenance, JUST TO RUN YOUR APPLICATIONS! The amount of extra work required just to host the software that views your pictures, plays your videos, and allows chat with people is absolutely insane.

I'm not talking about this particular repo; it's just indicative of how complicated things have become that you must do the equivalent of building and maintaining your own operating system just to get even simple things done. And I belive that it's unnecessarily complicated (once again, not this repo, but rather the state of affairs in general). We're at an inflection point in the industry, and haven't come out the other side yet.

View on HN

I could post my fully automated self hosting infrastructure, but it would be boring. Few bash ssh scripts and LEPP stack. None of this complexity and yet 10+ years uptime. Interesting nonetheless but think i'll pass on this. I have done these setups at cloudproviders for clients; that is definitely the risk averse version; if something goes wrong every month with AWS, it's just 'ok'; if something goes wrong every 5 years with your non-cloud, it's panic & heads need to roll (even though it costs a fraction).

Yup ! This I've used bash+ssh for the last 10 years for my solo founder and commercial products. Is sexy or even nice ? Absolutely not, but damn its simple and most Linux semi professionals can understand it with a glance.

Bash as infrastructure+maintenance is almost like SQL for data.

Everyone agrees is not fantastic, many have tried to improve and yet SQL + BASH is still around chunking along doing their unsexy things just good enough :)

Never underestimate software that is 'just good enough' :)

> I could post my fully automated self hosting infrastructure, but it would be boring.

Actually, I would find that not boring :)

It's not just running an app. He's building a setup that supports FANG-level infrastructure with autoscaling, self healing, etc.

Maybe it's unnecessary for tiny companies, but if you're dealing with infrastructure at scale then the complexity is unavoidable. The only question is whether you want to invent it in house or use a robust system that someone else built properly.

I elaborated more in my comment below

You don’t need all of this for a home server, the author is just having fun.

Systemd units and a bit of sh is enough if you just want your applications.

> There are 19 stacks in this repository. 19 pieces of software that require their own maintenance, JUST TO RUN YOUR APPLICATIONS!

There's tools like Yunohost [0] that can save a lot of work and are focused on making the subsequent maintenance burden easier too. There's an application catalogue [1] with a lot of popular self-hosted apps to choose from and new ones being packaged all the time. I haven't used it myself yet, but hear very good experiences from people that have. Especially those that lack the more sysadmin tech skills and just wanna click-install & go.

There's also Freedombox [2]. I'd be interested to know about other such tools and your experience with them.

[0] https://yunohost.org

[1] https://yunohost.org/en/apps

[2] https://freedombox.org/

Thank you so much for name-dropping YunoHost! The demo alone sold me on finally going the self-host route for things such as ebooks, music, and photos.

Isn't it how the general software philosophy is? A composition of decoupled, do-one-thing-well pieces working together to achieve a larger task?

Yes, but after the initial monoliths of mainframes, we had a golden era of operating systems, where all of the plumbing and infrastructure to run your applications was maintained by someone else (Microsoft, Apple, Commodore, Atari, SGI, Sun, Palm, Redhat, etc).

Now we've come full circle back to the bad old days where you need an entire team of dedicated people and arcane knowledge just to run your application software again.

I suspect that this will continue until we reach a point where the big players out of necessity come to an agreement for a kind of "distributed POSIX" (I mean in spirit, not actual POSIX). These are exciting times living on the edge of a paradigm shift, but also frustrating!

That’s actually very insightful. The OS era you describe was mostly local with not so many things happening in online interactions and way less devices collaborating to create for example your images folder.

Now that we mostly have the server-client model ingrained in every online activity there is an obvious complexity centralisation on the server side.

Might be interesting to see how web3 and other decentralised web movements might factor in to this!

Apt install your lamp stack

Deploy your files

Cronjob to apt upgrade every night

Why does that need a full team? Or more than 10 minutes every few years?

I want to visibly second this (with a comment, not only an upvote). I dubbed the LAMP stack the CHAMP stack for some time now (CH standing for CSS and HTML).

one does not simply "deploy"

tar -xvf of your git repository

Okay but what happens when you have a breaking change from one of your updates?

From a standard system upgrade? Never happened to me in 20 years. Between OS versions sure, once every few years spend 20 minutes building from scratch

If it did happen, same as if your machine caught fire. restore from your backup

For me it's when a library my app uses gets an update then my app fails.

No need to update the library. Or keep both versions.

Ummm I'm sorry, but I don't see how this is practically simple. Theoretically, maybe - [and I agree with the idea that there is potentially (hopefully) a paradigm shift in initial play now, with the thrust being multifaceted including companies' lack of upholding user privacy/selling you as a product, slowly increasing service charges, and lack of customer customization.]

I think tis thrust towards self hosting should make the construction of these systems closer towards what you describe, but it isnt really that simple yet.

For instance, if you want to self host even a decent all-in-one calendar/notes/etc system like nextcloud, it isn't just `sudo pacman -S nextcloud` and youre done.

There is an enormous consideration of how to construct the 1) network architecture, 2) encryption of disks, 3) secure/encrypted, incremental, and full filesystem offsite backup, 4) security and encryption of network, 5) secure interplay between self hosted and non self hosted data, etc.

An example of (1) may be that, much if the discussion around self hosting states that one should not open their LAN to the internet with port forwarding (as it's inviting you to be hacked), and you should rather point your registered domain to a VPN such that you must login with any remote devices to access your server (just to point out - likely here you would need to use a dynamic dns service to get access to your system, which is also another hassle for newcomers).

Further on (1), this involves ensuring that the choice of VPN is adequate (many older technologies have known security flaws, like oracle padding attacks, etc.) - and finding a combination of routers with correct firmware to allow for VPN servers, (also VPN clients on top of that if you care about not having your isp sell you to everyone) the correct hardware and network architecture that can properly mitigate the encryption of traffic on your network (e.g. can a raspberry pi really handle all of your traffic and maintain 1Gbps file download/uploads with vpn encryption?).

I recognize that much of this is not really possible to `sudo apt-get` on one system with recommended hardware, but the more FOSS out there that can get closer to mitigating all of this, and to have well packaged tutorials on how to safely and properly self host (e.g. complete google service replacement) while maintaining security and 10Gbps speeds, the better off we will be.

This really is what we should be driving towards. Companies can certainly offer their servers for customer use, but I think it should be expected that companies have the code that is used to store that customer data fully as FOSS, for auditing and to allow for self hosting. There's still money to be made for companies in helping people self host everything for themselves - we just have to push for it.

One decent enabler of this is companies that make software for data backups. This is another one of the enumerations above, because it is quite difficult to find a solution that has quantum secure encryption, excellent compression, incremental (only pushing diff like git) capabilities, and is reasonably simple to use. AFAIK, Duplicati is the only option that seems to be able to hit all of these well (hopefully someone can correct me).

Anyway, this was a meandering long way to say - self hosting is complicated, but i do really hope we can change that.

I’ve been self hosting for over 20 years on apt based systems, aside from changing from telnet/ftp to ssh, adding in letsencrypt, very little has changed. You can still collocate your hardware, but it’s cheaper to use a generic VPS. You can host from home, but even a pi will probably cost more in electricity use than a $3/month vps.

Backups are far easier now as you use a VM provider like linode, digital ocean, even lightsail, which will schedule snapshots. If you run your own hardware then mysqldump and restore to your backup server is a small shellscript and cronjob. Disk encryption is handled by your OS.

LAMP was trivial in 2002, it still is. You can use nginx or postgres instead of apache and mysql, but it’s broadly the same.

If you want to make your site complex and dependent on thousands of JavaScript libraries and frameworks which change every year or two, that’s fine, but you don’t need to, it’s a choice, one which adds complexity. If you want five nines or absolute guarentee of not failing, you need to think more about replication than just a nightly snapshot, but that’s not a problem solved with thinks like kubernetes.

If you want to scale to millions of concurrent users pulling terabytes, sure, don’t self host from your DSL on a pi. If you want to serve a personal site for hosting bits of stuff, it’s not hard.

I'm interested in this, as it somewhat fits my experience on local stuff. (I haven't spent on a Vps yet, as I haven't figured out how to do so in a completely secure, private, and perhaps maybe even somewhat anonymous fashion).

I have been able to set up a simple system hosting a nas with NFS locally very easily, but ensuring secure connection lan remotely has been a very big hassle, and migrating to things like zfs have also been frustrating. The use of containerization for everything is just absolutely absurd these days, but it seems like the solution everyone is using. I wish there were more of a community for people self hosting who make all encompassing solutions without containerization and complicated JavaScript apps.

Most of that is kubernetes, which is completely unnecessary for self hosting purposes. It's mostly for learning, or for the convenience of hosting multiple different services, trying many different containerized applications.

If your only goal is to serve Nextcloud, Plex or whatever to your family, you can get away with much less than that.

I agree that the number of different technologies here is shocking, but you get a lot more value as a result than "just to run your applications". Most notably that includes CI/CD, scaling, and redundancy, all with IaC. These things used to require a lot more administration and development effort.

Plugging together all these different tools has become so much work though that in many organizations the platform team(s) who are mainly occupied with doing just that take up a lot of engineering resources.

In my opinion the next evolutionary step would be for all of this to be bundled and abstracted away. Funnily enough, we pretty much have that product already with autoscaled "serverless" cloud services, GitHub Actions/Azure DevOps pipelines etc.

The biggest problem is probably that things like AWS Lambda, SQS, and DDB lack versatility and user friendliness. If we get some improvements on that front, many organizations might opt for that instead of dealing with their own K8s deployments. Even better would be if we had something like a stripped-down version of OpenStack, just focussed on doing "serverless" right, and rebuilt from the ground up.

Reply on news.ycombinator.com