Sovereign: Ansible playbooks to build and maintain your own private cloud | Hacker News Reader

Sovereign: Ansible playbooks to build and maintain your own private cloud

github.com

556 pointsc0restraint140 comments

There are so many servers and apps being installed by Sovereign that I'm certain few would be able to keep it secure (https://github.com/sovereign/sovereign/wiki/Software-used-by...). The big win for the cloud is that you're paying a fraction of the cost for access to a, typically, enormous security and operations team. If you want to build software like this that allows people to self-host, you need to scale down what you deploy to what a single person can reasonably manage. This isn't it.

Fun todo: Install this somewhere, nmap it for open ports, then ask "How many of these services had a remotely exploitable CVE in the last year?" "If one of these services had one tomorrow, would I know to patch it and take action faster than someone would takeover my box?" I don't see any containment mechanisms on any of these services beyond what's included by default so a compromise of one service likely leads to total compromise of the entire box.

I had to think about this a lot with AlgoVPN (https://github.com/trailofbits/algo), and we built a system with no out-of-the-box remote administration, strong isolation between services with AppArmor, CPU accounting, and privilege reductions, and limited third party dependencies and software. You can't count on a full-time, expert system administrator.

This touches on a problem I've been thinking about a lot. AWS, etc, have solved the problem of hardware operation. Containerization is doing the same for OS operation. I think the next thing is app operation.

For some things, I'm happy to use SaaS providers, where they are responsible for the whole stack. For others, I'm happy to use apps, where they just provide the code, and I provide the platform. But for a number of things, I want something in between: I provide storage and compute, they provide code and operations.

Bitwarden for me is a good example. They're a password manager who provides their backend as a docker container anybody can run. I like that, as I don't really want them to have my data, and if they go out of business, I don't want to be cut off from my passwords. But I won't run the backend myself, because I don't have the time and expertise necessary to make sure it stays secure.

Another good example is photo hosting. I would rather keep all my photos on space I control. But I also need modern, maintained software for syncing, serving, and controlling access to photos and related data. I'm happy to pay somebody to make and maintain that software, but not nearly as happy if that means that at any point they might shut down and take my data with them.

I suspect we're headed toward a future where people like Synology and Digital Ocean sell storage+compute, and then other companies sell and maintain user-selected software that runs on those environments. Basically, some sort of app store for servers. But I'd love to see this happen in an open, nonproprietary way, as the drawbacks of Apple's and Google's app stores have become pretty clear.

Sandstorm was a really nice solution to this, but it required each app to be integrated with it, which I personally think is what killed it. (Which sucked, because for what it supported it was the best option available)

Ah, interesting! Just reading the home page wearing my developer hat makes this stand out: "Each document, chat room, mail box, notebook, blog, or anything else you create is a "grain" in Sandstorm. Sandstorm containerizes each one in its own secure sandbox from which it cannot talk to the world without express permission."

The notion that every document is its own independent unit sounds pretty menacing to me. Could be fine for some things, but getting things running there is sounding like a fair bit of work to me, and very limited.

And then this part is especially bad: "[maybe someday] You won't have to deal with payments | Eventually, we hope to make Sansdtorm implement in-app purchases and deposit the proceeds directly to your bank account"

Right there a lot of incentive to integrate has leaked out. "Please build for our platform in exchange for no money" is not quite the worst offer I've had, but it's definitely not appealing. Looking through the Wayback machine, I see

But the part that really concerns me is that they seem to think that server apps can run like mobile apps. To me one of the most powerful things about SaaS products is that the aggregated use information drives both product and operational improvements and allows rapid response to bugs and issues. So as a developer choosing between this and a SaaS approach, this feels like having one foot in a bucket to me.

For example, I recently outsourced my mail hosting to Fastmail precisely because I want experts to run things. Would I be happier if the data were stored somewhere I control? Definitely. But not if that means the experts aren't paying attention anymore.

Sandstorm's design is a little bit menacing! And it does require a fair bit of work sometimes to fit web apps not built for Sandstorm into Sandstorm's model. (The holy grail here is apps built for Sandstorm, but the platform needs to be bigger before more developers do that.) Usually packaging for Sandstorm largely entails locking an app into a single-document model, and stripping out authentication (since Sandstorm handles it).

But the end goal is pretty well worth it: Any grain is incredibly secure by default, and for the most part, app vulnerabilities are irrelevant. A grain where only you have access doesn't need any sort of authentication or security in the app at all. And since each document is it's own sandbox, sharing a document with someone doesn't give them a way in to exploit access to your other documents as might happen with a vulnerability in a more traditional design.

The business model story for selling Sandstorm apps isn't super great right now, you probably could have a licensing model that requested network access through the Powerbox to check the license or something, but in many cases, there's already a wide variety of great open source apps that are free and just frustrating to host and manage without a platform like Sandstorm (or Cloudron). (EDIT: Now that I think about it, Sandstorm used to have a paid license key/feature key system that made no callbacks, I think the licensing info was encrypted asymmetrically.)

As for your support of SaaS data collection, I just can't really agree with you: People who want to give data to a developer can choose to do so, but I think it's ethically wrong to collect data without permission. (Sandstorm servers do have the ability to opt in to provide basic app usage data back to Sandstorm's development team.)

I love FastMail, and have been an enthusiastic customer since 2016. :)

I feel the exact same way and am currently trying to solve this by decoupling the application from the storage. So the "experts" control the business logic and UI, while I, the user, store the data. It's similar to what Firebase does for app developers but is split by the end user who can also control that data.

> But the part that really concerns me is that they seem to think that server apps can run like mobile apps. To me one of the most powerful things about SaaS products is that the aggregated use...

That definitely seems like the current state of affairs but I think there isn't a reason why writing server apps like mobile apps has to inherently throw away aggregated data. Especially with Google's research in federated learning, apps will soon be able to get insights across their users while preserving their privacy.

Please don't past-tense Sandstorm! The community is still working on it. Albeit a little slower. ;)

I actually didn't realize that. I think I confluated the death of Oasis with Sandstorm in general. I'll have to poke into running it:) Thanks for the correction!

Maybe something like this https://remotestorage.io/ becomes popular. I would love to make apps that use user owned universal storage/server.

> Fun todo: Install this somewhere, nmap it for open ports, then ask "How many of these services had a remotely exploitable CVE in the last year?" "If one of these services had one tomorrow, would I know to patch it and take action faster than someone would takeover my box?" I don't see any containment mechanisms on any of these services beyond what's included by default so a compromise of one service likely leads to total compromise of the entire box.

This is the same concern I have with self-hosting anything with sensitive personal information on it. Without continuous monitoring, alerts and periodic review of audit trails, it’s anybody’s guess what’s going on with all the self-hosters’ data. With larger companies that provide a SaaS solution, there’s a little more hope that someone is looking at this seriously all the time.

These are fair points.

If I were to provide services for me and my family (or for a small company), I won't make them publicly available at all.

I would have every device connected to them over VPN (OpenVPN, WireGuard, ZeroTier). Of course it would prevent self-registration, and would take some work to distribute keys — but by definition we are a small operation, so this is manageable.

No service would ever listen on a publicly accessible IP. The machine(s) hosting that would firewall off all other incoming connections, except the VPN and SSH for admin purposes. I hope I would be able to quickly address CVEs in these two services, plus the kernel.

A setup like this is already pretty standard with AWS, but you can reproduce it almost anywhere, including your own physical box(es).

The weakest link with this setup is the client computers. So inside the VPN you still need good security practices — but the attack surface becomes much smaller, and a DDoS becomes harder to pull off.

You're not expected to use and install all of the services offered by Sovereign, just as you're not expected to install all packages of your Linus distribution. I guess we need to make this more explicit in the documentation. Just pick services you plan to use during the installation phase as outlined in the instructions.

The documentation instructs you to pick the services you plan NOT to use:

> Don’t want one or more of the above services? Comment out the relevant role in site.yml.

fail2ban and rkhunter are in the kit, and that offsets some of the issues: you get some assurance and protection right there out of the box.

You can also comment out the bits you don't want from https://github.com/sovereign/sovereign/blob/master/site.yml before you run the top level playbook.

fail2ban is security theater. Turn off password-based ssh authentication, use keys only, and you’re done. You don’t need additional software for it.

For extra security, bind ssh to localhost only and run a tor hidden service on the machine for accessing it.

This is a pretty cool setup! I have been tinkering in the self-hosting world for awhile now and I would say my biggest piece of advice is to learn Docker. (Not trying to shill for Docker here or start a Docker vs. Ansible flame war.) In my personal experience, Docker images (and docker-compose orchestrations) are more pervasive than Ansible playbooks and when it comes to self-hosting for personal use, hosting all my services in Docker has made it much easier to deploy/maintain them. (Not speaking to business use cases since in that situation you should really have a deep familiarity with your tech stack and be able to roll-your-own Ansible playbook/Dockerfiles.)

Also, there are so many great FLOSS alternatives to Google Apps. This repo contains some, but here are some of my favorites:

* Drive/Calendar/Photos/Keep:

  - https://nextcloud.com/ (I prefer this over OwnCloud)

* Docs/Sheets/Slides

  - https://www.onlyoffice.com/

* Mail

  - https://mailu.io/ (basically a Docker-based deployment of Postfix/Dovecot/etc)

* Hangouts

  - Server - https://matrix.org/

  - Client - https://about.riot.im/

  - (I prefer Matrix.org over Jabber/XMPP)

* G+/Twitter

  - https://joinmastodon.org/

> Not trying to start a Docker vs. Ansible flame war

Don't worry, they serve very different purposes. You already probably know but Docker is for running applications in isolation, while Ansible is for provisioning and configuring hosts. For instance, you won't use Docker to harden sshd on your hosts but Ansible.

Are there any good resources (besides OP) for using Ansible in a home-server environment? Especially in a mixed docker + Ansible environment.

After figuring out Ansible work I started using it to manage several personal VPS machines and the various boxes at home. I just used the Ansible docs, which are pretty good.

Just start a simple project with Ansible and have a look at the repositories using Ansible to install Docker and deploy containers on your host afterwards.

You may want to put some effort (not that much) into managing your credentials with Ansible Vault[0] and you can try your playbooks e.g. on a Vagrant Machine[1] before applying them to a real host.

Any tutorial will do for the beginning but you should always notice what version of Ansible you are using (vs. the one used in the tutorial) as features change and also there have been some changes to the syntax to improve readability of your playbooks.

[0]: https://docs.ansible.com/ansible/latest/user_guide/vault.htm... [1]: https://www.vagrantup.com/docs/provisioning/ansible.html

https://github.com/davestephens/ansible-nas

Uses ansible + docker to setup and run a home server. Its feature-set is close to that of a typical FreeNAS setup in a home environment.

The official doc, StackOverflow and /r/ansible are good resources imo.

They definitely have overlap. People use Ansible for much more than hardening sshd.

Yet they are both security disasters.

How is docker container, running a process as unprivileged user with reduced capabilities security disaster? With Kubernetes you can even have network policy or service mesh with mTLS. Compare it to bunch of bare processes running on the same machine, lots of services listening on localhost, difficult updates due to different dependency version requirements by different software, requiring you do the manual, painful update only once a year. It may look bad at first, but container can be far more secure that common non-container environments. It's always matter of configuration and maintenance.

It has a huge attack surface, also a lot of its code runs with high privileges. There's plenty of documentation on this.

How is Ansible a security disaster?

A system designed to SSH into production host with high privileges and make changes automatically instead of enforcing staging...

That’s not really a security disaster unless you consider all of SSH a security disaster.

If you are concerned about Ansible SSH’ing in (which means you are concerned about any person SSH’ing in), you just do the standard SSH hardening.

...and yet SSHing on a production system to make configuration changes is prohibited in many large companies including some FAANGs.

Because you (or some intern hired a week ago) are 1 typo away from making a disaster.

For the record, security includes availability and this is an availability risk.

Sovereign can be used as is to create linux container with LXD without any changes. You can create container image and than use it across your immutable infrastructure.

Don’t waste time in Docker, for this purpose use lxd containers [1]. LXD containers are more secure than docker in general and provide support for unprivileged containers for over 5 years. You can use your ansible scripts to create and manage your container images the way you manage bare metal or a VM. Indeed the modern LXD can help you seamlessly manage VM or Linux container [2], because when you need to directly access underlying host hardware VM is still more secure than container. Also with mount syscall interception in version 3.19 of LXD [3], it is possible to mount NFS inside container in a more secure way with unprivileged container running in user namespace.

Initially docker itself was built using lxc [4], but then decided to re-invent the wheels to build its own libcontainer without any significant advantage over lxc, just NIH. Obviously given all the money which went into docker and than kubernetes using it. Docker is more famous in spite of being inferior because of the marketing money spend on it like Java (a language which is famous because, sun spend over 500 million in early years of its inception on marketing). Kubernetes is famous and a valid tool for google level of problems for 90% of startups LXD is a better fit. This is the same fight like old times when inferior technology wins due to sheer marketing like Blu-ray won but rendered not as useful, same is docker.

[1] https://linuxcontainers.org/

[2] https://discuss.linuxcontainers.org/t/lxd-3-18-has-been-rele...

[3] https://discuss.linuxcontainers.org/t/lxd-3-19-has-been-rele...

[4] https://vyomtech.com/2014/03/04/docker_and_linux_containers_...

Come on. I like LXD, and I used it before docker. Docker is a standard and community. You can find prebuilt containers online and layer your own additions with a single configuration line.

Don’t get me wrong, Docker has made some stupid technology decisions, but the network effect of so many adopters means that it should be the default choice for any container situation.

> You can find prebuilt containers online and layer your own additions with a single configuration line.

That's the kicker that got them over the line. Look at usage graphs and docker is running circles around Linux containers. I use docker professionally but privately will use lxd barring some complicated setups that are a docker pull away.

Truly can't stand some of their design choices, eg the utter distain for iptables and even non-technical ones like requiring signup to get the daemon on Windows/Mac, it's frustrating software overall whose only saving grace is the ecosystem around it.

> ...the network effect of so many adopters means that it should be the default choice for any container situation.

People assume that just because Docker and lxd do "containers", they are somehow equivalent and it's simply a matter of choosing between the two. But they are far from equivalent. It's not a choice between two competing technologies that achieve the same goal at all.

That’s what I wanted to say docker is only designed with specific goal to have one process running per container. So to run a system like Sovereign will need many many containers, one each for each daemon and then environment variables for each config settings. It’s a nightmare to manage. LXD is designed for multi-process with proper init and do not suffer zombie process problem of docker and unlike Docker supports unprivileged containers.

Personally I will prefer model like Guix System instead of container to run systems and services.I find that solution to be much more elegant than Linux containers or distributions. Linux containers initially came as chroot, jails and zones and than addition of cgroups and namespaces by google made it popular as lxc and later adopted and forked by Docker to make something complicated. These are bolt on solution to have immutable infrastructure. Guix is designed from ground up to be new OS for 21st century server and application infrastructure. May be it will or one of its derivative will become mainstream in time to come.

What makes it a nightmare to manage? My guess is it's all about how you do the declarative config, that fight is age old: http://mikehadlow.blogspot.com/2012/05/configuration-complex...

"LXD containers are more secure than docker in general"

Please explain. This statement doesn't look correct to me. Both use the same technology - namespaces and cgroups. LXC is just meant to host the full OS installs, so you have to manually do things like "apt upgrade", resolve all breaking changes manually etc. So you end up with bunch of VM-like full OS installs, taking lots of time to manage.

Docker is basically the same, except there are layers of filesystem data and that those base OSs are minimal. Minimal also means less attack vectors, btw. Now, in the image there are all the required dependencies and you can prepare new version in your laptop, resolve breaking changes, test it properly and then easily deploy.

So why are LXC containers more secure?

LXD/LXC by default launches unprivileged containers in user space and map the userid’s to hosts user id’s, now they use new kernel feature shiftfs which makes it much easier to provide consistent user id mapping inside a container across hosts, with consistent security model, it doesn’t exist in Docker as it was designed from beginning not to have init process which results in zombie processes inside docker containers and recommends only one process per container. Docker doesn’t by default launches containers running in user space, and there are so many design difference as I posted a link in another post which really makes it a security nightmare unless company like google provides you managed container services taking the burden away from you, but indeed the security vulnerabilities which impacted all k8s didn’t impact LXD running unprivileged containers.

Every image in LXD can be locally hosted including the base one privately, so do not need to rely on inspecting a hotch-potch of Dockerfile, scripts and pull from other docker images to know what’s inside.

Now with the release of LXD 3.19 they introduced interceptions of syscall so even unprivileged containers Running in user space can securely access hardware. So NFS can be mounted inside unprivileged container I haven’t tried the latest docker container but in the old one cannot mount NFS from inside a container without running it in privileged mode with kernel access.

[1] https://linuxcontainers.org/lxc/security/

The two are not incompatible. You can deploy containers with Ansible.

Indeed:) Ansible will happily interface with pretty much anything; consider that it officially supports Swarm (https://docs.ansible.com/ansible/latest/modules/docker_swarm...), k8s (https://docs.ansible.com/ansible/latest/modules/k8s_module.h...), and puppet (https://docs.ansible.com/ansible/latest/modules/puppet_modul...) for example:) It makes excellent glue if used with other things, or a good tool all on its own. Which is especially nice because it lets you try new things without changing your overall setup too much; you can deploy all your services with RPMs, and then move one thing into docker, and still use ansible for everything:)

I run everything in containers and I spin up all the images / containers from ansible scripts. Am I doing it wrong? I didn't think these two tools were in conflict.

They do complement one another but docker has tooling that solves similar problems. Does docker-compose (https://docs.docker.com/compose/) fit your needs?

I use ansible to build a machine from zero software installed on it to running multiple docker containers that also make use of various filesystem stuff on the host, as well as set up other non-docker things on the host like networking, selinux stuff, and crons. I like that ansible is not hardwired towards everything being a container.

I tend to run ansible to setup a VM, e.g. a few packages, lock it down to my taste, and then have ansible run docker-compose to deploy the application(s) of interest.

You wouldn't use docker to manage the host networking (which is where ansible comes in), but packaging whatever is listening on ports as a container works really well for me

No need for an Ansible vs. Docker argument when you can combine them both!

I run a similar project called Ansible-NAS - https://github.com/davestephens/ansible-nas - which originally came about because I fell out of love with FreeNAS, and felt I could do a better job with Ubuntu, Ansible, and a bunch of Docker images.

Sovereign is awesome, I've been watching it for a while, but I'm not keen on everything being installed directly onto the system which is what I tried to solve with Ansible-NAS.

</shameless_plug>

Docker use Dockerfile (ini) to create container images, so you need to create a soup of Dockerfile, bash scripts combine it with ansible scripts. So it is kind of incompatible, it's better to use LXD containers as I suggested in my previous post for this kind of purpose. Also if you know Docker always maintained one process one docker container so a multi-daemon system with SSH and various mail and groupware servers running is not right choice for Docker.

Using docker for such things is like putting a square peg in a small round hole.

Dockerfile format is not ini. https://docs.docker.com/engine/reference/builder/

It also doesn't usually make sense to put SSH in a docker image or container, as you can enter the container using docker exec

> I would say my biggest piece of advice is to learn Docker. (Not trying to shill for Docker here

The first step to learning Docker by downloading it, irritatingly, is that getting Docker Desktop on Mac or Windows requires creating a Docker Hub account and signing in! There’s even a long issue thread about this on GitHub and the response was totally irrelevant! Luckily for those who don’t want to jump through these hoops (through disposable addresses or reusable shared logins) or provide an email address, there are many people who have posted direct links to the different downloads available.

I remember when they changed that and the GitHub issue was one of the most hilarious examples of corporate doublespeak.

"Why are you doing this?"

"To improve user experience"

"But now I have to sign up for an account whereas before I could just download"

"Yes this will improve the experience for everyone over time"

I think you can use chocolatey to choco install it without going to the docker site.

> In my personal experience, Docker images (and docker-compose orchestrations) are more pervasive than Ansible playbooks and when it comes to self-hosting for personal use, hosting all my services in Docker has made it much easier to deploy/maintain them

Are you building every container your self? I hope so: https://blog.banyansecurity.io/blog/over-30-of-official-imag...

:) Yeah, that was one of the big reasons I emphasized that my setup was focused on personal use. I have decided that the potential vulnerabilities present in the official images that I use present an acceptable level of risk that I am willing to assume. Enterprise deployments and other people with different personal levels of acceptable risk will require a better (but much more time-intensive) setup.

Thanks for sharing these list of FLOSS alternatives. Have you had much experience with MailU and anti spam / deliverability issues? My needs are only a few emai's a week, but previous tests with running my own mail server resulted in a high % of emails ending up in client spam folders.

Are there any resources you recommend for learning to use Docker specifically for self-hosted services in a home lab setting?

IMHO the official Docker documentation is great! Their "getting started" page is a good overview of the Docker basics. https://docs.docker.com/get-started/ The CLI --help is pretty useful too. I have also found Docker's in-depth documentation to be very helpful when trying to figure out the more complex features.

I got started by just grabbing a $5 DigitalOcean droplet (can get them with Docker pre-installed) and then played around trying to setup a simple app. (I think it was RocketChat.) https://rocket.chat/docs/installation/docker-containers/

https://www.linuxserver.io/

not a resource but docker containers you might want to run in your homelab. checkout awesome docker on github or katacoda

For my fellow HNers, this is "private cloud" in the meaning of Owncloud, not of Openstack.

I hear about Nextcloud a lot more than I hear about Owncloud. Does anyone know why this project uses Owncloud instead of Nextcloud?

For some history: ownCloud is a very old stable product. 2-3 years back it got forked into Nexcloud because the founder of ownCloud had some disagreement about ownCloud's direction.

Both the products are backed by companies and both are doing quite well. I would say Nextcloud goes more and more into expanding it's use case and thus makes it product more extensible via plugins. This can be good or bad depending on how you look at it. Plugins go unmaintained/incompatible over time and are a constant source of pain when upgrading. Wordpress gets away with this because it has a massive community.

ownCloud on the other hand has decided to double down on it's roots of file sharing/syncing. I heard they rewrote their stack from PHP to Go now and the frontend is now React.

Based on how long this project has existed, I suspect they chose owncloud before nextcloud was forked and changing it hasn't happened/some people don't want to change their already working installations.

This project started before Nextcloud existed, I don't know why it hasn't switched, though.

NextCloud was started as a fork of OwnCloud by a bunch of OwnCloud devs that were unhappy with the direction the product was taking.

You can think of it in much the same way as OpenOffice vs LibreOffice: devs fork to make a new product, the "original" product stagnates and is mostly used for rent-seeking.

The downside of both is that, to my ears, both "OpenOffice" and "OwnCloud" better signify to outsiders what the product accomplishes, while "LibreOffice" and "NextCloud" really don't, unless you're already familiar with the product or product history.

Also Nextcloud puts a LOT more effort into publicity than Owncloud.

I think there's an open issue for the switch. Just need a good pull request by a volunteer...

Wow, didn't expect to see Sovereign at the top of HN today! I'm one of the project contributors.

If this project piques your interest, please consider contributing! We could really need more helping hands.

Ansible is easy to learn and most (not all!) problems due to new versions are easy to fix.

Also, if you only want to use a fraction of what Sovereign has to offer to reduce your server's attack surface, that's easy! Just follow the instructions.

On the general topic of Ansible and personal infrastructure:

Every time I attempt to use Ansible (or its kin) to manage my own network, it feels overly obtuse and ultimately unhelpful. Its gains seem to be rooted in configuring a large number of identical servers, and isn't geared for a handful of hosts with some commonalities and some differences. Writing playbooks feels like a still-imperative wrapper around shell commands, just in a bespoke and verbose YAML syntax.

Instead I am using my own script that runs a tree of files through a template engine, drops them on each host being configured, and then runs triggers based on what has changed. This seems utterly simplistic, lacks polish, eschews common practices, etc. But the overall configuration seems straightforwardly grokkable compared to the heavy tools.

Not a fan of ansible's yaml, but when I setup my personal sever (1 websever) with ansible I though it was pretty cool I could duplicate my entire setup in a VM for testing with one command. Then it saved my butt when I got locked out of my VPS instance cause I lost the key. Just spin up a new one and run ansible. I was back online in less than 5 minutes.

Not saying you couldn't do that with custom scripts, but I found that when I tried writing my own admin scripts I was solving problems that the ansible team has already solved. It's a matter of what you want to spend time on IMO.

I had a similar reaction, after trying out Ansible at my last job. We ended up switching to fabric, which is all in Python. It was mostly good but had some awkward warts, which it seems that they've mostly addressed in fabric2. Anyway, it might be worth a look, based on my understanding of your use case.

Fabric seems more down to earth, but doesn't itself solve the problem of actually defining the configuration of each host.

Having said that, I am to the point where it would be really nice if my ssh pushes ran in parallel, which is one of those robust niceties you give up by going your own way. So I'll have to revisit Fabric because it would be complementary - thanks for the reminder!

It definitely is target at a large number of identical servers, but is still very helpful for just a handful of hoses.

Putting everything in YAML is helpful because no matter what service you are setting up, the format is the same to read and understand. And I hope you are using modules and not shell command directly (except for the small cases it is necessary)

Ansible can be as simple or as complex as you made it. I bet your script isn’t as nice as an Ansible setup, nor as maintainable. Ansible’s templating engine is super flexible.

>I bet your script isn’t as nice as an Ansible setup, nor as maintainable. Ansible’s templating engine is super flexible.

Ansible uses jinja2, and so do I. I obviously can't form an objective opinion on the "niceness", having developed it. But I'll say that my setup puts which files are on which hosts front and center, whereas it seems like Ansible would want that splayed out into a directory tree of "roles" (of which I'd have about 25). With my setup, I've got that in 200 lines of python including host/group definitions (excluding comments), with the actual config files living in one analogous tree (eg conf/etc/network/interfaces).

Half the reason I threw my comment out there was to see what bounced back, as I am currently working on this system. One of the responses was about error checking, which resonated. So I've since given Ansible another shot, for managing the top-level (calling my templater for each host, apt upgrade, etc). I think I do like it for this, now that it's much closer to its sweet spot of doing similar things to every host.

I never really saw Ansible as a heavy tool. But a place I worked at in the past used cfengine, so my barometer for this is probably a bit different.

You had a point about error checking that I think is pertinent to my not being in Ansible's niche. If a service fails to start on a single host, it's because I'm already tinkering with that configuration and hence will notice the failure. Whereas with tens of highly similar hosts, you want to find out about exceptions sooner. I'm actually getting to this point with some common things (apt upgrade), hence looking into Ansible again recently.

Overall my goal is to write very few on-host configuration scripts, and prefer overwriting files. For example, most triggers are simply service restarts, which can also be performed by a full reboot.

You should check out Saltstack if you haven't already. It's actually not far from what your doing now...

For those in the market to run a private cloud, please try https://cloudron.io . Our motivation is to make it simple to selfhost apps . The main advantage is that we take care of automatic updates across all the apps we package. Happy to answer any questions.

Disclaimer: I am the co-founder

For anyone looking for a free alternative to this, check out: https://caprover.com/

It has many of the same apps as Cloudron and is completely free. And you can of course host your own Dockerfiles on top of it.

I think the pricing for Cloudron is way off. I'm not going to spend 5 dollars per month for a DigitalOcean droplet and then 30 dollars per month to host a few open-source apps on that Droplet. Especially since CapRover does 99% of this for free.

The target audience of caprover and cloudron is totally different. caprover is a PaaS. Cloudron lets you install ready to use apps like Ghost/GitLab/Discourse/OpenVPN etc. A person who knows how to deploy using docker and more importantly has the time to setup stuff securely, does not need either of these tools. Even there, people will argue why you need docker and why you cannot run with LXC. This goes as deep as you want to go :)

As for the pricing, agreed that it might be out of reach for personal use. I think it really depends on what you get out of the product. Our target at the moment is primarily family/small businesses/IT teams. For example, a business runs website/file sharing/mail/chat/analytics/crm/forum on a droplet. The server would be around 20/month (4GB server) and then 30 on top of it. For personal use, all my stuff is on Cloudron (calendar/contacts/email/blogs/website/rss/media/docs/notes). I cannot imagine relying on a product which is 5/month for all my stuff.

Besides, having complete control and owning your data is priceless. If you don't value that, I am not sure why one would self-host at all.

Is it really self-hosting if you're running it on someone's cloud?

Definitely.

I mean, you could play an infinite regress game. Do you own the hardware? Do you own the cage the hardware is in? Do you own the building that the cage is in, and the land that the building is on? And then we can go toward owning the power company and the connections to anybody your servers talk to.

But in practice, self-hosting is about control. If what you're running it on is a commodity cloud instance that you could get from a half-dozen providers, then any one cloud provider has very little leverage over you.

No. If you have dedicated hardware (rented or owned) and full disk encryption you have decent control over your data. On a virtual server you have no control and no privacy.

Depends on what you mean by control. It sounds like you're worried about different downside risks than I am.

I think like many other terms like 'cloud', 'private', 'start up' there is a lot of gray area :). It's up to you where you draw the line. For me, self-hosting means running software in a manner where I have control of the data/application code and the server. With that definition, running software on EC2/DO/Linode is self-hosting. When I self-host using these servers, I know what the server is running and where the data resides.

Also, I think there are other similar popular terms. For those who run in their own premises, the term is on-premise. For those running it home, usually they call it home lab/NAS/home server. Self-hosting to me encompasses all this.

Also, self-hosting doesn't necessarily mean just open source. There are some amazing closed apps out there that you can self-host - emby, confluence, teamspeak to name a few.

Two of my favorite spots - https://github.com/awesome-selfhosted/awesome-selfhosted and https://www.reddit.com/r/selfhosted/

I think it's also important to highlight that perfect is the enemy of good. I moved from Google services (A) to an open source platform... that I was paying them to host on Google Cloud (B). Sure, it's still at Google, but the ability for Google to mine data from it is significantly reduced. Of course, later, I moved to hosting it in-house (C), where I have direct physical control.

It's likely if my choices were A or C, I'd have never left A. But that B option eased the transition for me, and made it possible for me to get to the point I felt like the investment was worth it to create a fully on-premise solution.

There's a part D to this too, actually: I'm still using a service to manage the DNS and TLS for it. Eventually I should be able to move away from that too. But without the intermediate step, it'd be too prohibitive and frustrating to have moved to step C.

Great point about D. I self-host at home (it's a thinkcenter m600) but it depends on external services. I use DO DNS and Let's Encrypt TLS, backup to S3. Not to mention I rely on comcast for internet and public IP!

Interesting stack. From a first glance it seems to be a bunch of self-made Dockerfiles with an extra manifest. I like the idea, but wouldn't want anything proprietary for this relatively simple task.

I suppose this product isn't for me anyway, since eventually (not that long ago) I just bit the bullet and learned the basics of Docker and docker-compose. It's not that hard, costs nothing and is pretty rewarding, imo.

Yes, that's the core of our tech stack. All our packages are opensource (git.cloudron.io). Important thing for us is to have all the docker containers read-only and have a 100% reliable backup/restore mechanism because we do automatic updates for apps to give a SaaS experience to self-hosting. We also provide SSO for users across apps (so you can add your family or colleagues in a cloudron and do access control across apps easily).

This looks super cool.

Though somewhat offtopic, this line absolutely cracked me up.

>A VPS (or bare-metal server if you wanna ball hard).

I can appreciate a sense of humor.

There seems to be a few questionable picks in the readme.

* Why pick ownCloud over NextCloud? The former's forum had 139 posts in the last 7 days and the latter's forum had about 1700. Also some of the features in the former product are locked for enterprise only.

https://central.owncloud.org/about

https://help.nextcloud.com/about

* Tarsnap is a paid online service. You could try restic command to have encrypted backup to remote storages.

* cgit is an old project released more than 10 years ago and despite being written by the author of wireguard, we have far better stuff like Gitea (or its fork source Gogs) to have user access control with nice web interface for git project management.

I currently just let my desktop run constantly. It runs Windows in a KMS for games and sleeping breaks it so I never even suspend. It only seems to use about $10/month in power. I have been considering using DDNS through my router and Cloudflare and trying to create a iOS/Android app that will automatically upload my photos to the my DDNS for storage on my desktop. It feels a little crazy but the idea of syncing my photos to my own machine with no middle man is comforting.

DDNS seems like it's a local too good to be true for solving the dynamic IP problem. I'd prefer to have a static IP for my gigabit Internet but sadly Webpass doesn't allow it. Does anyone have experience doing something like this?

Ngrok as alternative to dynamic DNS?

DDNS maps a domain to your current dynamic IP. Its built into most routers. They will push the new ip to a DDNS service when it changes.

I looked at Soverign at the time I was setting up private cloud on a $5 VPS (prgmr.com with 1.5 gb ram)

I went with YunoHost.

https://yunohost.org/#/apps

I initially tried sovereign, but once I figured out I had to pay for tarsnap backup service, and that it did not have ansible for nginx setup (I needed that experience for work stuff), I went with Yunohost.

Sofar I am happy with YunoHost and subscribed to send periodic donation to the project.

Overall, though, if you are working with ansible at work, or want to advance in devops field, learning ansible and contributing to Sovereign project would be a good path to take.

Dumb question maybe... but why would you not just configure this yourself on a single virt/host? Most of these services would take less than a day to configure. So many questions on why this is a good thing.

Like, there are countless ways to configure your MTA and spam filtering- if you are going to have to dig through this config.. why not just roll your own?

Can someone explain to me why you need ansimble for this? or am I just being stupid and this is like an exercise to show what the toolchain can do?

Because if you want to switch providers, you’d have to do that all again. Or if your current provider’s VPS just dies (I’ve had it happen) you’d have to do that all again.

With Ansible, you just run it again and in a fraction of the time its back up.

If you're the type of person that configures servers on your own time for fun, than you would likely find automating the process 'just because' to be fun as well.

You do not need ansible for config of a personal server at all.

Oh cool cool, I thought I had a stroke or was missing something here. This is a cool idea in that case.

As the CTO of an established company, I cannot imagine a situation where I would prefer to maintain my own infrastructure vs using managed cloud services.

If I get locked in on a specific product, it's way cheaper to redesign that around an alternative vendor than it is to maintain a private cloud (Ansible, Kubernetes and friends included).

As a nerd, I'd prefer to do things myself, but I have business needs to attend to.

Cost. Depending on your scale and needs, running stuff on real hardware you manage can be a lot cheaper than AWS/GCP.

Performance, cost, security. For small businesses, it may be a wash (due to ability to hire system engineers), but for medium to large business rolling your own is almost always better.

Just depends on workload. Especially if your biz is bandwidth intensive that can be your largest single line item on the major clouds. Then you gotta pay engineers to find ways to reduce your bandwidth costs and the snake eats its own tail

Is there a benefit to doing this over something like Cloudron(1)?

I see this being for people that just want things to work without much of the effort to make it so. If that’s the case, a simple web UI that treats all the little solutions as “apps” in a way makes sense. Not plugging here, just curious to the practical everyday differences.

1. https://cloudron.io

Cloudron.io seems pretty nice, speaking as someone who uses/works on a different platform (Sandstorm.io). Cloudron is going to cost you money if you want the benefits of automatic updates and the like, which is a downside for some (I think paying for good products/services is worthwhile), but I absolutely think making management of a ton of apps simple like a phone is the key to self-hosted online services. One of the biggest features of a good self-hosting platform should be unified authentication and identity, so you aren't managing your account on a dozen different apps.

I've always been impressed by Cloudron's well-maintained app library and constant march of major feature improvements to the platform.

Just wanted to mention that Sandstorm is awesome. For some reason, despite being around for years, I only found out about it recently. I really think that it should be way more popular than it currently is. Lots of apps are outdated, so I’m planning on investing some personal time to contribute to it.

By all means, hit us up on the mailing list or in IRC if you have any questions or need any help!

I've used this project off and on for years, and it's always worked perfectly. I'd have an infosec conference to go to, I'd setup a host with wireguard, give my friends the cert, we'd all tunnel out though that, then tear it down after the con. Total cost a few dollars (most are $5/month)

Does anyone here on HN have experience using Sovereign in a team setting? I have a few questions:

* Mobile contact and calendar syncing: How well and reliably does it work?

* Calendar group features: how well do they work?

* Setup and maintenance: how much hassle is involved?

Doing that stuff is hard.

Sovereign doesn't solve all your operational problems.

I think it's suitable for personal use. I wouldn't run it in a production setting without thoroughly understanding all parts of the stack.

I would say it's good for personal use or to demonstrate what ansible is capable of.

Isn't this just duplicating effort that's probably already been done, in many cases by the first-party maintainers, in Dockerfiles?

I don't mean to start 'Docker vs. Ansible', I just wonder why if you wanted a quick way to setup a single-server 'own private cloud' you wouldn't just go with what already exists, and list the images you want in a docker-compose.yaml file?

(Which would additionally set you up for 'scaling' if you had any concern that you might be able to save some cash with a two or three smaller servers than one big one by the time you'd installed everything you want.)

Isn't docker just duplicating effort that's already been done in LXC or jails?

Are LXC and jails polularly used by maintainers of projects like owncloud, readlater, and such that Sovereign installs?

I explicitly said that I wasn't making it about Docker vs. Ansible - I don't care - I just mean that Docker is very often used by first party maintainers (and if not by someone else) to package these services, so use what's there; if it had happened to have been something else that took off in that way, an Ansible playbook say, then that, but it's Dockerfiles that are in that position.

Not everyone wants Docker installed on their servers. It’s not a requirement with this playbook.

Furthermore, if you do want to use containers, there are tools like ansible-bender[1] that use Ansible to build container images.

[1] https://github.com/ansible-community/ansible-bender (edited the link to point to the ansible-community repo)

But I'd have thought if you care that Docker isn't installed on your servers, you're probably not running this anyway?

It seems to me that the target demographic is people that just want the least effort minimal faff way of getting some services up and running for personal non-production use. And for that it was my suggestion that many of the services probably already provide a Dockerfile upstream, so the easiest thing to do would be to install docker-compose, list the images, and `up`.

This project has nothing to do with Docker, and there are many reasons why someone would want to avoid running Docker on a server (a separate issue from using containers). I'm confused about why you think Docker is relevant to this set of Ansible playbooks, and also why this would only be suitable for "personal non-production use."

The only requirement for a remote host to be managed by Ansible is python, and even that can be installed by Ansible itself using the `raw` module on an initial run with nothing but ssh access.

No need to gather a bunch of random Dockerfiles from various places, tweak them to be compatible, and create a docker-compose file from scratch... how is that "the easiest thing" when this is a complete set of Ansible playbooks where the work is already done?

I meant easier than creating these Ansible playbooks from scratch, when many if not all of the playbooks will do the same required steps as have already been encoded in Dockerfiles in many cases by the maintainer of the upstream project (the one being deployed) itself. There certainly could be value in collecting that in one place still.

The first line of the readme says it's for a 'personal cloud'.

The initial commit of this repo is from August 2013, a few months before the first public beta release of docker-compose.

This looks useful. Is the fork more out of date than the source it was forked from, or am I reading the dates wrong?

Oh, good catch, I would use https://github.com/ansible-community/ansible-bender

I just copied the first link that showed up in a search, which looks like it's the personal repo of the project maintainer but is pretty far behind the upstream repo now.

This is dated 2015. I hadn't heard of Docker in 2015.

That's as may be, but it did exist, and was popular.

Nice resource, will definitely give it a spin, also this is the first time I've ever come accross `Tarsnap` looks really interesting too !

Nowhere do I see mention of how to update and upgrade this thing after initial deployment... How does it work ?

I've used Ansible successfully to turn Windows laptops into Wintendo machines :)

One of the most important things when managing data is not losing it. Does this have a solution for doing backups built in or do you come up with your own solution like using something your web host provides?

I haven't used this myself but it looks like it uses nightly encrypted backups using Tarsnap

Did you miss the bit about tarsnap?

Apparently I did. Oops.

Version for a Pi.

Version for BSD.

I wrote something similar that's custom for my personal infrastructure:

https://github.com/sumdog/bee2

There are some blog posts in the README that go into how I built a lot of it. A lot of it is specialized for me though. I have a ton of rspec/tests but I don't have a real config schema or entirely useful error messages. I might add some in the future.

Looking at the list in this, I'd advice against nextCloud(ownCloud). I recently setup their official Docker containers and the web piece works alright, but their F-droid app continually crashes and I had to uninstall it and the nextcloud-client in Gentoo's package manager segfaults at home and refused to build at work.

I've read other stories of data loss with nextcloud. It might be better now but my initial experiences made me use syncthing. Syncthing does use relays if you're behind a NAT, but if you have openvpn setup, you can also force it to use a direct IP address as well.

If you're thinking if self hosting and have the time, I'd suggest building it yourself; borrowing (and properly accrediting/licensing) other open source projects, their ansible scripts and containers and such. You learn a whole lot about why this tooling is so complex.

Quit this damn nonsense with Ansible or whatever garbage fashion fad is in vogue these days and finally learn how to make OS packages so you can do configuration management with normal shell scripting inside of them, because that's what it's for. The amount of incompetence and insanity from incompetence has gone too far. Damn it, this is exactly why IT sucks so bad!!!

Maybe you've never worked with someone that sucked at shell scripting, or packaging.

Ansible is awesome for enabling people to do reasonably complicated things in a consistent manner, at scale, without having to write all of the boilerplate code to be able to do so.

This is forgetting the fact that Ansible is reasonably opinionated, which is great for lowering the barrier to entry and helping devs/admins to be productive quickly.

When I just need to Get Shit Done, Ansible is awesome.

I've worked with plenty of people who had no business working in IT, but some of them were smart enough to recognize that they could benefit from education, so I taught them how to correctly and properly program in shell. Others were just insecure assholes who didn't care about working with computers and were in IT just for the money but didn't want anyone to know they're incompetent (even though it was obvious). Those always wanted to needlessly spend money on buying shitty software or using "tools" like "Ansible".

> lowering the barrier to entry

That's the opposite of a good thing.

> When I just need to Get Shit Done, Ansible is awesome.

"just" is the keyword. "just" instead of caring about long term maintainability and security

LOL on your first point.

For your second, Ansible is specifically designed for long term maintainability and security.

HN has decided that running arbitrary commands on your production systems is OK.

And that doing packaging, staging with CI/CD and immutable infrastructure is unnecessary.

FAANG companies clearly disagree.

HN decides a lot of dumbass stupid things, which is precisely why the dumbest ones end up on n-gate.com and elsewhere (exempli gratia:"Twitter") being the butt of jokes. It's not shameful not to know something, but it's downright abhorrent and disrespectful acting like one knows and refusing to learn from those of us who have gone through the pain and suffering with stupid things over the decades. A person should constantly learn and strive to be a better person tomorrow than one was today, but no... "HN" cares more about being trendy than actually designing and implementing hardware and software for longetivity.

Reply on news.ycombinator.com