Taking back control of my digital life | Hacker News Reader

Taking back control of my digital life

screamingatmyscreen.com

105 pointsfallenhitokiri71 comments

I've been steadily moving everything back home ever since Snowden revealed just how much snooping is going on, and with the last few years worth of scandals, it seems i've made the right choice.

Initially i went the Raspberry Pi route, and in a few months i had 4 or 5 of them running, which was where it started turning into a chore. While i enjoy tinkering, i don't particularly enjoy keeping a host of machine up to date.

Fast forward a couple of years, and my home setup now looks like this :

- UBNT EdgeRouter for that sweet affordable hardware L2TP/IPSec VPN.

- Intel NUC6, Running everything internet facing in FreeBSD Jails.

- Intel NUC7, Running everything internally available through Docker, along with Time Machine backups.

- Synology DS716+, Holds all data from both NUC machines, as well as personal files (documents/photos/music/movies)

Resilio Sync is running on the external NUC to provide an "always on" node.

I have a couple of ODroid HC1 boxes that holds my backups, one at home and one at a remote location.

The only thing i have yet to migrate is mail, which is still located at GSuite. I've moved several times, but every time i decide it's not worth the effort. Since "everybody" uses GMail every mail i send will eventually get indexed as well, regardless of where i store my own mail. I use GSuite for hosting 5 domains, and i've yet to find an alternative that doesn't require either lots of money or lots of administration from me.

Sorry to go off topic, but could you explain how you;re doing Time Machine backups to the NUC - sounds very interesting.

Nothing magic. I've installed netatalk from the guidelines on the wiki http://netatalk.sourceforge.net/wiki/index.php/Install_Netat... and simply expose a shared drive through it.

Netatalk is an AFP[1] server.

[1] https://en.wikipedia.org/wiki/Apple_Filing_Protocol

Much obliged.

Recent versions of samba support time machine as well (via vfs fruit), if you are using high sierra (maybe sierra or later?).

Sadly Debian stable is still stuck on Samba 4.5, and vfs fruit isn't available until 4.8, which is in testing, but who in their right mind runs a server off of -testing :)

Same story with netatalk, latest version is 3.1.11, and debian has been on 2.5 forever, with 2.6 in testing.

I've set up a personal mail server using Debian + Exim + Dovecot (with sieve). It took a bit of time at the start, but besides the couple of hours it took to add DMARC a couple of years ago, and checking the backups are working, I never have to touch it. I give out a different address for each company, so dealing with spam is just blacklisting it (only happened an handful of times).

My main issue is that my ISP blocks SMTP(S) ports, meaning i'll have to host it on a VPS somewhere. While i would avoid being indexed, i wouldn't truly own my data anyway. I have a 300/300 mbit fiber connection, so in theory i could mount the storage over VPN, but that just adds more places things can go wrong.

Also, i remember from running my own mailserver 5-8 years ago, that keeping it off various spam blacklists is a job in itself.

Yes, ISP restrictions are a pain. If they only block outbound connections to port 25, you could receive directly, and use a smarthost (on a VPS or an email service) just to send. That's what I did when I ran it at home (my ISP didn't block, but the IP range was listed on Spamhaus PBL).

As for blacklists, that's not been my experience at all, but YMMV.

What's your experience with spam filters from Google, MS, etc etc and (if it happened) the server being down? Do most senders retry mail delivery according to the RFC?

Haven't had problems with spam filters. I did spend some time making sure everything was good (SPF, DKIM, DMARC, reverse DNS, etc), and I don't send any bulk mail.

Yes, senders do retry; I got plenty of back mail after being down for a couple of days, back in 2015.

Recently I was wondering what if all our phones and computers all connected to our own personal "platform" that we have full control over.

Route everything on the phone through a personal platform. Gain control of location, IP, maps caching, etc.

Centralize at a personal level.

I’m working on almost this exact thing. Still a ways off but get in touch if interested.

A platform is useful for the other services it provides: Google funnels your data but give you better search, MS swipes some info about your PC to give you better drivers, Facebook takes all your likes and dislikes to give you stories to your taste. [talking about stated interest here]

What would your personal platform do? How would it take that data and turn it into something useful while also removing the privacy concerns?

Best you can do is have something that blocks or spoofs that data before sending it to the "collector". So you gain the privacy and you lose on the service improvement. If this is legitimate concern you can try to drop the most intrusive platforms from your digital life. These would most likely be Google and Facebook.

Alternatively there are options for you to use with more limited effect: ad and tracker blockers, secure or anonymous browsers, VPN, stricter privacy settings on your devices, etc. The end result is usually the same: the less data you provide them, the less useful their service is. Which makes trading your info even less enticing.

Yeah, that's the crux of the matter. I'm one of the rare (outside the HN bubble at least) tech users who try hard to retain control of data and avoid cloud providers. I've never used Gmail, I have a Google account but almost never sign into it, I don't have a Facebook account, I aggressively block ads, tracking scripts, etc, etc. It seems like the main cost I'm paying for that isn't the setup effort (setting up a mail server for instance is mostly a one-time effort), it's the lack of some really neat features. I don't get Google alerts relevant to items on my calendar, the photos I take with my phone stay there until I manually transfer them to my PC, I don't have an easy way to look up which restaurant I visited in another city 2 years ago, and the list keeps getting bigger.

I really like the idea of having centralization at a personal level, but it seems like the most you can get out of it is fairly hassle-free synchronization of your stuff between devices. Which is nice, but I don't see how it could provide Google-style features that are useful mainly because of the aggregation of many gigabytes of user data per day.

Yeah, that's what I was also curious about: to what extent can you use the data you collect from yourself to get some genuinely useful outcome? How do I get a better news feed, search results, restaurant recommendations, etc. using the data on my personal platform?

Because there are 2 sides of this issue: not giving up your data and using it productively.

One thing that seems reasonably possible is to get restaurant recommendations if you built your own app for that. You'd have to tag restaurants you enjoy or frequent manually, in order to avoid providing location data to Google, but with that input you could rank restaurants in any area by how similar they are to your preferred places. This has some complications anyway because Google Places API, if you rely on that, doesn't provide access to all the tags you see in Google Maps, and you'd have to replace some of that with freeform text search.

But stuff like search results? Auto-tagging items/locations in your photos? I was very impressed by the stuff Google Photos can do, but those features obviously rely on some very well-trained neural networks.

If there existed a promising "personal platform" solution, I'd happily support it by donating and hopefully code contributions, but I'm not aware of anything much better than Owncloud, which itself isn't dramatically different than using rsync/duplicity in cron jobs.

If everyone has their own centralized platform then perhaps there can be a mechanism that lets them share their personal data with others in a crowdsource way. Or better yet, perhaps everyone's platform can be parts in a large map reduce.

By the way, so many people will have the same problems, so it seems possible that solutions to common problems can be shared.

> Google funnels your data but give you better search, MS swipes some info about your PC to give you better drivers, Facebook takes all your likes and dislikes to give you stories to your taste

Google's search results are on steady decline over past years, result filled with ripoff pages, often original pages are buried deep in the list full of ripoffs. Started using duckduckgo much more because of this. MS drivers tailoring or whatever you meant are presumably a joke, I know which driver for which hardware I need and if not simple system scan should be enough. And FB feed is crap too, worst point was when feed was completely randomized on every reload, very frustrating. Even now its pretty bad - some posts hang on top for a week, interesting new stuff is buried somewhere deep.

I would be much happier without any of those. Good effect is, these 'platforms' are getting worse and I am using them less and less (FB is at the end anything but social), or using competition

> Google funnels your data but give you better search, Facebook takes all your likes and dislikes to give you stories to your taste.

These two are kind of questionable; I'm not convinced that the personalisation gives me better search results and I use FB exclusively to communicate with people I know.

What they really give you in exchange for monetising you is that they do the sysadmin work for you. That's not nothing and for the average user is really quite a benefit. It's just that people vastly prefer a zero-cost to even a low-cost provider, because it's easy to get started and harder to switch.

To an extent, this is kind of the idea behind sandstorm.io. Run the services you want on a box under your control (or a vps, but still yours).

Given that there are finally good non-google options for a lot of these (OSM + Thunderforest + Graphhopper gives a really nice map experience for instance) I'd love to see it go further. As always, it's hard to find time to work on this though.

Your carrier still has your location logs, sells them to third parties, and provides them to the government upon request. You cannot meaningfully control this data.

Out of interest, have any self-hosters here tried one big machine with lots of VMs, as opposed to a stack of NUCs/Pis?

The recent Intel scares have got me interested in firmware and Libreboot again, and a HN commenter[0] pointed me at some extremely beefy (e.g. 16 cores!) AMD motherboard/CPU combos, that it’s possible to run fully-free firmware on.

Without doing much actual research, Xen on a big box seems like it could be a good way to have a single physical machine where I ‘deploy’ different services in VMs.

Anyone gone this route, or can think of any problems with this approach? It seems like it could be a little easier to manage, but I’m sure there are extra considerations for networking and security.

[0]: https://news.ycombinator.com/threads?id=153791098c

I’m contemplating something similar! If you want you can even run windows in kvm with pci pass through for native gpu performance.

You might be interested in the https://www.reddit.com/r/homelab subreddit. The prevailing wisdom there is to buy used enterprise hardware, install a hyperadvisor, and set up separate VM's for everything. It's easy to scale, easy to tinker with, and really flexible.

OP here - all of my services run in a VM on the Dell server, including some other VMs used for work. Works pretty well, but you pay a slightly higher energy bill.

As long as only a few users (in my case between 1 and 6) use the services you'd be surprised with how little resources you can get away.

My initial test VM (VMWare) I used had 1 CPU and 1GB memory - it ran Postgres, miniflux, docker registry, gitea and wallabag without problems. CPU mostly idle, memory without a lot of load ~350MB. (Alpine as base for docker)

There's the security issue of general VM escape, and sidechannel attacks across VMs (eg Spectre).

And of course you want redundancy against losing an entire machine (say the PSU fails and overvolts the 12v rail). But more centralized ad-hoc infrastructure should be easier to grok/manage/backup against this.

My new router is pfsense running on KVM/Debian on an i5-4430. If I start running more Internet-facing services from home, they'll probably be on that box rather than my Libreboot KGPE.

FWIW, Newegg has been clearing out those KGPE boards, occasionally selling them for $80 - https://pcpartpicker.com/product/yvJwrH/asus-motherboard-kgp... (the last one was on flash.newegg.com and so doesn't show up on PCPP).

Non-high-density G34 heatsinks are a bit tough to find these days, but there's a thread where someone details the steps of mounting a 212 Evo. The Raijintek Aidos worked for me just by fashioning the appropriate length of steel bar, drilling four holes (two for the G34 mount, two for securing the existing Raijintek piece), and adding the appropriate screws/springs. The fan mounting on the Aidos does suck though.

The main issue with this approach for me is that Google et. al. clearly have much better physical security than my apartment. In exchange for in-principle privacy improvements and possibly forth amendment protection you take on a huge risk of burglaries, fires, floods, power outages, etc, plus of course the workload of being your own sysadmin.

Also, if you're paranoid, your data is more exposed. You can turn off all your devices with disk encryption when they're out of your control (usually) but if you turn off your NAS while your away from home it's useless. And if it's on, physical access, and therefore your data, is easy to obtain by the moderately motivated.

Google is like a feudal lord: they might own you, but they'll protect you from everyone else weaker than them.

That’s all true. I don’t actually think that anyone is out to get me in particular, I just find unaccountable concentrations of power and authority distasteful (and troubling). So, I try not to encourage them.

I’m interested in self-hosting for autonomy more than security.

That said, the data Google controls will likely outlive Google the legal entity/business-model of today. Including that huge security apparatus. The long-term (say 10 years from now) security of your data is moot if you can’t control it.

Google have been quite good with allowing exports, and promising to delete things (probably more like perma-hash, but I’ll take it). However, they’re increasingly behaving in ways I don’t expect or like. It’s been great, but they no longer seem like good stewards to me.

As for physical security, you risk burglaries, fire, floods, etc, but that's why you have remote backups.

I keep everything except the boot drives on encrypted drives, so that in case of burglary no data is readable. The boot drives hold no data or passwords, only enough to start up and allow SSH logins. It's a small chore to login and manually mount the drives, but IMO worth it.

As for physical access, besides the 40kg German Shepherd Dog roaming my house, the same rules apply to access as from the outside: 2FA, and limited login attempts. I do expose more services on the LAN than i do on the internet, but everything requires authentication.

For personal cloud stuff i use Resilio Sync. It's not dependent on a single machine being powered on, and i have a couple of machines at different physical locations (both _mine_, as in hardware and sysadm tasks) "hosting" the data.

OP here - basically the same, but with a 60kg SwissyDog which actually is pretty useless to protect the house :)

I can also power down system when leaving for a longer period and simply WoL them once connected to the VPN.

Recently I've been playing around with something like this, or really; getting back to something like this. Years ago I was an idealistic graduate student who didn't have any "free" (you-are-the-product) digital service or social media accounts - I used Lavabit (before it got shut down) for email and a couple of cheap VPS providers to run a little blog and IRC bouncer. I had a dumb "burner" phone, a RockBox-based MP3 player, ran Tomato on my router, and ran Debian on most everything else.

I think the main factors that killed it for me, that made me drink the Kool-Aid so to speak, were a combination of getting an Android phone and the shutdown of Lavabit (with all the hassle that incurred - notifying dozens of colleagues, mailing lists, etc of the change). Concurrently with this I had just gotten my first industry gig at a pretty large networking equipment company, on a team with mostly older, mellowed, senior engineers for whom programming was "just a job". I didn't want to seem like a "paranoid weirdo" who had some black-hat alter ego. I sold out.

Since, I've pretty much moved wholesale to Google. I still don't use any other digital services - Google has basically become my one-stop shop, for better or worse. I use Android, Chrome, Gmail, Drive, Music, Books, Search, Maps, Keep, Photos, basically the whole damn suite. It's a beautifully unified and seamless experience. I feel in-general, Google gets just about everything right (I don't use Docs - I still write docs in LaTex, haha). It's quite a 180 from what I had before.

But ultimately, I think this has caused me a lot of cognitive dissonance. I've spent a lot of time thinking about how to "get back" lately, but this is tempered by how much control I've already given up and, well, what is frankly a pretty damn high quality and convenient experience and there are some things (like Maps, and Photos) which I really don't want to give up. I also don't hold any delusions that anything I do is going to be "more secure" from any threat model, really. I guess I just miss all the DIY. The creativity and control.

Anyways, recently I've been building an ARM64-based "mini-homelab" around an Archer AC1750 router running OpenWRT, a stack of three Odroid C2 SBCs, and an Nvidia Jetson TX1 (with a 50K LUT FPGA on the m.2 PCIe slot). I also have an ARM64 VM in the cloud. Once that's all set up, I've been considering how much I can "get back" under my control.

I enjoyed google, and used docs and sheets for budgeting and planning. Then one day it was all gone.

I'd had two google accounts, one created a decade ago on YouTube and later connected to my gmail-google account. I used the same login for both, and could switch between them without problem. I had used my Youtube-persona for most of my google docs work. That was all fine until suddenly I could only access docs from my other gmail-persona. They had without warning or reason changed either the account type or the app permissions for my account type. Not a word of warning, or even a message. There's no reverting it, no one at Google has been able to undo the change or recover my documents.

NEVER AGAIN will I trust any important information to any cloud company.

That was the day I started to take everything off the cloud and access it through a NAS at home (with off-site backup of course). I've never slept better

I guess I'd be curious about how you deal with e-mail, since that was my negative experience with a cloud provider (Lavabit), and I think in-general e-mail is a much harder (and maybe inadvisable) service to self-host.

I always like to read experiences and opinions of people who lean one way or the other with regard to all this stuff, or are at least cognizant of it at all, since it seems 99% of people just use whatever they happen upon first and solves their problem.

I don't know if it's worth mentioning here, but i find one level of indirection enough. I have for the last n years paid for various email hosting providers (outsourcing the hassle of set-up, without going into the side discussion of how much hassle that is or not) but been able to switch effortlessly because i own my own domain. I've merely had to switch MX records when i wanted to switch, my emails are all still on my home computer (offlineimap, but i should really switch to whatever the kids these days are using) and my contacts were none the wiser.

Good question. I'm still stuck with Google for my personal mail, but I use an email client on my PC to download emails and archive them, and periodically do a full download of all my Google content. Not the easiest or most fun solution, but I found that hosting my own email would be a prohibitive amount of work.

While I'm a fan of self-hosting (I do it for email, contacts/calendar sync, RSS, etc), I'd say the problem there was a lack of backups, not the cloud service itself.

People think of backups as a way to protect themselves against hardware failure, but that's a reductive view; their purpose is to protect against systems failure, and a company is a single system. Sure they have their backups, but as you experienced, from the outside this is irrelevant - the system can fail as a whole, and therefore should be treated as a single copy.

Following the 3-2-1 rule, that might mean creating a second Google account with which you share all documents, and some process for backing up to outside Google (even if it's a regular manual use of Takeout).

Using a cloud service that can't be backed up is no different than trusting an hard drive not to fail.

Very valid point. I didn't think to use several google accounts. That might have solved my problem... Of course now I'm very diligently backing up everything anyway

The last thing keeping me on Google is a sheet scripted against calendar for budgeting. I can't find a good open source calendar with an API that will let me query for all events within a date range. Seems like the type of software that should exist in spades. I don't even really need a UI.

"I think the only thing I would like to have automated but have not yet is getting photos from our phones on the server, right now they still go through iCloud."

I use Nextcloud for this, every picture I take is directly synced to my own basement. I can browse them and share them directly from the ui. Nextcloud also offers a nice webui for your email and allows syncing of you calendar and contacts from iPhone or Android. For me it allowed me to switch away from Google for my phone's back-end.

OP here - I actually looked into NextCloud but for most file sharing went for Resilio, so this would be the only use case for NextCloud which seems a bit thin. The Qnap NAS actually provides functionality like this, but I didn't have a chance to take a closer look at it. And I assume I still have to manually open the app once in a while for a sync, which is one step below where I'd actually want to get to.

I don't have to open the app, my pictures sync when I am on my home wifi and on the charger. But you can also set it to immediately upload after a picture was taken.

This is a great read, but I'd like to hear more about the nitty gritty, what server hardware you use, it seems like you have 3 pis and then a bunch of really advanced enterprise-level stuff for other work... I'd love a guide on what you did to get here. The internet really needs something like that.

I'm not the OP, but I have been doing this for some years now. Here's how I went about it.

I got out of social media in early 2012. Deleted FB and Twitter. I do maintain a presence on LinkedIn, but I never post anything there, nor do I login. It's mostly there because employers expect a LI profile. It's just a bland copy of my resume.

I moved my email from Gmail to my own domain from 2013-2015. It took almost two years because I had Gmail since April 2004, a few days after launch. My whole digital life was tied into Gmail, and moving stuff over takes time. I still have that Gmail account, but again, I never log into anything Google, and it simply forwards any email to my real account. I get maybe 1 email a month from Gmail.

I host my email with Fastmail, but I've also selfhosted, and tried O365. It's easy to switch, and I keep an offline backup of everything.

Now, my hardware and software setup.

At home, I have a Synology NAS with ~12 TB of usable storage. This is my primary data store - I have software that automiatically backs up all photos from my Iphone, Time Machine, Veeam, Rysnc, and email backup.

I own my own physical server (Dell R720XD) which is colocated in a datacenter in a different country. This server has 96 GB of RAM, 16 TB of storage, and 12 Xeon cores. It usually runs between 12-25 VMs, some of which are production (personal), hosting my websites, git, various projects, and so on. It's connected to the internet through a gigabit connection with a 10 TB monthly bandwidth cap (I never come close to 1 TB!). This costs me ~$85 a month. I don't do BGP, though I could if I paid some more and put in a proper router and got my own IP range.. as it stands, the 8 public IPs I get are enough for me.

The colo server is connected via an always-on, site-to-site IPSEC VPN to my home LAN (through my EdgeRouter X). I have symmetric gigabit, so it's practically like having the machine in my house.

My NAS backs up to the colo server, and the colo server to the NAS (via rsync). The NAS also backs up to a local USB drive, for quick restores.

All my home data lives in four places

1. The local PC/laptop/phone. 2. The NAS 3. The USB Disk 4. The offsite colo server.

Likewise, all my colo data lives in the same four places.

I can VPN in to the colo server and get connected to my home lan from anywhere in the world. Any sensitive data is encrypted - and anything really sensitive is not stored on a computer.

This setup works very well for me and my family, and I control all my data and am not dependent on a cloud provider. I also have a bunch of other servers running in my home, but they aren't part of my core infrastructure.

The cost?

~$1200 for the R720. ~$1000 for the NAS. and ~$85 a month for colocation costs.

Do you have measures in place to give access to someone trusted in case something happens to you? With a specialized setup like this hosting, for example all family pictures, I’d always be scared of having a low bus factor.

Yes and no.

My spouse can easily access anything family related - pictures, documents, passwords (keepass) from their computer. The NAS and the Colo Server both present as shared drives on the network and they use it almost every day.

The USB drive is plug and play and has copies of anything important.

However, all my git repos, websites, tech projects etc will probably not survive my passing. I don't really care about that.

They will still be backed up and accessible on the NAS, so if someone in my family (I have a few technically adept people in the family) wanted to check it out, my spouse would be able to give them access. Honestly, I don't think anyone would care, but who knows.

All passwords and access details are in KeePass and my spouse uses that everyday so knows how to use it,

My colo server is paid a year in advance every year, so it would chug along for a few months before the provider shut it down and shipped it back to my house (again, my spouse knows how to get it shipped back).

All in all, I think my family would be able to access anything truly valuable without too much trouble, and who cares about the other stuff anyway.

Few more questions:

* How do you backup Fastmail?

* How'd you find your colo provider?

* If it's in a different country, how does the latency affect you?

Ditto on the colo provider question.

Which virtualization do you use on server and how do you handle remote admin and security updates for guest VMs and host?

I use HyperV. The server reboots once a month for security updates.

Guests - it depends on the OS the guest is running. I follow standard best practice for each OS. The server has encrypted drives, and each VM disk is also encrypted.

Remote Admin - SSH, Remote Desktop, Powershell etc. Standard management protocols. Since I have the IPSEC VPN always running, I can use anything.

I'm aware that I could be hacked if someone sufficiently skilled wanted to take me down, but that's true of anything stored online. I've achieved a reasonable level of security and I have control over my data.

Not the one you are responding to but OP.

* I simply use OfflineImap[1] to backup my mailboxes to a share on my NAS.

* I was lucky and did not have to search for too long since my webhoster of choice (Hetzner) also offers colo in the country I'm living.

* I use my setup located in Germany from the US a few times a year and when a friend of mine was in China he used my VPN and a dedicated storage pod as well.

It did not feel like one of us was bothered by the slightly increased latency. But I also do not care about the speed of my terminal rendering characters that much, so maybe I am not sensitive enough to latency to give a useful data point.

[1] http://www.offlineimap.org/

1 - Synology has mail software that can backup any IMAP mailbox

2 - Searched, emailed, asked questions. I ended up with three or four quotes and picked one

3 - It's only a few hundred miles away geographically. Latency is under 30ms. This is 10x my home lan (2-3 ms), but in practice I have no issues with SSH or Remote Desktop. Both feel as fast as having the machine on my LAN.

It feels weird that your setup is seen as unusual. Maintaining a home NAS, self-hosting everything, offline and remote backups: This is how I've done things for years (decades?) Besides a brief stint with Gmail, which I weaned myself off of a few years ago, I'm not tied to any particular online service at all. I'm not dependent on the whims of some company, whose interests do not align with my own, to maintain my access to photos, financial records, music, E-mail, etc. If I lost all my online accounts tomorrow (besides my ISP and a single VPS I still use), life would go on as usual.

What's strange and foreign to me is the world where people rely on other companies for so much of their online lives.

Agreed. That's exactly how I feel as well. I'm only dependent on my own machines.

Do you use Synology iOS app for backup to NAS? Is it limited to photos, or can it also handle media like music or video?

One issue I've found with a similar approach is that many mobile apps are limited to iCloud, Dropbox and Google Driver for state synchronization. Only a few apps (e.g. https://2doapp.com, https://notebooksapp.com, https://www.omnigroup.com/omnifocus/) make the effort to support open standards like WebDAV and CalDAV.

iOS now supports storage providers, so you can use a dedicated application to transfer files between devices via SSH/SMB/etc, but many apps cannot export/import all state via a single backup file.

> Any sensitive data is encrypted

Is this done on clients (E2E) or server? If the server, how do you manage encryption keys?

Re: encryption.

The server is encrypted, as are all the VMs (except ones I don't care about, say running a small web app or whatever).

Most stuff is in KeePass, and the database is then backed up.

The encryption keys are stored in KeePass. That database is my most important thing, and I have many copies of it that stay in sync. For eg, I have a script that emails it to me once a day.

I use an app called PhotoSync on iOS to backup pictures. The synology app can backup music and video (as far as I know), but I prefer PhotoSync.

The USB drive at home is NOT encrypted. If someone steals that, they will have access to my docs and pics, but not to KeePass of course. This is done for ease of access in case something happens to me.

Very interesting. Saved for later - I may need to dig a bit deeper into this at some point. I have moved Gmail to Fastmail but the server and NAS-setup is interesting.

Is it possible to get "safe enough" NAS devices at this time?

OP here - since there seems to be some interest on the details I will try to answer them in as much detail as possible in a longer blog post. I already wrote about the network setup[1]

The server is a Dell Precision T5610 (20 physical cores, 64GB memory, 4 Samsung Pro SSDs and quad gigabit NIC additionally to the internal one) and a QNap TS-531X with Seagate IronWolf drives. I would recommend against using PIs at this point, see other comment[2].

Right now all services are running in docker in a dedicated VM on VMWare Workstation. The server also hosts a full work environment I can RDP and SSH into, depending on what I do, when I am traveling only with my iPad Pro (RDP because my last job involved a lot of Java for which I prefer IntelliJ). At some point I should likely migrate to vSphere, but I had the workstation license already and it was quick to setup.

[1] https://screamingatmyscreen.com/2018/7/building-our-home-and... [2] https://news.ycombinator.com/item?id=17966264

I'm probably beating a dead horse here with Docker Policies available and all, but i personally don't trust Docker in production.

Each Docker image runs it's own stack, and each stack has the potential to contain vulnerabilities. Even with services such as Watchtower ( https://hub.docker.com/r/v2tec/watchtower/ ), you're still not safe. Some images are abandoned for years, others are only updated once their "final" product is updated, meaning you could be hosting a handful of vulnerable services without even knowing it.

I host all my internet facing stuff on FreeBSD in jails, though Linux with LXC would do just as good. I have one stack to update, and once that's updated, everything else is updated as well.

Nothing I host is exposed to the Internet but only available locally. When on the road I VPN back home.

But generally I agree, for Internet facing services I would likely make a few different decisions than in this case.

The amount of lost time spent sysadminning and configuring and securing these things is not a potential risk, it is a real cost.

Getting NSLed or TOSed is a potential risk, not a guarantee. For most people who are doing nothing controversial or interesting and who are just using the cloud to receive service notifications and correspondence from friends (who are also doing nothing controversial or interesting), it isn’t a loss of control.

This is a real trade-off, and neither is right. But don’t pretend that keeping a half dozen complex services up, backed up, and secured isn’t a huge time investment. You can’t get that back.

No, not really. It isn't a huge time investment if you know all this well (let's say you sysadmin for work).

Of course it is not feasible for most people to do this, just like it's not feasible for me personally to rebuild my car's engine, or grow vegetables in my garden. Both of which my neighbor does with great ease.

The point is, I use my skills to make my life better, as everyone does. I enjoy it, and it has the side effect of keeping my data private. For things I'm not good at, I either spend time getting good, or I pay a professional for his time and skills.

My mother uses Google and Apple services to back up her photos and data, and I wouldn't dream of forcing my approach on her.

My spouse asked me to set all this up for them, since they share my feelings about data privacy.

OP here - to me, it is not.

For most services I picked SQLite, so the whole backup is one file / directory. Software can easily be updated or test instances started since all is driven by docker. For the few services that need a DB there is a central one serving all hosts. Backups regularly (on a nightly basis where sufficient) go to the NAS. The VM docker is running on gets regular snapshots. Some initial setup and that's it. The whole setup is as movable and reproducible as I could justify making it over a weekend.

Securing theres is an interesting point. I take the risk of a local attacker. Other than that all I got to keep secure is the VPN, which I would have to do anyway.

I am not trying to make it sound like this is something you spend 10 minutes on and call it a day. I am doing this (running prod services) to various degrees for nearly two decades. If you are just getting into all of this, sure, it will take more time and dedication.

I see your point, but I do not necessarily agree with it. I would argue that even with "nothing controversial or interesting" it still is a loss of control. You do not control your data. You do not control if the service will stay available to you and / or your friends.

Ideally, the experience of the time spent leads to innovation and development of personal sysadmin, improving the process. There's a vast difference between technological choices that make sense for corporate IT versus an individual - eg all this "cloud" stuff aimed at simplifying hundreds of machines doing the exact same thing.

Hopefully we're heading towards a future where any non-tech person can easily setup a "personal cloud" by buying a commodity plug-in device, and have secure remote backups at multiple friends' places with a few clicks. Alas, money to actually fund software development was the real win of the surveillance industry.

But to echo your comment - the problem isn't really the time spent administering your own infrastructure, but rather when circumstances prevent you from spending that (focused) time.

The MVP of this approach would be buying a $60 external USB HD and backing everything up to that on a daily basis. Works great if you "only" have a few TB or less of data.

Here's a bash script + the set up I'm running to do that: https://nickjanetakis.com/blog/automatic-offline-file-backup...

I have a similar home setup, but don’t recommend using Pis for disk-intensive workloads due to SD Card volatility. My HomeKit setup, for instance, is Dockerized and runs on an ODROID board with EMMC storage, and the build server for my ARM containers boots from an SD card but has an USB hard drive.

At this point I would not recommend using PIs for anything long running at all. I have two v3 and one v2 and all of them crashed at some point, one of the v3 was idle during that time.

The one PI I still run as build server operates of a write protected SD card and reboots regularly.

I am not 100% sure why the PIs crashed yet (still have to do the investigation), but if errors like this already show up in a server rack with AC attached I'm a bit skeptical.

The usual suspect when i comes to Pi SD Card corruption is a bad power supply. The Pi will (attempt to) draw more power than it can, resulting in a corrupted SD card.

I've also run a large part of my internal network on RPis, but have since moved to Intel NUC machines, where i replaced a bunch of Pis with 2 NUCs. Storage like you has always been handled on a NAS. Synology in my case.

I don't have need for much computing power, so the NUCs handle that just fine.

See my other comment : https://news.ycombinator.com/item?id=17966507

I'd love to see a YT channel or a website that teaches how to do stuff like this for the non-tinkerer. I'm thinking along the lines of Primitive Techknology's channel where he simply shows the process step-by-step.

Very cool, would love to see more content on this topic. Setup guides of individual services, issues you've run into, more on hardware choices, and definitely more on your smart home setup.

OP here, thanks! I posted a few more details here[1] and will follow up with an in depth blog post covering as much as possible rest.

For the smart home setup I am mostly using Elgato Eve SmartPlugs, their door and window sensors and Homebridge for everything that does not integrate with HomeKit natively. Any light source I cannot directly plug into a power outlet (ceiling mounted e.x.) is at least remote controllable via RF, not IR, and I am currently working on figuring out how to send the control signals through an PI or Arduino with an RF transceiver.

[1] https://news.ycombinator.com/item?id=17966311 [2] https://github.com/nfarina/homebridge

Reply on news.ycombinator.com