Ask HN: What are some fun projects to try out on a spare Linux file server?
I installed Kubuntu on an old PC and am currently using it as a remote dev environment & FTP server. Any other cool use-cases for a spare Linux box?
- reddit.com/r/homelab
- reddit.com/r/homeserver
I'd say the content on /r/homelab is about 33/33/33 distributed between
1. Masturbatory massive rigs at your home that consume thousands of watts of power but look cool and its fun to say you have a 10-node vSAN cluster next to your dehumidifier.
2. People who like to pirate content like its their job - setting up Plex and all the newsgroup/torrent apps to auto-pirate shows and movies. For a 13 year old, that can be cool. For an adult who clearly has the cash to spend on non-essential items and in a world where time is money: I do not get it.
3. People with an old PC running Kubuntu for a remote dev / FTP environment who want to do more cool shit with their gear!
So as long as you enter the homelab reddit knowing this, you will have a better experience there.
I definitely second Pihole. I might suggest experimenting with virtualization: so you can turn that single computer into 5 or 6. Tools like Proxmox or even just installing KVM and managing it with (https://virt-manager.org/). You could Also look into running your own installation of Gitlab (although due to the footprint of the install, I might suggest keeping that isolated to a virtual machine)
It took me about an afternoon or two to set up, was fun to do, and now my favorite TV shows just appear in Plex for me to watch whenever I want to. Half of the time they're not available on Netflix either, so I'd have to figure out what streaming provider they're on, deal with their special websites, not get notifications when a new episode airs, etc. It's just a better experience.
Or show is only available on the Nth $9.99/mo streaming platform, and I already pay for 3 others. Normally I sign up for a trial with a fake email address (getnada is great!), but some require a (valid) credit card, and I don’t have an unlimited supply of those.
Or the show simply isn’t available in your geographic region.
I have no idea what they're talking about though. Installing qBittorrent takes about a minute, adding an RSS feed one minute more; that's it. I think I configured my setup about two years ago, the files just magically appear in whatever directory I asked them to.
How much more time does this consume compared to opening the web browser, going to <whatever streaming site>, and so on? I'd say, about 300% less.
Edit: to whoever is reading this, after pressing the downvote/flag button, please do the following: set up a VPN to any third-world country, go to your favorite streaming provider, and have a gander at the breadth and diversity of their catalog.
I second this, very laid back with good information.
[1] https://old.reddit.com/r/homelab/comments/bshlr9/the_real_co...
Think of it the other way. Instead of taking work home with them to create a hobby, maybe they found some way to get paid for their hobby. Maybe this is fun for them and getting paid for it during the day is just icing on the cake.
I don't really know though. If that's true, I'm jealous of them.
Instead it is just people who buy an bunch of old enterprise gear on eBay. Though every once in a while someone posts something really cool, which is the only reason I stay subbed. Like a little while back someone figured out how to add 10Gb Ethernet to a NUC.
I have a couple of old Dell T320s that I bought for a project last year. I finished the project and powered them off because they make my office too hot. I really should think about selling them.
I am getting ready to to build a cluster out of Raspberry PI 4s as there is some cloud services that I am paying for that I want to host at home. At least those are pretty power efficient.
It's the same reason why we have cars that have hundreds of horsepower when you can't drive above 65 mph on most highways and even accelerating fast can get you a ticket for "exhibition of speed". It's for fun, and a lot of the people on homelab use it to learn for their job.
I'm always going to pick option 1; guess I'll never be an adult :|
What he's talking about is all those people with 160TB+ servers that do nothing but seed torrents of movies and tv shows they may or may not even watch.
(A civil thing, not a criminal offence).
Note: You can digitally "rip" an audio CD as a digital file; technically you are making a backup of the data that forms the audio on the CD. Provided you are only doing this for your personal use (not sharing), and you keep the original (that's an important part) - I can't see how such a law would even be enforceable?
Furthermore - what if you have old software from say, an IBM/370 system - that your business needs? You don't have that system any longer because the pieces all died, and they went to a museum, but you still rely on the software written in COBOL, so you run it on an emulator on a PC...
...obviously, the software is no longer running off of tapes or punchcards or whatever - ie "format shifting" - does this law cover that scenario too? "Sorry, chap, you'll just have to stuff your business because your antique computer died, but them's the breaks, mate?" - my sincere apologies... :)
Seriously, though - it just seems a trifle limiting if that's the case - ultimately preventing the use of backups and shifting of those backups onto future media (can you copy a floppy to your hard drive and run it from there?)...
Here in the United States, backups are allowed, of any media; but I do think there is a technical limitation in that you have to own the original copy, and you have to make the backup yourself. Thus, a copy of a ROM for an emulator would be a-ok, as long as you own the original cartridge, plus the original system it runs on - but you have to pull that ROM image from the cartridge (and any images needed for the emulated system from the original system) yourself - a copy from another source is not "legal".
In practice, nobody does this, though - but it probably could be checked for via serial numbers in the ROMs, if they wanted to enforce it.
The IBM/370 system example seems pretty farfetched. Especially since they said format shifting was a civil thing, not criminal.
In general, there aren't a lot of clear cut laws that state "you can do X, but not Y". Mostly because that's just not how our legal system works. Most of the time, it's more about your intent rather than what you technically did.
The most clear example of this is the differences between first degree murder, second degree murder, and manslaughter. All different crimes with different punishments, but ultimately you technically killed someone.
Another more relevant example is how cybercrimes are treated. Just because a system let you have access to a file does not mean that it was legal access. Otherwise, none of them would be enforceable since it's always a machine granting you access (that you shouldn't have). It's all about your intent.
Usually, if you really are just making your own backups from your own copies, you don't have much to worry about since, at least for now, it's not really enforceable.
That means they downloaded torrenting software, accessed a tracker and knowingly joined a swarm of peers sharing illegally distributed content.
If your convoy "crashes the gate doin' 98," you don't get dispensation just because you happened to be driving a Chartreuse microbus[1].
4. People who upgraded their PC and have an old PC that is currently doing nothing (but could).
Sure I could have bought a NAS, but that would have cost more money and I wouldn't have learned nearly as much. And I wouldn't have been able to transcode streams.
- Syncthing node https://syncthing.org ;
- WireGuard server https://wireguard.com ;
- Pi-hole or just lying DNS Server https://pi-hole.net, https://gitlab.com/moviuro/moviuro.bin/blob/master/lie-to-me ;
- Nextcloud https://nextcloud.com/ ;
- Your own little blog/site with https (LetsEncrypt), nginx, etc.
- My calendar, to-do list (via CalDAV, and using GoodTask), and contacts are all hosted on the instance, including shared family calendars
- It's my "personal Dropbox" but with 12 TB of storage, and using OnlyOffice it's also my personal Google Docs with a web editor
- NextCloud's app syncs a complete copy of all my photos from my phone automatically (in addition to using iCloud Photo Library)
- I have a script that auto-imports all footage from my GoPro when I plug it in and makes it accessible via NextCloud, so I can share raw footage with other people. When I edit video, it's all via my iOS devices and LumaFusion connecting to the SMB server (which, granted, isn't NextCloud but I can preview all the files via the NextCloud interface)
- It's all backed up to the cloud as well
- It's accessible externally (I use Traefik for load balancing and LetsEncrypt for SSL)
SSHFS however works wonders, and doesn't require the VPN at all.
> Running any type of server on the system that is not consistent with personal, residential use. This includes but is not limited to FTP, IRC, SMTP, POP, HTTP, SOCS, SQUID, NTP, DNS or any multi-user forums.
That said, I run a small stack of personal stuff, including a public resume-style site, and I've never heard a peep from them. Because there's nothing commercial I feel like I could make an argument my site is "residential".
I did have my server pwned once though (I came home and it was very clear from the fan intensity that it was mining bitcoin) so I'm not sure I'd recommend it to anyone.
Namecheap offers this when you buy a domain through them.
I have cups installed so that my printer needs only be setup once to work for all my machines. I'm thinking about putting my scanner on it as well (ideally I'd figure out how to use the buttons on the scanner making a copier), but since I never use my scanner this isn't really likely.
As a file server this is the obvious place to put all your shared media, then use kodi https://kodi.tv/ or volumio https://volumio.org/ or some other media center to share your media to all rooms. Setting this up is next on my list of projects, but so far I haven't got it working. Actually I had it working at my last house, but it was too hard to use so we didn't - let this serve as a lesion: make sure not only that it works, but that it is convenient to use.
Using it as a central hub for your home automation is an idea. I've resisted so far because https://www.home-assistant.io/blog/2016/01/19/perfect-home-a... makes a good point: I can't come up with a use for home automation that isn't worse than what I have without it.
My fundamental idea is I have one powerful, reliable, always-on machine. Anything that I want to run all the time should be on it. My other computers are only turned on when I want to do something on them.
Users need to create a new profile for each new computer I buy. Firefox sync sortof works but I shouldn't need to store this data externally if I don't want to - and if I do want to it should still auto setup.
I used to do this at my previous work. It’s frustrating that browser vendors keep caches like that.
FreeNAS has a nice package around getting it all setup. Now that I have a family I don't have time to fight with computers all the time.
I'm a BSD user at heart, from way back. I use both, but BSD (as opposed to GNU) feels right to my fingers.
All of the above are factors.
[1] https://github.com/zelon88/HRCloud2 [2] https://github.com/zelon88/HRConvert2 [3] https://github.com/zelon88/HRScan2
300 Watt(average)/1000 * 24 hr = 7.2 kw/hr per day
7.2 * 30 * $0.14 (roughly tax, generation + distribution in my area) ≈ $30/month to run the machine.
I have a nearly 10 year old HP Microserver with 4 SATA disks and the average usage is less than 50 watts.
AMD's upcoming Ryzen 9 3950X is also rather exciting if it performs as well as promised. 32 threads for peak 105W TDP is really more than even most home lab builders are likely to ever need.
I can understand it being a concern if you are trying to run multiple machines like this (beowulf cluster?) - or if you just don't want to be wasting the power by leaving the machines running an idle...
...but really, at idle without anything happening, I doubt a system even pulls half that. The best way to find out would be to plug the machine into some kind of "kill-o-watt" monitor and check what it uses at idle vs under load.
I would guess that a relatively modern machine, with a basic-level CPU, running headless and using SSDs or laptop hard drives - or just one or two standard desktop drives - likely wouldn't pull more than 150 watts (or could be made to do so).
I idle at 100W +/- 10W, which is ~$10/month. Sure, I could get a VPS for $5/month (with much lower specs), but the massive amount of local storage plus the ability to easily host multiple VMs really outweighs the extra $5/month cost to have everything local.
I run a handful of VMs and docker containers, such as pfSense, emby (similar to plex), HAProxy, samba shares, snapraid+mergerfs for storage, etc. Worth it to me, might not be for others unless they're interested in server administration.
When this dies I'll probably downgrade to something smaller and shoot for <50 watts, but it's not worth it until it actually fails.
That said you can run home servers with WAYYYY less power usage if you prefer. Single board computers are good bets, like the Raspberry pi 4 only use a max of ~15watts. Repurposed laptops have low power usage too.
My little pizza box R210 II PFSense router uses 30 watts (even though the PSU is 250), once I turned all the BIOS power settings down.
If you get something like a R620 from ebay, assuming the US, you can get one that uses less power.
Modern hardware should pull even less.
Anyway, if we're talking about fun, I can't think of anything more exciting than seeing what happens in this scenario.
I imagine the way you expose it makes a difference, too, in what kind of people you attract.
...but if you've never played around with a security camera system like ZoneMinder allows for, it can be an interesting experience. Like I said, complete overkill for my home security camera system - but it would be perfect if you set it up like an appliance, on a beefy system - because it needs a lot of CPU to process frames and such - and you have a lot of cameras to work with.
You can hook up virtually any kind of camera; from a simple USB webcam (heck, you could probably even get a parallel port black and white quickcam to work if you were crazy enough), to a set of cameras on a multi-input video digitizer card, to a full-on professional level PTZ (pan-tilt-zoom) camera (with PTZ control!).
It is fairly complex to set up properly; one could easily create a business around it selling turnkey camera security appliances if one wanted to, I believe - though I don't think I've seen any such business like that (unlike how there are/were tons of companies doing VOIP using custom Asterisk systems).
The whole application is written (IIRC) in Python; I don't recall what it uses for the video processing (whether OpenCV or some other library). It uses multiple processes to handle everything (it isn't a monolithic system), so it is fairly robust. But as I mentioned, the more cameras/video streams you have (and the higher resolution those cameras are), the more CPU you need to process everything. I'm not sure if there is a hard limit, or a way to segregate things across multiple systems (ie - some kind of clustering?); I think you can specify a separate DB server...
Anyhow - it's something different; I had it at one time running on a PII-350 with 64 Mb of RAM years ago - the last machine, IIRC, was a P4-933 with 256 Mb of RAM - and I could process a couple of streams of 640x480 color video from a couple of IP cameras (all I needed for my home system). But that was the limit for that machine. A more modern machine would probably work much better.
In addition, I also host local files in a temporary way so other machines in my home network can pick them up easily with `php -s`.
Running a home server on an old laptop is very useful. If you're worried about power usage, just set the CPU to the lowest MHz like I did.
If you're doing it, please don't forget to look at boinc, because they need all the help they can get with computing power.
"Free" Wifi in the whole world after that as most AP allow DNS requests.
I did exactly the same as stated in the README
t1 IN NS t1ns.mydomain.com. ; note the dot!
t1ns IN A 10.15.213.99
Then I use proxychains with a sock tunnel (ssh -D ...) to run whatever I want through the tunnel.
The idea is you download it, you run `make`, it asks you a few questions, and then you have a working Dockerized deployment configured via Ansible, with versionable configuration.
I'm sure you could do something like this by running VMS on simh and a vlan/pty and use your modem's port routing nat feature to send it telnet traffic.
My go to "spare Linux server" environment has been RasPis for a while though since headless they are pretty cheap, with an actual x86 there are perhaps some interesting PCIe, PCI ,or ISA cards (depends on how old it is) that are fun. Setting it up as a media transfer station to move things off 5.25" floppy or 3.5" floppy or Zip disks or something can be a fun use as well.
It relies on a webserver, MySQL/MariaDB, and PHP, so a great project to try to get something useful running. Version 4 is going to be released soon and brings a lot of nice changes. I've been trying to update the Wiki to make sure content is up to date and isn't missing important information.
Also amazing play:Sub iOS app (https://apps.apple.com/us/app/play-sub-music-streamer/id9553...) supports Ampache API and has quite decent UI. All this allows me to have my music anywhere I go
- ZFS fileshare
- Plex Server
- Time Machine server for Macs
- CUPS server, my printer is old and requires drivers, so now I can print from my phone and computers without drivers
- Run your own IRC bouncer if you're into that kind of thing
- Run a seedbox, just for Linux ISOs of course :)
I had to stop using plex after they insisted on knowing everything I was watching. Mine runs pihole, a print-server, and a fileserver for kodi now.
It's a little different, since my "server" is actually six ODroid XU4's glued together with Docker Swarm and NFS, but it wouldn't be hard to adapt for your thing.
I'd be happy to share that script if you'd like but it's not terribly elaborate.
I was wondering how to make the system physically retain the discs, to be clear that I'm only building a better cache, and the disc still represents a rights physical token. (Not that this is very practical or necessarily necessary, but it's an interesting exercise.)
I was leaning towards seeing how well I could modify the mechanics of an old consumer electronics 400-CD carousel changer, to make it work with whatever Blu-ray drive I found. That would also let me do some other things. A backup plan would be to combine a drive with an additional simple mechanical mechanism, to divert an ejected disc to a stack that is physically retained.
As a note, do not use slim dvd or bluray drives for this; I went through several and found that they would have many read errors for DVDs that play fine in actual players and full-height drives.
I am not aware of a good way of getting the full menu experience on bluray discs. For many discs, stripping the menus saves 1-4 minutes of time when watching the movie though, so I haven't worried about it too much.
[edit]
Also, if your DVDs have any dust on them, clean them with a microfiber cloth and spray for cleaning glasses before inserting. ddrescue will usually complete eventually, but it can literally be days with a dirty or scratched disc.
For everyone else: if you don't care about menus &c. makemkv is great. It's not libre software, but the linux version is perpetually in beta so it's free-as-in-beer (though I purchase a license because it's just so darn useful).
I too actually bought a license, because it's one of my favorite pieces of software that perpetually "just works".
[edit]
I looked at a recent download and a lot more of it delivered as source than I thought (even under a GPL or LGPL in many cases). makemkvcon is shipped as binary only though.
1) I plug in a cheapy USB Blu-ray drive plugged into my laptop (Running ubuntu 18.04).
2) I use the software MakeMKV. MakeMKV is open-source-ish; the license is still proprietary, but you can grab the Linux source and compile it yourself off of their forum.
The nice thing about MakeMKV (and the wonderful Matroska format in general) is that it'll retain all the audio tracks and subtitles, which is especially useful for anime. MakeMKV doesn't do transcoding or anything (as far as I know); it just breaks the DRM and puts it into an MKV file.
3) I upload the file to the NFS share of my server
4) I have a script running in my that listens to a folder for new mkv files. If it sees one, it moves it to a working directory, then does `ffmpeg -i $FILE -c:v libx264 -c:a flac -crf 20 ~/processed/rendered_$FILE`
This runs on repeat in the background, and I sometimes have a queue of a few dozen things at a time in that folder (if I'm ripping TV shows or something). As of right now, if one of the nodes crashes, there's no ability to restart and I have to manually move the file back to the queue folder. I've been working on writing a thing with RabbitMQ to fix that, but it's not done yet.
5) Once it's done processing, I manually move the file to the folder hierarchy for Emby, and I typically delete the raw blu-ray source, since those files can be gigantic.
---------
Handbrake is great, and you can tell it to retain multiple tracks and subtitles and whatnot (so it might actually be totally fine for what you're doing), but it's not nearly as versatile as ffmpeg, and as a result I don't really use it for anything more than a quick one-off conversion.
EDIT: Just realized that you said that you want to retain the menus and whatnot...I think you could get away with using something like ddrescue for that.
Also, I have noticied that for poor DVD transfers crf 20 is insufficient; the result is visibly worse than the original DVD (probably because libx264 is wasting so many bits on preserving the MPEG-2 compression artifacts).
I'm surprised that crf 20 isn't good enough; I admittedly don't own a ton of DVDs nowadays (I bit the bullet and upgraded most things to blu-ray), but on my TV I typically can't tell much any difference after the rip. Granted, none of my DVD transfers have really been "bad"; they're not grainy or gross or anything.
I used to use crf 25 for blu-rays, which worked fine for most things, but in movies that were darker (literally, not tonally), I could see a few compression artifacts, but I don't seem to notice those when I bumped it 20. The ffmpeg documentation itself says that 17 or so should be visually lossless most of the time.
FLAC is also just really well-supported pretty much everywhere nowadays, so it's kind of worth it to me regardless.
ffmpeg -i Les\ Misérables\ \(2012\)/title.mkv -map a:1 -c:a copy copy.mkv
ffmpeg -i Les\ Misérables\ \(2012\)/title.mkv -map a:1 -c:a flac flac.mkv
ls -s *.mkv
1749629 copy.mkv 3506729 flac.mkvI had a Plex server running on a spare machine, but I found I only used it once a week or so (Friday/Saturday nights) and the rest of the time it was just wasting energy.
That said, I also use these things for a whole ton of things; I watch a lot of movies on there, but since I use docker swarm liberally, pretty much any pet project I have gets deployed there. For example, and I forgot to mention this in the previous post, I built my own basic home-security system by using ffmpeg with a bunch of webcams and Raspberry Pi Zero W's, which communicates to a docker container that streams HLS to hard disk.
Of course I could probably get away with turning these servers on at an as-needed level, but I like being able to just spin up a container whenever I want.
For basic learning on somewhat larger datasets (sub-TB), I have found that my 750 TI SC GPU has been very adequate; the only time I maxed it out was while doing "live training" of a tensorflow CNN from images generated from a driving simulator - so I was using part of the GPU for 3D rendering, and part of it for the training. This was for some "self-driving vehicle" experimentation...
But running the model against the same simulation (that is, having the model drive instead of watching me drive and learning) actually used much less GPU; while I wasn't getting stellar frame rates in either case, they were adequate for the purpose.
I'm just mentioning it, because many people assume you have to have a crazy amount of compute power with a GPU for ML tasks right off the bat (and the associated higher costs of such a system), when in reality that may not be the case (depending on what you are doing and how you are doing it), and you can get away a lesser system.
Where I ran into problems with my GPU, though, was in the case of training using larger datasets (that is, non-streaming) - I couldn't fit them all in the GPU's memory (I think the card only has 1 GB), and so I had to do a batch system - and while it worked well, it was a lot slower because of shuffling data to and from the GPU.
But for my learning purposes, it still did a fine job IMHO; of course, if I were doing such tasks for actual analysis or other "professional" (and paying) purposes, a better card would definitely be in order.
A couple years ago & at the last minute, I needed to cover for a sys admin on a new, complicated system that didn't yet have much automation or monitoring. I only got 2 hours of instruction before he needed to disappear for two weeks. We had a status page that showed the status of services, but it wasn't wired to email alerts, PagerDuty, etc.
I really didn't want to keep staring at the status page. So I wrote a simple bash script that curled the status page once per minute. Then the script grep-ed thru the html looking for a line that indicated there was a critical failure. Finally, I did a curl post request to a slack channel that said "the servers are on fire" if there was something to report, and occasionally posted a "the servers are not on fire" if everything was fine. It was a great hack that made life easier while I learned how to properly manage the system, and help with wiring up the monitoring. It was pretty quick to write, too!
Some new tech that hasn't made it on the list yet:
- https://github.com/issmirnov/zap - recursive URL expander, can be run at DNS level. Ie, "n/" -> "https://news.ycombinator.com/" or "f/g/homelab" -> facebook.com/groups/homelab
- netdata
- https://varnish-cache.org/intro/
- AFP/NFS/samba for file sharing
- https://github.com/janeczku/calibre-web - web UI for calibre ebooks
- https://jupyter.org/ for python scripts and web scraping
- https://github.com/cdr/code-server for an IDE in a tab.
- https://www.portainer.io/ if you use docker
- https://yourls.org/ for a private URL shortener in your home
Then use any of the other ideas here, except deploy them through Kubernetes.
- OnlyOffice; this is basically an open-source version of Google Docs. It's rather heavyweight - you will need 2 hosts for this to run and the install uses many varied components, so this is a great opportunity to work with virtualization or containerization.
- EtherCalc; a shared spreadsheet, much easier to setup than OnlyOffice.
- WARCproxy; record your web browsing to a WARC archive and play it back later.
- Install nginx, setup a 64GB VM, install a graphical Linux desktop on it, and use NoVNC and nginx websockets to allow external access to it.
- Play with Yacy, a search engine.
- Seed some Debian torrents and help out the community.
At home I use Mozilla Webthings gateway but running on Lubuntu Server instead of the sugguested Raspberry Pi. I even host a local only voice assistant that leverages Gateway to control device states. https://hacks.mozilla.org/2019/04/introducing-mozilla-webthi...
Note: I reccomend a dedicated access point over hostapd because I had a lot of problems with it circa 3 years ago.
[1] https://www.dell.com/en-us/work/shop/tp-link-tl-wa801nd-300m...
rm -rf /bin
One day while the lab was in session, I was just playing with stuff and tried executing a fork bomb on the server. The moment I executed the fork bomb, all the 120 machines in the lab became unresponsive, including mine. I didnt realize at that time that they were using the same server for code execution as well as auth server. No one knew what happened and the class was dismissed.
An elastic search for whatever you fancy searching