The Perfect Media Server
perfectmediaserver.com
perfectmediaserver.com
One data point about btrfs, FWIW: I've been using it since 2016, the only "issue" I had is that it forced a failing drive into read-only, while other filesystems didn't detect the upcoming failure, thank checksuming for this.
The main missing feature for me is caching on faster disks, though there are some ways to get that. No mention of LVM nor LUKS either, I guess the understanding is either still a bit superficial (everyone has their own areas of interest), or they didn't want to complexify the setup too much.
For self-hosting, I can recommend https://yunohost.org/ which makes easy some traditionally hard parts of self-hosting, though it is quite opinionated (which isn't always a bad thing). Certainly saved me some time on maintenance, though I do see the limitations.
The holy grail of a "perfect homeserver" for me would be to run everything in GUIX, with a declarative configuration I can locally test before deploying, and rolling back. More packaging efforts are needed before this can become reality. I'd also like the option to have simple distributed storage, trough something like hypercore. This would help for backups, and for overcoming low-bandwidth links.
But maybe it's the Internet that needs to be fixed, with public key routing and content-based addressing, plus "heavy" clients and light, dumb servers instead of pushing everything to the browser.
Btrfs raid1 on 4 drives has worked well for me. For media storage i use mergerfs and snapraid.
Staying away from unstable features: Is RAID 5 & 6 really considered unstable still?
“RAID1” in btrfs means “two copies of each file are stored on two disks” - so it’s closer to RAID10 in mdadm terms. It’s pretty nice as you can do things like having 3 x 1TB drives + 1 x 3TB drive and it Just Works to give 3TB of usable space.
> Staying away from unstable features: Is RAID 5 & 6 really considered unstable still?
Yup :(
Also last time I checked (I’d love to be corrected about this) even RAID1 still isn’t ideal (in particular, when a drive fails in a RAID1 array, you only get one chance to mount it r/w and rebalance your data - after that, the array is read-only, and you need to recover by creating a new array on new disks and copy all the data across)
1. unmount the FS 2. make sure the dead/missing/failing device is disconnected 3. mount using -o degraded option. 4. run btrfs device remove missing <mount point>
It depends on having at least as many good drives as required for mounting the FS (which is 2 for raid1).
https://www.reddit.com/r/btrfs/comments/7os094/removing_a_de...
I think you're referring to this (https://www.spinics.net/lists/linux-btrfs/msg63370.html) bug, which was fixed in kernel >= 4.20 and >= 5.0 (according to https://lore.kernel.org/linux-btrfs/CAJCQCtTRseEwoN4cbsAaE_Y...)
That being said, the bug only causes the filesystem to become permanently degraded if the number of devices fall below the required number of devices needed for the raid type, and IIRC was caused by btrfs being unable to write metadata correctly. If you had a 3 drive raid1 pool, and one of the drives failed, it would happily continue working in degraded state.
Don't know if raid5/6 are still considered unstable, but would be very interested as well.
//edit: shish2k was a bit faster and says nope to raid5/6
But TBH, I am fine with some down time in case of a drive failure on my personal server. I try to maintain up-to-date backups anyway, and that's a low-probability event, especially without many drives.
Liking the view on Jellyfin over Plex. I'm a relatively long time Plex user (switched over from XBMC (now Kodi) to Plex in ~2011), but also immensely dissatisfied with the direction they've taken, focusing on growth to satisfy PE backers over the needs of the user base that enabled them to even qualify for any PE money. It looks to me that their focus is increasingly on "collecting analytics" on their users behaviours (logging everything they watch) and providing ad-supported streaming services, instead of trying to provide the best self hosted media streaming service (as it was when I started using Plex).
> One data point about btrfs, FWIW: I've been using it since 2016, the only "issue" I had is that it forced a failing drive into read-only, while other filesystems didn't detect the upcoming failure, thank checksuming for this.
One more data point about BTRFS, is that it doesn't support parity-stripe based redundancy models (raid5/6, raidz{1..n}, etc), which makes it less interesting for a lot of people.
Another note about ZFS is that it doesn't allow for growing of parity-stripe based volumes (although there's a tweet referenced in TFA that notes there should be a pull request with that feature expected soon). This can make it very expensive for people who don't have the funds to buy disks in groups of 6 at a time (my personal preference for a raidz2 zpool)
Before someone looks at man btrfs-device and gets confused: doesn't reliably support those levels. Support for parity-based redundancy models has been in the code for ages, but that part of the code is not considered mature.
It does however support raid1 on multiple drives, and raid1c3 and raid1c4 for increased redundancy.
With Btrfs raid1, you can pool I.e 4 drives, meaning it will store 2 copies of each file on different devices. It’s not your traditional raid1. With raid1c3 it will store 3 copies of each file, and raid1c4 stores 4 copies.
So while technically not raid5, a 3+ drive Btrfs raid1 pool will provide almost the same redundancy. Think of it more like raid10.
Only if you use deduplication (off by default), in which case the rule of thumb is 1-5GB of RAM per 1TB of disk. Otherwise you can get away with less memory.
The CDDL is a non-issue in most cases, and actually preferable to the GPL as it allows for the code to be integrated into BSD and Solaris-derived systems.
The CDDL is essentially the same license as the GPL except it only applies to the files under the CDDL as opposed to the GPL which imposes its rules to the entire project for every project that links it. Any software under the CDDL is exactly as free as in freedom as GPL code.
CDDL: https://tldrlegal.com/license/common-development-and-distrib...
GPL: https://tldrlegal.com/license/gnu-general-public-license-v2
Short description of the difference in the terms section on wikipedia: https://en.wikipedia.org/wiki/Common_Development_and_Distrib...
>This is a free software license
People are free to dislike ZFS, but it certainly is free software
Does btrfs work on anything except Linux? Apparently ReactOS too creating limited Windows support, but it's not very portable. It's a strange thing to praise btrfs for when comparing to ZFS.
>coming from Oracle,
Fun fact, btrfs comes from Oracle.
Plex seems like they’re in a bind. They need to diversify their user base because their current one is very heavily based on people sharing pirated content (every time I point this out someone says they they only use Plex with Blurays they’ve legally purchased and ripped... and good for you! But you’re not close to the normal user) and I figure they’re in constant fear of legal action shutting them down. At least the streaming, DVR etc functionality allows them to make a plausible claim that people use their product for entirely legal means. I’ve gotten enough value out of my lifetime subscription that I won’t feel cheated if Plex disappears, but it’s good to know solutions like Jellyfin are nearly there when/if it does.
Apparently Infuse[0] supports Jellyfin now. I use it with Plex and have found it to be far and away the best TV media front end. So you might want to take a look at it.
I agree with your take in the first sentence so much unfortunately.
One thing I will not run is btrfs, aka ENOSPCfs. I saw it fail again, and again, and again, and again, AND AGAIN, and again, and again. Then I saw it fail some more. The promised features never materialized, and the drama and surprising behaviors never ceased. All costs, no benefits? I'll leave that on the shelf, a few spots down from reiserfs (which was plenty sketchy BEFORE you know who did you know what).
https://rachelbythebay.com/w/2020/12/11/feedback/
As to file systems in general, I think they covered them very well..
As to Plex, he also mentioned the others:
Jellyfin and Emby are two notable players in this space (playback on any device) but we should also mention Kodi. Kodi is great if your use case is local only as automated media transcoding for the target device isn't something that this software does. But, for me, Plex is still the best overall in this category. It has the slickest UI and the most features. Despite this, the progress being made by Emby and Jellyfin recently is very impressive and these are both projects to watch closely.
AFAIK, it's one of a kind. Flexraid on Windows was the closest I had found (a product I still use on an old install - software and owner disappeared - although I did manage to track down the patent and the developer once, seems to have moved on to other interests/employment)
It's often simpler to just mount /dev/sda if you just connected it and need to do a one-off. But if you want something reliable, for automount for instance, I don't see why anyone would use something else than UUIDs. I even put those on stickers on my external hard drives.
Another tip I like is using noauto,x-systemd.automount to mount on-demand, and unmount after a set inactivity timer: https://www.freedesktop.org/software/systemd/man/systemd.mou...
Some partitions don't have UUIDs, such as LUKS volumes with a detached header or plain dm-crypt volumes. In that case, I use a /by-id/wwn-* path to uniquely identify the partition. Of course, those lines are in /etc/crypttab, not in /etc/fstab.
https://wiki.archlinux.org/index.php/Persistent_block_device...
I am talking about UUIDs, that basically any (Linux at least) filesystem has. Those are a bit like filesystem labels, but generated during filesystem creation. Their format makes collisions virtually impossible. LUKS volumes have one, and partitions inside them have a different one too.
This is my prefered way of identifying partitions, as they are persistent even when you move them around with lvm, dd, or others. I think that's also the default in many distributions.
PARTUUID is also a nice way to identify partitions when you can't rely on a partition UUID for some reason. It works like UUIDs, but instead of being stored inside the partition, that information is part of the GPT (GUID partition table).
If you are more interested in the drives than the partition content, by-id makes sense, though I think the use-case is rather narrow :)
uuid is different but you can access it with /dev/disk/by-uuid
That's pure genius. Gonna steal that idea outright. LOL - I'ma go even one step further in my own personal setup, and encode it into a QR code sticker so I can just snap it right into my clipboard from KDE Connect on mah phone when I need to. ;)
QR codes aren't a bad idea, but clipboard sync doesn't work quite well on wlroots-based compositors in my experience :)
Just make sure to put a bit of ASCII below your QR code.
for example, you dd a disk and by-uuid is ambiguous.
meanwhile by-id still finds the original disk
[0] https://github.com/systemd/systemd/blob/master/rules.d/60-pe...
Alternatively, you can accomplish most of the same with OpenMediaVault just with less polish.
I’ve had a pretty good experience running a basic NAS server for my family. It runs Plex well in a BSD jail and has a GUI that’s usable for everything I’ve needed so far. ZFS support made setting up a zraid pretty easy and it supports easy backups to a number of cloud providers. Of course it’s BSD so no native docker, but from what I’ve seen most people choose to run Docker in a vm instead of on native containers.
I’m thinking of trying out proxmox for my upcoming home server because I want to give GPU pass through a shot, but I’m definitely impressed with TrueNAS and would recommend it.
I also wanted a media server on my desktop, with mixed results... Gerbera and MiniDLNA are both annoying to configure and buggy. If there's something better out there, I didn't find it.
Of course, non free (as in freedom)
Have you tried xpenology?
After my first Synology died I bought a beefier refurbished workstation on workstation4u.de and installed xpenology on it.
It works surprisingly well.
https://sourceforge.net/projects/dsgpl/files/Synology%20NAS%...
There are also threads on various FSF mailing lists calling out Synology for GPL violations.
Of course you don't get to use Synology services like quickconnect, but that's understandable.
It's not free software. You didn't pay for it. you don't have a license to use it.
If synology is violating the gpl that means that they have issues to fix, it doesn't change what xpenology is.
There are other projects that are actually free software like FreeNAS.
That's not "an issue they have to fix" nor is it a grey area; it's the essence of the GPL license terms.
The "issue that they have to fix" is that they haven't provided updated sources.. dunno, haven't checked on it in a while .
I don't have an opinion on whether there is non-GPL software in a base install of Synology software using the XPenology bootloader as I haven't researched it. My opinion is confined to "copy their entire product" is not a determining factor with respect to GPL licensing. (IANAL.)
The xpenology people claim it is an "open source alternative" to synology. It is not. They are lying. They are just hacking at it until it runs unlicensed on non synology hardware.
Is OS X free software? If you hack up an open source bootloader that lets OS X run on non apple hardware does that magically make OS X free and open source?
This is not complicated. If you are using xpenology you are pirating software. If you are relying on xpenology to store important data you are a fucking idiot.
Well, may a lightning strike me if this isn't a textbook example of red herring!
I feel I owe you a response, having written the original comment.
First of all, you are not bringing any proof to the table, except your opinions.
You may be right, you may be not, but mentioning "important data" in the context of "the perfect media server" makes me think that you arguments are not very convincing after all.
What's so important in a media server?
Everyone of us who sets up a media server already owns the entire library, in digital or analog format, of media content that is going to be played from the server, so there is no problem if something gets lost, we can rebuild it, we have the technology, we can make it better than it was. Right? Right!
It would be ironic if we were arguing about how much GPL code there is in Synology DSM to use it to watch or listen to content downloaded illegally, wouldn't it?
Anyway, even though it is illegal (I own hundreds of DVDs, but I am not allowed to rip them and play them from my computer), I do it anyway.
Lets not forget about 09-F9-11-02-9D-74-E3-??-D8-41-56-C5-63-56-88-C0
Having said that.
The Synology record on GPL violations is abysmal (just search for "Synology" on trac.ffmpeg.org or serviio forum or samba.org mailing list archive, if you don't believe me) I think using a bootloader to install the base Synology system (most of DSM is under the GPL, unfortunately for them that's how that license works) _only on x86_ (Synology runs on many other platforms) and then install Jellyfin or Plex (if Jellyfin doesn't work) and never use the Synology proprietary software again is fair use. It's a Linux distribution after all, there is nothing inherently illegal in running it.
I know about FreeNAS, but as a media server is too much work and some things do not work (Jellyfin for example) or are not as simple to set up.
iXsystems and Synology make money from selling HW.
In your example XPenology is a bootloader that lets you install Darwin on non Apple X86 HW or a better example would be iOS jailbreaking, which has been considered "not illegal" and never been tested in court (because Apple knows that losing that battle would be very bad for them).
p.s.: Anyway, using a NAS without setting up proper replication and backups to store "very important data" it's equally idiotic.
p.p.s.: you could ask, so why don't simply install Linux? And you would be right. IMO in the context of "the perfect media server" for people that do not know how to setup a Linux media server, but have a spare Dell Optiplex, it's a viable alternative and the risk are basically zero or, to be completely honest, less than watching TV show and movies illegally.
I have yet to mess with headless containerized Kodi on the Pi.
File systems like ZFS have software RAID which require all drives to bought up front. In order to add more capacity, the whole array must be more or less rebuilt from scratch:
https://raid.wiki.kernel.org/index.php/Growing#Expanding_exi...
Ceph gets it right but is too heavy weight for home use. Btrfs can add new drives to an heterogeneous pool and rebalance data but it still has problems with the parity implementation.
There's gotta be a way to do this...
1. Add more vdevs. For example, you can start with a pair of drives in a mirror. Then when you need more space, you can add another pair of drives in a mirror and add that mirror to your pool. This way lets you smoothly increase your available space over time but you do end up with half your drives being parity while still potentially suffering a data loss if two drives in a mirror break at the same time.
2. Replace all of your drives with larger drives (one at a time so the pool can resilver the data onto the drive). This is nice because you can do it as your old drives fail, but the new space won't be available until all drives have been replaced.
3. There is a feature under development for raidz expansion. (Raidz is like raid5/6). The work on the alpha is up at: https://github.com/openzfs/zfs/pull/8853
Or rather than making your panel dependent on some already-out-of-date-when-shipped embedded computing hardware, attach an external embedded computing device to the HDMI port (e.g. roku, firetv, appletv, shield, etc). and liberate your app experience to a more powerful device.
A Fire TV Lite (2020) starts at $30.
- Old Videos Games via shared folders to local setups of retroarch via Steamlink to my TV. I have literally thousands of retro games available from Atari VCS to PS2 I can play at a moment's notice. Using playnite I've also pulled together all the games from Steam, Gog and Epic into one place which all streams relatively well to my TV via Steamlink.
- Books, Comic books, Papers, and Magazines. I use ubooquity over calibre since it just fits my style and needs better. I have almost 100,000 pieces of readable media organized and available.
- Photos, we travel a lot and I like photography. So I have hundreds of thousands of photos also served up via plex.
I use two computers to do this. One is a cheap file/media server that's running plex and ubooquity. It's attached to two Mediasonic 4-bay probox serving up somewhere around 70TB of raw storage. The other is a higher-end daily-use computer with a decent GPU that can push the steamlink games and emulate old games with filters really well. It has 32GB of RAM but only around 4.5TB of storage -- which is fine since I use the file serve for most everything.
I should probably consolidate both machines down to one higher-end system and offload my desktop activities to a lower-end system but this setup is more something that's grown organically over time and wasn't really "designed".
However, the end effect is, I can plop in front of my TV, and choose from a virtually unlimited collection of media. And when I want to read something, I'm exclusively tablet now on the couch or in bed, and have virtually unlimited amount of reading material available as well.
I'll watch, play, and read something pretty much every day, so this setup has proven itself immensely. It's also allowed me to eliminate rooms full of media, game systems, and other
The only remaining hangup that I'm not happy about is how to best handle audiobooks. Right now I shove them into plex as another audio library, but it's kind of a mess and plex thinks they're all music or something. Tbh, I don't really like how plex serves up audio anyways. I also love it if plex just supported ebooks in the way ubooquity does, then I could unify my servers and also more easily serve reading material outside my home network.
The last remaining media I really want to be able to serve up in someway without converting them all are old chiptunes, amiga mods and other demoscene tracker music. I have a huge collection but haven't yet found a good way to serve them up in a way I like and have them accessible in the ways I have my other media available.