Hacker News Reader

Comment by tombert | Hacker News Reader

My latest project has been to start ripping my entire blu-ray and DVD collection (which is massive), and using Emby to serve them. Since I didn't want my laptop to overheat by compressing the video, I wrote a script to transcode files on my server to a smaller size once the ripping is done.

It's a little different, since my "server" is actually six ODroid XU4's glued together with Docker Swarm and NFS, but it wouldn't be hard to adapt for your thing.

I'd be happy to share that script if you'd like but it's not terribly elaborate.

View on HN

I'd like to do this someday (Blu-ray format is awful, compared to DVD VOBs).

I was wondering how to make the system physically retain the discs, to be clear that I'm only building a better cache, and the disc still represents a rights physical token. (Not that this is very practical or necessarily necessary, but it's an interesting exercise.)

I was leaning towards seeing how well I could modify the mechanics of an old consumer electronics 400-CD carousel changer, to make it work with whatever Blu-ray drive I found. That would also let me do some other things. A backup plan would be to combine a drive with an additional simple mechanical mechanism, to divert an ejected disc to a stack that is physically retained.

What do you use to rip those? I am about to embark on a similar project, planning to use vobcopy so that I can retain all the DVD features (multiple subtitles, multiple soundtracks, menu, etc.), but I suspect there's a better option - I don't want to handbrake as this seems to lose all of those features.

So, my current setup for this is as follows:

1) I plug in a cheapy USB Blu-ray drive plugged into my laptop (Running ubuntu 18.04).

2) I use the software MakeMKV. MakeMKV is open-source-ish; the license is still proprietary, but you can grab the Linux source and compile it yourself off of their forum.

The nice thing about MakeMKV (and the wonderful Matroska format in general) is that it'll retain all the audio tracks and subtitles, which is especially useful for anime. MakeMKV doesn't do transcoding or anything (as far as I know); it just breaks the DRM and puts it into an MKV file.

3) I upload the file to the NFS share of my server

4) I have a script running in my that listens to a folder for new mkv files. If it sees one, it moves it to a working directory, then does `ffmpeg -i $FILE -c:v libx264 -c:a flac -crf 20 ~/processed/rendered_$FILE`

This runs on repeat in the background, and I sometimes have a queue of a few dozen things at a time in that folder (if I'm ripping TV shows or something). As of right now, if one of the nodes crashes, there's no ability to restart and I have to manually move the file back to the queue folder. I've been working on writing a thing with RabbitMQ to fix that, but it's not done yet.

5) Once it's done processing, I manually move the file to the folder hierarchy for Emby, and I typically delete the raw blu-ray source, since those files can be gigantic.

---------

Handbrake is great, and you can tell it to retain multiple tracks and subtitles and whatnot (so it might actually be totally fine for what you're doing), but it's not nearly as versatile as ffmpeg, and as a result I don't really use it for anything more than a quick one-off conversion.

EDIT: Just realized that you said that you want to retain the menus and whatnot...I think you could get away with using something like ddrescue for that.

I have a similar setup and I'm curious what your reason for using "c:a flac" over "c:a copy"

Also, I have noticied that for poor DVD transfers crf 20 is insufficient; the result is visibly worse than the original DVD (probably because libx264 is wasting so many bits on preserving the MPEG-2 compression artifacts).

I use the same script to compress everything, and typically FLAC tends to be a bit more compressed on Blu-ray. I dunno, it seemed like a good idea at the time :)...I should probably do a more-rigorous test to see if it actually decreases the size.

I'm surprised that crf 20 isn't good enough; I admittedly don't own a ton of DVDs nowadays (I bit the bullet and upgraded most things to blu-ray), but on my TV I typically can't tell much any difference after the rip. Granted, none of my DVD transfers have really been "bad"; they're not grainy or gross or anything.

I used to use crf 25 for blu-rays, which worked fine for most things, but in movies that were darker (literally, not tonally), I could see a few compression artifacts, but I don't seem to notice those when I bumped it 20. The ffmpeg documentation itself says that 17 or so should be visually lossless most of the time.

Don't know if it makes a difference, but after checking my history, I was using -crf 21 and bumped it down to -crf 18. My brain is telling me I read somewhere to change cfc in steps of 3, but I'm not sure how reliable that is. So I never tried 20.

I couldn't stop being curious, so I just ripped the blu-ray for "Big Fish", which was 28.9GB raw. After running `ffmpeg -i big_fish.mkv -c:v copy -c:a flac new_big_fish.mkv`, the resulting video was 25.5GB...indicating to me that FLAC compression is worth it, since there's no loss in quality.

FLAC is also just really well-supported pretty much everywhere nowadays, so it's kind of worth it to me regardless.

Your test is possibly invalid; without any -map options, ffmpeg will not copy all of the source streams. I don't have big fish, but Les Miserables(2012) I found that flac was about twice the size of the DTS source, which met my expectations:

    ffmpeg -i Les\ Misérables\ \(2012\)/title.mkv -map a:1 -c:a copy copy.mkv
    ffmpeg -i Les\ Misérables\ \(2012\)/title.mkv -map a:1 -c:a flac flac.mkv
    ls -s *.mkv

    1749629 copy.mkv  3506729 flac.mkv

Fair enough point; can't believe I forgot about that.

As tombert mentioned, ddrescue to generate an iso is probably fine for DVDs. Note that you will need to use something to initialize the DVD or many drives will just refuse to read out encrypted sectors; something like "dvdbackup -I" will work fine.

As a note, do not use slim dvd or bluray drives for this; I went through several and found that they would have many read errors for DVDs that play fine in actual players and full-height drives.

I am not aware of a good way of getting the full menu experience on bluray discs. For many discs, stripping the menus saves 1-4 minutes of time when watching the movie though, so I haven't worried about it too much.

[edit]

Also, if your DVDs have any dust on them, clean them with a microfiber cloth and spray for cleaning glasses before inserting. ddrescue will usually complete eventually, but it can literally be days with a dirty or scratched disc.

For everyone else: if you don't care about menus &c. makemkv is great. It's not libre software, but the linux version is perpetually in beta so it's free-as-in-beer (though I purchase a license because it's just so darn useful).

I think MakeMKV is slightly better than "free as in beer", because you do have the source to audit, you just can't distribute it willy-nilly like with something with a true FLOSS license.

I too actually bought a license, because it's one of my favorite pieces of software that perpetually "just works".

I thought the bulk of makemkv was closed source, with only a porting layer that is built and linked as part of the install process?

[edit]

I looked at a recent download and a lot more of it delivered as source than I thought (even under a GPL or LGPL in many cases). makemkvcon is shipped as binary only though.

So do your machines just idle or do you have a way to put them to sleep and only wake/poweron if someone actually wants to watch a movie?

I had a Plex server running on a spare machine, but I found I only used it once a week or so (Friday/Saturday nights) and the rest of the time it was just wasting energy.

If you only use Plex infrequently it can be a bit of a power waster for a standalone server. But one of the really cool things about Plex is photo backups from your mobile devices. If you'd rather not have them up in a third party, then it's a really great solution. Also Plex has streaming audio options, though it could be better with audiobooks. My storage array for Plex is in the same place as my home back up so it's never really wasted for me since I store all my files in one location and just open them at whatever computer I'm at.

The ODroids peak at ~20W of power each, and idle at a lot less. All six together, at peak will be ~120W, which isn't so bad, and it seems to idle at around ~40W, which is a number I'm ok with, especially since these things live deep in my basement, far away from the AC, meaning that the heat they generate at least isn't being re-cooled.

That said, I also use these things for a whole ton of things; I watch a lot of movies on there, but since I use docker swarm liberally, pretty much any pet project I have gets deployed there. For example, and I forgot to mention this in the previous post, I built my own basic home-security system by using ffmpeg with a bunch of webcams and Raspberry Pi Zero W's, which communicates to a docker container that streams HLS to hard disk.

Of course I could probably get away with turning these servers on at an as-needed level, but I like being able to just spin up a container whenever I want.

Reply on news.ycombinator.com