shitty CAPTCHA’s and simple checksums have been ruled to be “effective measures”, so this would probably be too.

However, Section 1201(f) is designed to allow developers leeway in reverse engineering to make "independently created computer programs" talk to other "computer programs". But this usually distinguishes between talking to a binary (good) and reading a media file (bad).

My guess is the encoder is probably legal under 1201(f) because it’s reverse engineering a DMCA covered application to create new media files for it to use. But the decoder would be illegal because it reverse engineers a DMCA covered application to extract copyrighted media.