If you sell your decoder or encoder the company will, eventually, kick your butt. Getting money involved raises the stakes.
If you distribute a file that is sold by the company, even if it was transformed by a decoder, the company will, eventually, kick your butt. (A recording or sample of how it sounds post-transformation for demonstration purposes would be reasonable.)
If your software requires files that must be paid for in order to function, then it’s critical that you do not bundle those files with your decoder or encoder. That would be piracy.
How your software was made matters. Did you have access to non-public information about anything involved? It seems like the answer is no, so you might be fine. It sounds like you legally obtained a copy of the file, hardware that runs it, and figured out how they work.
What terms of service did you agree to? While not everything listed may hold up in court (which gets determined BY a court IN a court so don’t think that’s an easy win), the company is surely within their rights to ban you from their online service.
If you want some relevant legal advice, check out the GameCube Dolphin emulator and how the team stays legal. I love learning about how stuff works, so I’ve followed lots of console jailbreaking and reverse engineering news for years. There is a way to do this stuff legally. HN is focused more on finance than hacking. So talk to some hackers (who aren’t trying to break the law).
I saw you ask about contacting the company. If this was a blog, I’d love a followup about what they say and do, but alas I’ll probably never see it. (I guess the worst outcome is that they spend more money on making their proprietary stuff more locked down?) Regardless of what they say, their word is not the law. Know the law, know your rights, and know when it’s safe to stick your neck out and when it’s not. I am obviously not a lawyer.
Oh, and which US state you are in and what they are based in will be relevant.