I've been using Jellyfin for roughly 5 years, most of my library are Blu-ray remuxes (50-100 GB) and I haven't had a single one fail to play. My setup is Apple TV with Infuse connected to a LG C9 with a modern Yamaha receiver.

Transcoding is a whole can of worms I chose to avoid by playing everything directly.

This is almost certainly a you problem. Tons of people play remuxes with Jellyfin.

I would check that your client supports the requisite codecs, and has a sufficient network connection. Or alternatively check that your server has enough horsepower (GPU or CPU) to transcode.

Are you sure the client and the server are powerful enough? Do they support all required codecs? I have issues with 4k movies because my Jellyfin is served by a mac mini, and my client is an old Android tablet that doesn't support hardware decoding, but that is hardly Jellyfin's fault. Though they just updated ffmpeg to the latest version, so it might help.

What's wrong with making a friend an account on your server so they can log in? I've done it, it works fine.