The other subs-related annoyances (takes time to appear; transcoding/DS behave differently) appear in web client only and not when playing trough a "proper" player (mpv-shim/kodi/whathaveyou), which also support convenient timing adjustments. Maybe switching clients is a work-around that works for you. jellyfin-mpv-shim would be the smoothest if you just want to continue issuing local playback ("cast") from the web interface.
Some people work around this by making sure they only store supported codecs, either by being selective with media or offline transcoding (which can be instrumented with software like https://github.com/HaveAGitGat/Tdarr).
There's nothing "magic" here. Your options regardless of software are inherently:
1. Make sure to only store and stream supported media (as explained above)
2. Do the decoding and playback in a capable client (like mentioned above you can make it fairly seamlessly with the web client together with jellyfin-mpv-shim)
3. Live transcoding (may require tweaking drivers and codecs on your server; significantly increases server hardware requirements and load)
More info here: https://jellyfin.org/docs/general/clients/codec-support
Android TV (a Shield, in my case) did have bad audio sync problems with several common audio codecs. My bad for trying to cheap out, should have just gotten the AppleTV to begin with. Bites me in the ass nearly every time I try to save money in that way.
jellyfin-mpv-shim, kodi, and the offical apps work well IME.
But, I agree they are overall distracting - even when I can hear a conversation perfectly, I still find myself staring at the subtitles.
Because I like to watch movies in languages I do not understand.
Also, I've always had a tendancy to hear egg corns where most people hear the correct words.
So it's subtitles — especially for shows with poor audio mixes, where people whisper, talk quietly, or with heavy accents.
It is a known issue: [1]
> Every time a new Christopher Nolan movie comes out [...] one complaint which seems almost universal at this point is that the sound mixing in many of his movies has resulted in dialogue that is frequently muffled and sometimes even downright inaudible.
[1] https://www.slashfilm.com/577777/christopher-nolan-sound-mix...
Honestly, I think after some point I have gotten so used to them that they drift in the background and don't distract the way you may feel.
That being said, I understand where you are coming from, I am quite particular about the style of subtitles. I would like them to be white, proportional font, transparent background and black drop shadow/outline. Most other styles I find distracting but that may be because I am used to this particular style. For example I don't like Youtube's default translucent background for the subtitles.
I also have some friends for whom English is a second or third language, despite being very good with the language generally they sometimes have trouble, for instance with strong accents or bad sound mixes, so find subtitles (in English) useful.
Not that it will help, but I have three kids and they ALL do this. I have no idea why they do it, none of them have any issues with their hearing so it makes zero sense.
Personally i find it both annoying and frustrating. I can hear just fine, and having the words on the screen is distracting but they refuse to watch TV without them on?
The show starts playing, they demand the subtitles be turned on...
I dont think watching "Brooklyn Nine-Nine" requires the subtitles so they can absorb information ;)
But the biggest reason is that house acoustics suck. Sitting in a drywall box is not the best thing for intelligibility or sonic appreciation. I used to operate a small music studio out of a house, and I spent more than a trivial amount of money to make the acoustics decent for such work. I daydream about doing similar treatments in my living room, but I don't make my living doing audio work anymore, so it's hard to justify the time and expense. So I turn on subtitles or put on headphones.
If I wear closed over ear headphones and watch something on my PC then it's fine, because I can hear quiet sounds much better that way.