That pause is presumably also present when playing music, but no one notices because each song is its own track. Audiobook tracks split in places where a pause would never occur naturally.
The second-most-annoying trait of Jellyfin in my experience is technically not a bug, but should definitely be fixed. It hardcodes expectations of how to determine what season and episode number to assign to files. And if you overwrite them, it will re-mangle them on scan unless you take precautions to stop it.
As far as Jellyfin is concerned, your files must include a tag of the form S__E__ somewhere in the filename. Being located in a folder called "Season 2" with names like "11 - name of episode" won't work. Substrings like 第2集 won't work. Setting actual metadata tags on the file ( https://wiki.multimedia.cx/index.php/FFmpeg_Metadata ) doesn't work; jellyfin ignores them.
Let me provide a per-folder template for extracting season / episode information. Jellyfin already lets you prefer embedded artwork to whatever it finds on the internet. Why enforce hallucinated episode numbers when explicit ones are clearly present?