I occasionally use that feature. But absolutely every single time is by mistake and it has negative effects on my mood. Still love Firefox though.
I pretty much only use it for videos yt-dlp + mpv can't open though.
>I managed to somehow disable the button that appears on the video without totally disabling the feature
Not too hard really. This seems to do it:media.videocontrols.picture-in-picture.enabled -> true
media.videocontrols.picture-in-picture.video-toggle.enabled -> false
media.videocontrols.picture-in-picture.keyboard-controls.enabled -> true