I had the same issue with Spotify, so I switched the Touch Bar to only showing my customized layout at all times, regardless of which app is backgrounded.
This effectively makes the touch bar a useless feature, but at least I can customize the button mapping and always know the buttons will be where I expect them to be.