A reasonable game is performance focused and is spending most of the computing power directly on itself (even if perhaps not optimally and not utilizing specific GPU hardware to its full extent) and not in OS routines, so an OS providing a specific "game mode" with only the bare minimum amount of OS the game needed to run can perhaps bring a 5% or 10% performance improvement, but not 50% and definitely not 100 times performance increase.
For software that was written in a performance insensitive way (i.e. not games) you perhaps could achieve a 100 times performance increase by a full rewrite avoiding various overheads. However, you would not really need an OS change for that, the main performance-relevant changes would be in the app itself and you can get almost all of that by running the optimized app on a standard OS - but you would need a rewrite of all the actual software.
The example of this article about UnrealEngine and Vulkan is a good illustration - a game targeting the specific hardware directly could achieve the same performance (and more!), however, they are not going to, nobody is going to rewrite the game because that's expensive, so getting an OS abstraction layer like Vulkan - which inevitably adds some extra performance overhead, not reduces it - is the only reasonable way to go.