One that I want to give thumbs up in particular is Age of Wonders (https://www.gog.com/game/age_of_wonders). It's a 2D turn-based strategy that was released in late 1999. Even back then it was unusual in explicitly supporting Windows NT, and perhaps that's why it continued to work just fine with 2K, XP etc.
But what's even more interesting is that they made the UI to resize according to the resolution, and resize it does - it supports literally any resolution (I tried it on 4K!), and it will properly resize everything. Of course, the widgets and the fonts are still bitmap, so everything's tiny - but even so, still impressive.
All in all, I'd say it's a good example of Win32 API stability. Windows upheld its part of the bargain, and a game like AoW that used all those APIs (DirectDraw etc) strictly as documented still works. The ones that don't work are generally the ones that made assumptions that were valid then, but no longer are.
Even then... Windows 98 doesn't work under VirtualBox, unless you run the VM with the Legacy paravirtualisation interface, and with the ICH AC97 audio card, which Windows 98 doesn't have drivers for ... nor does it have drivers for VirtualBox' VGA adapter.
Those problems are easily worked around, of course, but one day, either VirtualBox will drop its Legacy emulation code, or SciTech Display Doctor will stop working with VirtualBox' VGA adapter, or the sound won't work anymore, ...
I'm one glitch away from being unable to play a game I really like, and bought as a teenager...
Because of that low-level emulation approach, it's not exactly fast. But modern high-end PCs can just about handle something on the order of Pentium 300 + 3dfx Voodoo, which is adequate for pretty much any game from the 90s.
Ehm... I think you learned to work around these issues so gradually that you don't realize how complicated the mess that you've solved is
Only the Legacy VirtualBox paravirtualisation interface provides all of the BIOS interrupts and glue code that the Windows 98 installer requires. The Sound Blaster 16 audio card will work out of the box in Windows 98, but VirtualBox itself has trouble copying the virtual audio out onto the host, playing only 3 seconds of the audio at a time and skipping a little; eventually it crashes. So, use the ICH AC97 audio card instead, but then you need to find drivers for it.
As for the VGA adapter, that used to work out of the box (once you installed the VirtualBox Guest Additions), but the additions' support for Windows 98 was dropped a long time ago, so that's no longer an option either.
Put simply, you need the Legacy interface to run the installer, SciTech Display Doctor for the VGA card (which is now free), and Realtek AC97 Win9x audio drivers (which you can find very easily).
Then it all just works... for now.
Tbf, also about $30 ebay purchase away from buying an early 2000 era laptop that will just run win98 and the game natively - so not all hope is lost :-)
It's often a much better prospect to see if it's available on GOG with any given fixes or notes on how to make it work.
The OS is backwards compatible; nVidia's / AMD's video drivers and hardware, well, not so much...
XP is the only way to get all the right grass textures.