But realistically, you’re never going to stop a motivated app designer who is dead set on making their app an unique snowflake art project rather than a tool that users need to use.
Because with the current status quo, the platform that dominates everyone's mindshare is HTML/JavaScript/CSS. Which has a really rudimentary concept of UI controls, and human interface guidelines that spend 90% of their effort on begging people to manually implement usability features that we used to get for free with native GUI toolkits. And I think that we might need to get away from that mess before it's possible for anyone to have any energy left over for worrying about HCI on the level that we used to in the late '90s and early '00s.
The only solution is heavy moderation. But very few players have market share to force developers to do what they say.
But someone could just as easily respond to today's UIs with "Yes, it's wonderful that every single app looks identical, as if it was all designed by one pretty boring artist with no creativity whatsoever" and that would also be a perfectly valid take.
When I first used MacOS I was surprised about tute consistency to access the settings of every program, even third party, with the same shortcut cmd+,
Media software and game launchers were usually the worst offenders.
Certainly the late 90s was the heyday of desktop consistency on Windows, in the 95/98/ME era, I think driven largely by the conventions Microsoft established in Office. And I believe Mac OS gave pretty good platform-level guidance then too, so things were generally okay with a few exceptions— stuff like media players that have always been more on the fanciful side.
Microsoft reimplemented this stuff from scratch all the time. Not just in MFC itself but Office too.
So, no, it was quite the opposite - it was uncommon to not rely directly on the basic OS widgets. Off the top of my head, the two toolkits that I remember that didn't do that were Borland's OWL (which quickly died out in post-Win16 era, since Delphi/VCL was strictly better), and Qt, which while not using native widgets tried to approximate that look and feel as much as possible.
Even in Java land, their first take - AWT - wrapped native widgets. It wasn't until Swing that they moved on to rendering their own, and it was widely derided as looking inconsistent with other apps as a result of that.
MS Office had its own UI toolkit and routinely invented new UI paradigms that weren't exposed in any Windows API, leaving people who wanted to look native scrambling to reimplement. This was particularly the case for toolbars. MS Office first invented the so-called "coolbar" and then the ribbon. Internet Explorer also rolled its own toolbar styles in ways not supported in the base Windows API e.g. toolbars with large icons and sliding sub-sections. Inventing custom toolbars was practically a sport on Windows; Netscape also did it.
At the time the most popular media players were WinAmp (totally custom and themeable to boot), RealPlayer (custom UI https://andrewnile.co.uk/blog/remembering-realplayer/), Quicktime (custom UI) and Windows Media Player (mostly but not entirely native).
Even the base utilities that came with Windows weren't consistent with each other. It wasn't uncommon in the Win 9x era to find programs still using Win3.1 style file dialogs ... a few are still buried in Windows today!
The problem got worse when you examined the artwork. The stock icon library in Windows was anemic, so dev platforms frequently had to expand the core library with their own. Delphi apps could be easily identified by the distinctive icons in their buttons (https://zarko-gajic.iz.hr/wp/wp-content/uploads/2017/02/delp...).
Restyling window decorations was also very common. Microsoft themselves did it routinely, for example their flagship Encarta encyclopedia app had totally custom widgets and window styling: https://winworldpc.com/product/encarta/1999
To get online most users were running something like CompuServe (custom web-style main UI https://thedayintech.wordpress.com/wp-content/uploads/2013/0...), or AOL (custom UI https://www.reddit.com/r/nostalgia/comments/ehxb1g/the_aol_h...), or MSN (custom UI https://cdn.prod.website-files.com/6179a66d5f9cc70024c61878/...)
Windows apps of this era were much like web apps are today: they shared some common code for things like rendering menus, buttons or widgets in their settings screens, but the main UI users interacted with were almost always custom widgets that were extremely varied between apps. Win32 was nearly impossible to style compared to HTML so this represented a large investment of developer time, but a custom branded UI was believed to be worth nearly any cost. This is something fundamental to how humans work and is pointless to fight, a lesson the web platform fully embraced giving it an advantage over other UI toolkits of the era.
With respect to toolbars / coolbars specifically, one thing to remember was that those weren't kept for Office/IE use only, but rather shipped as reusable components ("common controls" etc), and so other apps could and did pick them up. Indeed, well into late 00s, the common fashion for Windows apps was to try to look like the most recent version of Office wrt menus / toolbars.
Also, I do recall that those apps which tried to look flashy with fancy custom styles etc were often perceived as unprofessional, and quite a few people (myself included) deliberately avoided them where possible - and it wasn't difficult to do, with natively styled alternatives readily available. I distinctly recall my own late-90s Windows desktop, and it was very consistent.
I'm sure people who care could make a consistent desktop, but our memories differ on how popular that was. The theming craze was a 90s thing, so even when apps didn't roll their own brand like MS themselves did, apps often let you apply custom themes to change the look.
In theory it's easier than ever to do that. You could create a browser extension that used user stylesheets to restyle websites to have a consistent look and feel. People make that effort to build ad blockers but not to build consistent looks, so I guess there isn't that much demand.
And a lot of people have to use the applications as supplied, e.g. Slack or Microsoft Teams. Which can be accessed via a web browser, sure, but dedicated apps for these are also nice because they have a dedicated spot in the app switchers.
1. Companies will always want to brand their apps with their particular UI styles.
2. In order to prevent the above, the OS would have to deliberately NOT expose the ability for apps to control their own pixels.
Doing 2 means you are making it impossible to support many application types (photo editors, games, etc.).
NOT doing 2 means that app companies will eventually use the same APIs that the photo editor and game applications use.
Nobody forces you to install and use apps made of a different toolkit (or version of said toolkit) from the one shipped with the desktop.
You can use only Cocoa apps on MacosX, qt6 apps on a kde plasma 6, gnome/gtk4 apps on a gnome3 desktop or whatever is the equivalent in the windows 11 world.