(disclosure: my work. https://oldvcr.blogspot.com/2023/11/the-apple-network-server... )
What is interesting is that after A/UX died, some alternatives did spring up, such as Yellowdog Linux (which was NOT a macOS/UNIX hybrid, but rather vanilla Linux).
It is a shame, because if Apple had continued in investing it it, we would have had MacOSX much, much earlier.
With NeXT, Apple got OpenStep which (IMHO) offered a superior programming model than Toolbox/Carbon. They also got a superior display subsystem, networking, and better first party tooling. Those are the things that made what got called "MacOS X". Just a Unix dragging Toolbox around wouldn't have gotten Apple where they are today.
Is win32 fundamentally worse than Carbon or Cocoa? if so it seems to have not meaningfully effected the success of Windows.
Win32 is a bit different than Toolbox/Carbon. For one Microsoft had superior first party development offerings. The best development tools on the Mac were from third parties that were increasingly disinterested in the platform. Windows was also an order of magnitude larger of a market. An ISV suffering through the pain of Win32 had a huge TAM in front of them. They could be profitable by only reaching a small fraction. An ISV targeting the Mac platform had a much harder time breaking even to say nothing of profitability just due to the size and nature of the Mac market.
http://preserve.mactech.com/articles/mactech/Vol.13/13.06/Ju...
It is similar to using Carbon.
Already in the Win16 days, anyone that cared about productivity was using C++ frameworks like OWL, Turbo Pascal, later MFC/VCL, Delphi, VB.
http://www.charlespetzold.com/blog/2014/12/The-Infamous-Wind...
https://learn.microsoft.com/en-us/windows/win32/api/windowsx...
However still painful versus the C++, Delphi, VB alternatives.
What made OpenStep so useful was that it was immediately _available_, but imagine if it was used as a source for tech to be added to the continuing A/UX development, instead of basically being Step One.
I suspect Apple went to AT&T for royalty-free terms and were quoted a price that, in the troubled Spindler/Amelio era, they simply couldn't afford for a niche product (the servers).
I never used A/UX, so I can't comment on its relative quality.
But it took half a decade for Apple to ship OS/X after the palace coup that brought Jobs back and killed off all their indigenous OS development. And what they ended up shipping with the first revision of OS X didn't really feel very "Mac" like, and ended up being this weird fusion of NeXT tech with a sort of pseudo-Mac UX.
I guess I wonder what was wrong with the A/UX tech that stopped them from utilizing that as a foundation for something new.
EDIT, answering my own question: from WP "cooperatively multitasks all Macintosh apps in a single address space by using a token-passing system for their access to the Toolbox."
Not sure if this would apply to "hybrid" apps that used the Toolbox but with Unix syscalls as well, but I'm guessing yes, and that sounds like a serious limitation. Pure (non-GUI) unix applications would be fully multitasking, but GUI apps would have the same kinds of limitations as on System 7 -- poor/no memory protection, cooperative multitasking, bad memory management.
Any OS they released to replace the classic OS, which at that time would have been System 7, needed Macintosh App compatibility, and they wouldn’t have needed Linux or Unix or BSD or OPENSTEP or BeOS if they could have just shut up, clamp down on the mission creep and ship Copland. Apple’s Executive leadership in the mid-90s was unable to execute on that, and none of their side-projects were going anywhere in a sustainable (to Apple) fashion either.
They wanted to do the right thing and synchronize the OS kernel switchover with the PowerPC transition, but chose to pursue a radical software compatibility break alongside the hardware swap.
When that failed for what should have been predictable reasons, they had no choice but to fall back to a pure emulation strategy with no OS architectural changes, and the opportunity to put the MacOS on a new path was lost.
I can see that a large company like Apple wouldn't have gone near it at that point though.
In your original question I think you're forgetting the context of the Unix market of the time. It was the high Unix Wars and A/UX was a minority position of a minority player.
The buzzword of the decade was Object Oriented and anyone not elbows deep in an existing Unix was trying to build an OS around objects. It was entirely unclear if jumping on the Unix bandwagon at that time would have a future. Even with Apple's acquisition of NeXT it was their high level OpenStep that was of most interest rather than the BSD base layer.