The answers at your link explain pretty well what kind of APIs these were and they provide sources, e.g. the several books that have been published under titles like "Undocumented Windows ...". There were no magic APIs, just ordinary functions that covered various features not yet implemented in the publicly available Win32 API.
Keep in mind also that the Windows kernel system calls were never accessible directly, like in Linux or the like, but as an external developer you were at the mercy of the user libraries provided by Microsoft. If the documented user libraries did not cover all the features actually implemented by the operating system, there was no way to use them without reverse engineering.
This is now pretty much ancient history and later such APIs have been documented by Microsoft, e.g. in the "Windows Protocols Specifications", mostly as a consequence of lawsuits and investigations by the European Union and FTC.
However when that happened, around 2000, the damage had already been done and the competition had been eliminated.
The dominance of Microsoft in office suites that has been achieved during the transition from MS-DOS to Windows 95, than Windows 98, has succeeded in good part because of the great increase in the number of computer users during this time.
For newbies, the MS Office suite took very well advantage of the new features enabled by a GUI and MS Office was very easy to use, while the Windows versions of the older MS-DOS programs, like Lotus 1-2-3 and WordPerfect were much less polished than MS Office, while simultaneously losing their previous keyboard-based user interface, which for experienced users had been much faster at accomplishing any task than any Microsoft GUI.
So the MS Office GUI alternatives were neither easier to use nor faster, therefore not even their former users had any reason to stick with them, while all the newbies were naturally attracted to MS Office.
Moreover, starting with Windows 95, Microsoft has convinced somehow all the hardware peripheral add-on-cards to stop documenting their products, but instead of that deliver them together with only Windows device drivers.
Since that time, the non-Microsoft operating systems had to rely mostly on reverse-engineering for being able to run on modern hardware. Many years later, the situation has improved somewhat, but only for certain classes of peripherals, like network interface cards, where the vendors typically also provide Linux drivers now.
This has hit also the old MS-DOS programs like Lotus 1-2-3, which had tons of drivers for various kinds of now obsolete Super-VGA video cards, so on new computers with Windows 95 or later they had to run in some less performant video mode that was supported directly or through emulation on modern graphic cards, unlike the Windows programs that could take full advantage of the hardware. This has contributed to the obsolescence of the better MS-DOS programs, because even if Windows 95 could run them in Virtual 8086 mode they did not work as well as on the older computers for which the MS-DOS programs had drivers that took advantage of enhanced features.