Trying to design a perfect system is a pipe dream. There will always be interactions between components that weren't expected. The best we can do is fix the problems as they come and occasionally break some api's to build better systems with hindsight, but backward compatibility is such an essential requirement to customers.
I completely agree, however Windows desktops are in particularly bad shape in this regard as they are the result of continuous evolution from the very first version of DOS with an attempt to keep as much backwards compatibility as possible every step of the way. Even the modern NT-based versions of Windows brought over a lot of bits from previous OSes. They were not a completely "clean" rewrite. For example, the Windows 3.1 program manager was still available on 32-bit versions of Windows XP and to this day you still can't name a file any of the following on Windows for compatibility with DOS special file names: CON, PRN, AUX, NUL, COM0, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, COM¹, COM², COM³, LPT0, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9, LPT¹, LPT², and LPT³. In fact you can't even name a file any of those with an extension, such as CON.TXT.
But how do the special file names cause security issues? Or are they just an example of general cruft?
Mostly an example of cruft, although I do see the potential for this unexpected behavior to cause software to misbehave or crash.
There is a difference between a design flaw and an implementation flaw though. It seems like for PDFs, those are mostly design flaws, e.g. doing potentially malicious silent tasks in the background without the user's knowledge. Although it could be argued that the OS should be the component guarding against those types of attacks instead of the PDF implementation.