Back before the web was prominent, software had to be distributed via physical media, usually floppies and later CDs. This inherently limited the reach and complexity of software - physical media is expensive, especially floppies - so it required either the mailing of software by the creator(s) to users, who would have to pay a fee, or a substantial investment by a publisher, which again incurs a (larger) cost to the end user. Almost any widespread software would require a middleman (publisher) and the end user having to go to a store to purchase it.
Allowing anyone to create and distribute software in a format simple enough to be used easily by anyone has democratized this system. With more programmers who might not practice the craft professionally, and a desire for speedy development, higher-level languages and frameworks sprung up to fill the knowledge gap and enable even easier development.
That's not to say that we should not appreciate older software - as you wrote, the aesthetics ought not belie the challenge and art of software which had to fit into tight spaces, and the constraints required usually forced developers to place function over form. It's a different, not necessarily better or worse, software development paradigm, and older software can be as impressive as or even more so than newer software. But we shouldn't write off the new things either, even if they're worse in areas.
(I don't think all of it looks completely ugly, either: Windows 3.1, for example, exhibits a number of traits of modern flat design - single colors, mostly a lack of shadows, many simple icons and interface elements. Windows 1.0 even used the hamburger menu icon. What's old is new again.)