"I find it really interresting that such a simple concept as a tagged union, which appeared at least as far back as ALGOL 68, could be completely abandoned during the 90's and 00's where OOP reigned surpreme"
From a modern perspective, it may be difficult to understand just how crushing the OO consensus was in the 1980s and 1990s. It still reverberates in our language design today. OO was good design was OO. If it didn't fit into OO, that was ipso facto proof that it was bad. Close the book. No further evidence required. Sum types? Kinda overlaps inheritance. Bad design. Not OO, therefore bad design. Why is Java a better language than C++? It's more OO, so QED.
OO here, by the way, is specifically that variant that emerged in the form of C++ and Java. Very important to have defined, enforced "private" keywords and deep hierarchies. Smalltalk by this definition is not an OO language. (And also, therefore, bad. Also possibly heretical for trying to steal the term OO... and I mean all the implications of the word "heretical".)
My "Software Engineering" course circa 1999 was basically "How to Design Software via OO". (It has the distinction of being the one course from my entire college career that I'm not sure I agree with a single thing I "learned" in that course. Even at the time I had done enough real work to be suspicious, and from here in 2021 I find the whole thing risible. But I got a 4.0, so....) Formal diagrams with strictly enforced diagram languages like UML, because OO was the future of Good Design and UML was how OO was going to help bring programming to the masses who would only have to draw Object Hierarchies and then {magic goes here that never was worked out} and perfect programs!
The internet hit programmers and programming earliest, and now there's only a remnant of the OO dogma just hanging on by its teeth in school curricula that haven't hardly been changed in 25 years, and it lacks the power to envelop the students the way it used to.
There was of course always a counterculture that didn't listen, which is where you get Smalltalk, Perl, Haskell and its ancestry, Python, etc. But you have to bear in mind that that was a counterculture...
... and you know, even today, Java and C++ are still pretty darned popular and it's not that hard to find people who still only know one of those (or maybe both) and still think OO ≡ good design ≡ OO, and the vast chaotic landscape of languages that don't even have "private" is just kid stuff for people who can't handle Real Design. Which is OO. Because OO is good, and good is OO.