You want the Unofficial LEGO Technic Builder's Guide: https://www.nostarch.com/technicbuilder
which is an excellent reference for how to build steering mechanisms, differentials, transmissions, suspensions... and Sariel usually offers two or three different ways of accomplishing any given goal.
Highly recommended.
I have fond memories of hacking together a 10 dpi photocopier using the light sensor and a few motors.
I thought "here's a good way to prototype building robots and having fun too" - and I could use my old kit with it.
I found that buying that first kit rather steep, but I persisted. Over time I purchased more parts, hit up the Lego-Pitsco-Dacta store to get others (stuff like pneumatic pieces and gears), and a few of the Mindstorms expansion sets...
I sunk several hundred into it all before I came to realization that it wasn't cost effective; today I know it isn't.
If you want to prototype (or outright build) robotic systems, just go with "raw" materials (especially surplus or junkyard pulls). Small scale stuff can be easily bodged together with hot glue, cardboard, and RC servos. At the time, a great controller for it all was the BASIC Stamp, today it would be an Arduino, RasPi, or ESP8266 - or any number of other options that are all low cost.
Still - where Lego (and other building kits) shine is the modularity and standardness of the system. There have been attempts, some fairly successful, to do the same with more robust (mechanically and structurally) parts for kits (makerbeam, vex, actobotics), but they don't all have the traction the Lego enjoys from being a near first-mover, and not requiring tools to put together.
But if you don't mind tools - there are plenty of lower cost options out there (not saying this because people don't know it - plenty do).
That said, Lego Star Wars has been taking over our house, and we now need to buy a bigger house.
The tolerances are far worse than with Lego, but there are many more possibilities to build workng machines.
I'm just reading a book about technic history with Fischertechnik, where all kinds of differentials, steering mechanisms, planetariums (yes, with several planets, ecliptic, and so on) and even several types of mechanical clocks are built.
Perhaps only by those "not in the know" so to speak. For the general consumer, yeah mostly.
But if you're into DIY robotics for any length of time (or perhaps other STEM pursuits), you likely know about it.
That said, Fischertechnik makes Lego look cost effective...
Dirk Fox, Thomas Püttmann ISBN: 978-3864902963
Hell, they even do it at the Lego store. I was in Bellevue WA and they had about half a dozen of one of the Star Wars sets, and the guy there suggested I might want to buy several of them because they were about to stop production...
(Alas, though, my Lego fetish is Architecture, and some of the Creator stuff).
Nothing is stopping you from reusing pieces, they're made to the same quality as always. Newer sets don't have as many "speciality" pieces as they did in the dark ages (2000-2005) and the instructions have NEVER had more than 1-2 models in them.
If you want mechanical stuff, buy the Technic range.
And there is no "artificial scarcity" - they just happen to release new products constantly, and they can't keep every single set in production; so sets are retired after 3-4 years.
Finally, scalpers aren't the only ones to buy from; I find below-RRP sets all the time on Craigslist & similar.
But I don't disagree with your points. I think if you lose control over any hobby, interest or indulgence, you end up in a bad place. I warned about lego because I didn't consider the dark side when I started :)