Average UX improvements are negative and have been for quite a while. (There's enough noise that it's hard to say exactly how long a while, admittedly.)
Previous version only made you click past the warning once. New version makes you click every time, and there's a noticeable delay before the config settings appear.
Previous version had comfortable spacing, with everything clearly readable without excessive wasted space. New version adds an absurd amount of padding to waste your screen space.
Previous version let you sort by column, so you could easily see all the setting you'd changed. New version removes this feature.
about:config is mostly used by the kind of person who complains about user-hostile policies and declines to accept advertising and spying, so if you want to do these kind of things then making it worse can help drive those users away. Simply deleting it is too obvious, but sabotaging the UX has plausible deniability. If you dumb everything down enough you'll only have dumb users, and dumb users are more easily exploited. I can think of no other reason why somebody put serious time and effort into actively making it worse.
(The old UI is actually still available at chrome://global/content/config.xhtml, but now that it's no longer supported it's easier for them to remove it later.)
You might be partly right. I mean, I've worked in UX for more than a decade and I've never met anyone who did this deliberately. But over-optimizing for the novice experience essentially produces the same result.
A decade ago it was trivially easy to find improvements that benefited all users, but these days platforms are a lot more mature and there are fewer universally beneficial usability improvements like that left. At this point making improvements for one persona often result in regressions for others. It comes down to a business decision of which users matter and how the changes impact the bottom line. And you sort of have to make a call because if your company decides not to optimize for a profitable set of users then your competitors will.
I've never worked on UX for a web browser, but my guess is that they're mostly optimizing for novice users and safety at the expense of the power user experience. I don't see that changing as long as their business model depends on appealing to the largest possible number of users. Of course I wasn't in the room when the team made that call and you may be right that they're engaging in some dark pattern shenanigans to actively sabotage your experience and make you leave but I seriously doubt it's that explicit -- no one's actively trying to drive you away, they just don't particularly care about you. You might be able to find a niche web browser just for expert users that meets your need, but I'm not not sure how they'd sustain the business.
Having said that, there are lots business models in which power users are profitable. I've worked on expert systems (military, finance, ETL, data science) that often have crazy high learning curves for novices because whenever there's a tradeoff they optimize for the efficiency and stability for expert users. Having said that, these systems can be expensive (for example ~24k/yr/seat) so the businesses can afford to focus on optimizing for a small number of expert users.
Edit: Didn't see your last line about chrome://global/content/config.xhtml , now I can get the old behaviour back. Thank you, thank you, thank you.
https://www.mozilla.org/keymaster/gatekeeper/there.is.only.x...
Considering how much things can be achieved via about:config registry in Firefox, then you're probably right. Not sure if entries can be still loaded with userChrome.css and userContent.css - back then I was using these to preconfigure browser for friends. These could be dropped into freshly created profile (via -P switch, without launching browser) and act like sorta "answer files".