See the game Trials Fusion for an example of this, I think it worked well for them profits wise?
Enthusiasts are a pretty small segment of any customer base.
You're discounting the value of good design and creator experience.
There might be a decreasing barrier to entry for creating more content for a game, but that doesn't mean everything will be of the same quality.
You sell them your brand and the quality behind that brand. This is arguably what is being sold for any product.
For games in particular, it's also not a new concept. Mods never really competed with expansions since the expansions promised a higher level of quality.
Today, you can look at games like Tabletop Simulator, Minecraft (to a degree), Mario Maker, or anything with Steam Workshop that also has DLC. The existence of usermade content is not really an obstacle for selling (well-made) DLC.
Warcraft 3 Frozen Throne was an amazing hit, partly because it opened up the editor even more for content like DOTA.
However you can think of it this way. The game shipped is a point in space. The possible mods to the game are nearby points in this gamespace. Given tools dictate the work involved to get to that new point in gamespace. The better the tools the larger that isosurface is for some given max amount of work. The problem for game studios is that the things they want to sell might exist within the space that is work path accessible to the mod community.