To attempt to steelman the argument AIUI: If we replace the EROI of fossil fuels with the EROI of renewables, a much larger portion of the economy will have to be dedicated to generating energy. This will make everything more expensive, and a lot of activity we take for granted today will be too expensive to be maintained, even though it might still be technically feasible. Even the leisure time to invent new and better energy sources will be displaced by people needing more time to chop their own firewood etc[1], and free capital to invest in new inventions will similarly be much harder to come by than when we were all comparatively rich due to the fossil subsidy. There will be knock-on effects in the complexity of society, which is currently able to maintain a high level of complexity and specialization because the high costs of complexity (e.g. longer supply chains) is being subsidized by the low cost of energy.
It's not a terrible argument, though it clearly is making a doomer assumption at every turn. Currently, google tells me that the EROI of crude oil and solar PV are ~20 and ~10 respectively - which does represent a significant tightening of the total wealth available if we contemplated switching everything today. The more optimistic counterpoint, is that the current EROI of fossils is diminishing, and will meet the EROI of renewables at some point, where by this logic the maintainable complexity of society would stabilize. Hopefully that still allows us to have a microchip foundry somewhere, though I haven't seen anyone reliably crunch the numbers one way or another.
[1]pointing out that Nikola Tesla invented a lot of things while employed digging ditches is not part of a proper steelman.