Remember "dark matter" is just a placeholder name for a number of observational anomalies. All of which can be accurately explained by introducing a single extra particle that interacts weakly with photons. So what you call "invisible stuff" is not necessarily that outrageous. A lot of new particles have already been proposed for other reasons, and some of them fit the requirements quite well (SUSY, sterile neutrinos etc.).

That said, I believe the main reason that many physicists prefer some kind of dark matter "particle" is that it's extremely difficult to modify general relativity (GR) without breaking the theory. GR provides such accurate predictions that modifying it just the slightest is almost guaranteed to break something somewhere else (e.g. at some other scale), which forces you to introduce nasty (if you ask me) scale dependent effects.

This does not mean that we shouldn't modify GR, it just means that it's very difficult to make it work. And physicists tend to like simple theories that work (for reasons such as https://en.wikipedia.org/wiki/Occam's_razor).