The main issue is that, typically, if you're storing data directly in the DOM, you're storing redundant data. For example, if you've got an input that should have a green background if the input is one string, and a purple background if the input is another string, what state should you store here? Fundamentally, there's only one thing that's interesting: the input's value. Anything else — say, the class used to set the background colour — is derived from that value. And therefore we always want our data to flow in one direction: the input value is the source, and any other state or UI updates are based on that.
But in practice, it's usually hard to maintain that strict flow of data. For example, you'll probably have multiple sources that combine in different ways to create different derived data. Or you might have multiple different ways of updating those sources that all need to perform the same updates to the derived data. Or you might have multiple levels of derived data.
It's definitely possible to build applications like this — I started out this way, and still use this for simple projects where the state is fairly easy to manage. But as projects get more complex — and as the state gets more complex — it almost invariably leads to weird glitches and broken states that shouldn't exist. Hence why state management is the core of most modern frameworks — it's the most important problem to solve.
N.B. `useMemo` in React has nothing to do with the DOM — React will never thrash the DOM with unnecessary changes, that's the point of the diffing process. `useMemo` is instead about reducing the number of times that React builds its own VDOM nodes.