I strongly agree with you.
I'm fairly involved in Elixir, so I run a risk of catching some frustration for agreeing with you here.
I get why they are going the direction they are, but I think it comes at increased complexity for beginners (that don't want to use LiveView). For example, Phoenix components (which are usable in static server-rendered controllers) live in phoenix_live_view package.
All of that said, Elixir / Phoenix is the best development experience I've ever had and I have no plans to stop using it.