>But I find all hell breaks loose when you want to get a slightly different formatting than what's offered. Then suddenly the entire system becomes something you need to fight against.

LaTeX was developed with the goal of freeing the user (a researcher, typically) from wasting time on layout and focusing on content (scientific research, usually).

Plain TeX is much more flexible, but you may argue that it is much lower level (or is it?).

For general typesetting, I can’t recommend ConTeXt enough. Its philosophy is nearer to TeX than LaTeX, and it gives you full control on layout, too. Much smaller package than the full TeX/LaTeX ecosystem. And it’s scriptable with Lua!