The other typesetting system is troff https://en.wikipedia.org/wiki/Troff (official: https://troff.org), but it isn't a "tried to build something better than TeX/LaTex" since it predates TeX by a few years. Arguably TeX is an attempt at something better than troff.
The case for troff is pretty strong: https://rkrishnan.org/posts/2016-03-07-how-is-gopl-typeset.h...
But honestly LaTeX is so ingrained in academia that it is probably worth sticking with it. You might even learn to love it.
[1] https://en.wikipedia.org/wiki/Lout_(software)
[2] http://patoline.org/ (seems to be defunct)
[3] http://sile-typesetter.org/
[4] https://docs.racket-lang.org/pollen/
The first two are much less important today than they were in the 80s, because an increasing amount of reading is done on screens that are not fixed width. It's noteworthy that bibtex is also showing its age, and biblatex has not been a completely successful replacement.
Not only does it use Knuth's algorithm for typesetting, but it also includes many features unknown to LaTeX (you can even typeset paragraphs with constant spacing between words if you agree to put your extra spaces between letters or even by hiding them in applying a tiny horizontal zoom to your glyphs!). Support for arbitrary OTF fonts is much easier than with LaTeX also.
I agree that TeX's errors is one of the worst things about it. Strongly suggest you track your changes in git, I found it was often easier to go back to some version that compiled and re-add the changes, rather than try and track down the missing bracket or extra backslash that broke it.
Why do you say that? The first error points you to the bad line on your source code, with a short description of the error. I agree that the output is too cluttered and it's often difficult to find the first error. Many wrappers for latex (for example overleaf) show the errors in color boxes and then they are much easier to spot.
https://ctan.org/pkg/texfot?lang=en
texfot – Filter clutter from the output of a TEX run
The package provides a small Perl script to filter the online output from a TEX run, attempting to show only those messages which probably deserve some change in the source. The TEX invocation itself need not change.
I would highly suggest using it!
You have MathML for math, better defaults like OpenType rather than bitmap fonts, and more emphasis on handling variable screen sizes over single-size paper documents.
[1]: https://typst.app/