I don't know if I would call a math library that uses templates so liberally "high performance". High performance also includes compile time in my opinion.
Yeah. Avoiding templates almost certainly leads to losing run time performance. The compile time is a drop in the bucket.
I get the template hate, they take a while to wrap your head around and can create cryptic bugs. Nonetheless they can be extremely powerful and enable performance and reduced complexity by being a bit complex upfront.
Are there any benchmarks to show it would be noticeably faster to compile with a non-templated design?