And why exactly can't that be made part of an existing library?
the C++ linear algebra libraries are basically syntactic sugar for interacting with optimized Fortran libraries like BLAS, LAPACK, etc. (I'm sure I'm overlooking some complexity here, because the C++ libraries, while linking to the same Fortran libraries have very different run times)
Yours interacts with ScaLAPACK and other distributed memory libraries.
Why would it not be possible to simply extend say armadillo or eigen to interact with distributed memory libraries? If you need more syntax, then extend the interface.
The other major difference is that sequential libraries tend to get away with letting users not have to worry about where data resides. This is of fundamental importance in distributed libraries, and, for this reason, it is usually a bad idea to think of simply modifying sequential APIs.
Looking at ScaLAPACK, it's been developed since 1995. I've never touched it, but it's probably many many lines of code (and maybe a few PhD thesis) with all sorts of kinks worked out that will take you decades to iron out yourself. To throw out all that knowledge/work/man-hours and to start from scratch seems like a waste.
1. What granularity to distribute the entries in the matrix is a long and subtle argument which doesn't provide a clear winner for every operation (the current conclusions are different for LU with partial pivoting vs. reduction to tridiagonal form). I would by no means say that the approach used by ScaLAPACK is wrong, but only that it is unnecessarily complex and only one operation purposefully targets the finest granularity case.
2. Again, I appreciate the complement, but I don't think that arguments from authority are valid, nor do I think that one can be the "best". I have a large number of colleagues doing wonderful work, much of which I find extremely impressive.
Also, it would be good to disclose that you're affiliated with the project you were promoting.
With that said, if you were familiar with the source code and APIs of both libraries, I think that you would see a clear benefit. This is supported by the way that Elemental is growing and arguably has more functionality than ScaLAPACK (with the notable exception being a parallel Schur decomposition). Over the past couple of years, the library has primarily been developed to support my research goals, but a large number of research groups are actively using it now.
I think you have unintentionally violated the last provision. Please don't post about things you have not taken the time to investigate at all, which is true by your own admission: "I've never touched [ScaLAPACK]".