"That's because no one even knows how to start doing "science" in this direction."

No true, there is an entire field of usability science. It is all still limited to experimental learnings as opposed to theoretical deterministic knowns.

Currently usability testing is only be applied to end users. There is no reason task analysis and the Jakob Neilson's 10 heuristic rules of basic usability cannot be applied to software itself. I apply the science of usability to written software.