Not so much a "you're holding it wrong" as much more directly "we didn't expect it to be used that way". The Linux Kernel team was using it in a DVCS way with a mailing list as the primary "remote work in progress ref storage" and local refs mostly just local personal branches and tags. The "Hub" model of everyone on a project having access to nearly any and all refs in the project is different from the model of the original git developers. Neither model is "wrong" just one is more unexpected when working on the other.

(As a Windows user, I certainly can't argue that sometimes the filesystem as database has been a performance hit when using git. Though Windows filesystem performance isn't always slow, just performs differently, especially with corporate anti-virus tools involved.)