The filesystem is a proper database, just not a relational one.
Linus focused heavily on performance when he wrote git; he used the filesystem because, as the main Linux kernel maintainer, he knew that the Linux VFS and filesystems were fast enough for these use cases.
(It's the use cases that have changed; it was not expected back then to have more than a few hundred refs in a single repository.)
Ah, yeah, "you're holding it wrong", though use cases haven't changed, it's closer to the expected common case of expectations turning out wildy wrong (Why would you ever expect people to stop NAMING things at scale???)
But also the core property of the filesystem database has always been low performance for a bunch of tiny things
(As a Windows user, I certainly can't argue that sometimes the filesystem as database has been a performance hit when using git. Though Windows filesystem performance isn't always slow, just performs differently, especially with corporate anti-virus tools involved.)