For references in manually written text, it doesn't matter at all - as opposed to programming language design. E.g. in C the equality x[a] == <star>(x + a) holds because we start at 0.
OTOH if we started at 1, it would become x[a] == <star>(x + a - 1). Mathematically it does not really matter as well, but if you're doing lots of pointer arithmetics in C, causing massive bugs by forgetting a crucial " - 1" might be all too easy. No idea if this was inherited from predecessors or an original design decision. Might also be just a coincidence and the reason to start at [0] is an entirely different one ;-)
(sorry for the <star>, I mean * -- but using multiple of these is interpreted as formatting/italics and I don't know how to avoid this)