You can't do graphs without numbers, so the history makes sense. Finite graphs are relations over finite sets. You can't do much with finite sets before inventing numbers.
A concept of number is not required to draw or describe graphs. Indeed, if you wanted you could define numbers in terms of graphs. For example, you could start with the combinatorial game Hackenbush and build up numbers from there. https://en.wikipedia.org/wiki/Hackenbush