> I am really confused as to how it ever developed
Upon digging, I actually found it fascinating. As I understand how it happened for Proto-Indo-European languages (with a bit of healthy speculation because for the earlier parts there's not much reliable statistical data to confirm it exactly, and there are always some competing explanations by different linguists):
First, before PIE, the older language had no genders, but instead had cases with a twist similar to what they have in Basque language: instead of our usual nominative+accusative cases it had absolutive+ergative cases. The difference is as follows in two sentences: "cat walks" and "dog eats bone". In modern PIE descendants we have "cat" and "dog" in nominative, and "bone" is in accusative case. In modern Basque and the hypothetic language before PIE the "cat" and the "bone" are in absolutive case, and the "dog" is in ergative.
So basically, there was a word form that meant "something does action upon something else" and another word form that meant "something receives an action from another actor or performs the action by itself".
Then these word forms were reinterpreted as "people or animals that do things" and "inanimate or dumb things". So it became two genders: animate and inanimate. People started assigning new words to these genders (sometimes randomly).
Then the plural form of inanimate gender started meaning collective and abstract things. As an analogy, "cows" would also start meaning a collective "herd", not only multiple cows; "grains" would start meaning "cereal", not only multiple grains of wheat. Over time, people started conjugating the plural inanimate abstract words in a pattern similar to singular animate, thus a third gender appeared - many thousand years later we called it "feminine", the former animate gender we started calling "masculine", and the former inanimate gender we started calling "neuter" (literally means "neither").
After the word gender patterns established over time, new words and new word forms started being distributed among these genders also without strict rules, sometimes by chance, or by analogy with existing words that sounded similarly in that period.
Thus we had three genders of Proto-Indo-European languages that developed over millenia for no particular reason and without a specific goal, just naturally and due to fossilizing old speech elements being reinterpreted a little bit differently by new generations — over many generations over thousands of years.