A lot of the difficulty came down to a lack of shared concepts. To me, partial derivatives are very natural concepts and I would get a few sentences into describing something in terms of partials before I remembered that I was talking to people who didn't even know what those were. (Ditto for differential equations, vector spaces, orthogonality...). Over the years I've learned to a limited extent how to convey ideas without drawing on terminology which most people aren't familiar with, but that has taken decades of practice.
I absolutely benefited from going to university early -- and I also benefited from staying in school. I only finished high school a year early, but spent 4 years taking high school and university courses concurrently. While I gained very little socially from high school, I had a couple excellent teachers there and learned some useful material in other subjects. I've met people who dropped out of high school to attend university full time who struck me as shockingly ignorant of e.g. history.