You mentioned history: There I saw too little in detail and attention to causality to take the content seriously. Also the history courses spent far too much time on the really old stuff, including 2000+ years ago, and not nearly enough time after 1800. And for the old stuff, they neglected the roles of domestic animals, masonry, military technology, metals, open ocean sailing and trade, the longitude problem, coal, steel, and steam. For after 1900, eventually I learned that from TV, e.g., the series Victory at Sea.
So, I was effectively blocked from anything significant in non-fiction until math and physics in my last two years of college; there I was a double major in math and physics but at the last moment dropped physics as a major and was only a math major.
From grade 5 on, I very much wanted the math and science. Being force fed fictional literature for 14 years was a really big bummer -- I was torqued and still am.
During those bad 14 years, at one point I did push ahead: My last three years in college were in a relatively good school with, actually, a quite good math department. But, to save money, my freshman year in college was in a state school I could walk to. There they wouldn't let me take calculus but forced me to take a math course beneath what I'd already covered in high school. So, I got a respected calculus book, dug in, and learned freshman calculus on my own. Then, in my sophomore year in the better college, I started with their sophomore calculus, using the same text as Harvard, and did well. So, I taught myself freshman calculus and, actually, never took it or got course credit for it.
Bad high school? Not comparatively: It was by a wide margin the best public high school in the city and, really, better than the few, small private schools. MIT came recruiting, and the year before me three guys went to Princeton and ran against each other for President of the Freshman Class.
I see a problem: English, as essentially just fictional literature, was required and got pushed hard for 14 years with nothing else even comparable. I was and remain torqued.
Now, here on HN, fictional literature is very much not the source of "the books we talk about". Good.