“I don’t buy that a lack of long form books are the culprit”
I agree that they might not, in themselves, be a necessary requirement, however: the ability to engage with material, short or long, at a level of deep focus and intentionality is important. And one of the (extremely common, historically) stronger methods of doing this is with long form content, the less passive and more challenging the better.
It touches on the topic of generally transferable— or diffuse, neurologically speaking— skills. It’s what is frustrating when speaking with folks who insist on ideas like “I shouldn’t have to take all these non-STEM courses”. A truly myopic world view that lacks fundamental understanding of human cognition, especially for the subgroup with this sentiment that will readily affirm the inverse: Non-STEM folks should nonetheless have a strong grounding in maths and sciences for the modes of thinking they impart.
Why the difference? It’s a strange, highly mechanistic and modular view of how critical thinking faculties function. As though, even with plenty of exclusivity, there isn’t still enormous overlapping structures that light up in the brain with all tasks that require concentration. The salience network in particular is critical when reading challenging material as well as during stem-related thinking, eg Math. Which, ironically, means the challenging courses involving analytical literature are precisely the courses that, taken seriously, would lay down neural pathways in a persons salience network that would be extremely useful in thinking about challenging math problems with more tools at your disposal, more angles of attack.
It really shouldn’t require much of an intuitive leap to realize that reading and interpreting complex works of literary creativity or other areas of a GenHumanities topics will help impart the ability to think in creative ways. It’s in the task name. Spending 3x16 hours in a class for a semester, roughly the same for work outside the class, 6 or 7 times throughout a 4 year college stretch is a very small cost for value.
I think the foundational failing of education for the past decades falls into all of these gaps in understanding and the inability to engage with the learning process because too few people can even articulate the relevance of highly relevant material.