I have never understood this take on CLRS because I was basically this archetype - I went back to school for a MS in CS, at a mediocre state school, after a BS in math during which I never took a single programming class and my first algos class was out of CLRS. So I literally taught myself how to program the summer before by going through the Django tutorial and jumped straight into algos. I wasn't some kind of math genius either (or else I wouldn't have jumped ship after the BS). But there's not a single proof in the back that spans more than a page or two so I cannot fathom what people find difficult about digesting it. And the exercises are basically Leetcode problems, well known to everyone by now.
After many years of referring to the book over and over (during prep seasons for interviews) and helping other people through parts of it I think the reality is most CS students just don't understand that there's more to CS than JavaScript and Django and CLRS is a real shock more so than challenge.