To start, I believe that the idea of "working memory" is largely valid. Think of it as the number of distinct ideas you can hold in your head at once, sort of like trying to hold a phone number in your head when you've just heard it for the first time.
The general consensus in psych is that this number for the average person is in the single digits and is relatively static in adulthood till a decline in old age. It's been my observation that people with really incredibly small working memory cannot do algebra. The amount of numbers/ideas held in their head is too large, and multiple students in this group described the experience of attempting an algebra problem as feeling like sand constantly slipping through their fingers.
Many of these students grew up in rich neighborhoods with good parents. They had most advantages you can imagine, seem reasonably intelligent when you talk to them, but Algebra will always be beyond them.