The problem is that the considerations are never adequate when they don't actually involve people who are directly affected.
I remember one time my mom told me about a meeting she attended hosted by an organization in our state called ‘The Council for the Blind’. I think it was about how and where to get your COVID-19 vaccines.
The presenter was a sighted man on the council. Apparently, although he could see his own notes, he had some trouble getting his PowerPoint presentation to display on the Zoom call. He spent 10 or 15 minutes fiddling with it before an attendee finally butted in and said ‘You know, most of us won't be able to see your slides at all even if you get them working. Why don't you just go ahead and post a link to them after the presentation?’. There are lapses of judgment that are possible even when you have every reason to, abstractly and externally, ‘consider’ the position of someone with a disability you don't have, but that just never occur when you have that disability.
I know it's a silly story, but that kind of stuff happens ALL THE TIME. It's also really common for websites or software to have accessibility features that are so ill-considered, they're practically unusable.
You're right that positions for, e.g., blind software engineers, could be hard to fill for small companies. But you don't have to involve disabled users only at that level to make sure that they're part of your development process. You could have disabled users as testers with much less difficulty, by reaching out to relevant organizations for (perhaps paid) volunteers.