Accessibility is often thought of as just making things accessible to people with disabilities, but it's not just that. Accessibility is mainly about maximizing the reach, not just for people with disabilities but people in any situation.
For example, people who live in South America usually have a lot worse internet connection, and more likely to be on mobile networks, than people in the west. If you're audience is mainly in South America, working on making your application accessible, means making sure your application has low bandwidth usage, can deal with high latency and jitter, and works well for phones.
One only has to look at the definition of the word "Accessibility" to see why you're wrong: "accessibility - the quality of being able to be reached or entered - the quality of being easy to obtain or use - the quality of being easily understood or appreciated".
Not sure where the focus on people with disabilities came from, it's important, but accessibility is much bigger than just that.