The issue goes much deeper than "anyone" can go to college. Their is a huge education disparity between wealthy zip codes and poor zip codes. So, yes you were in an AP English class. How many of your fellow classmates do you think were on free and/or reduced lunch? How many students in the regular English classes were on free/reduced lunch?
The educational disparity is the real root of this problem. You add in that the average salary of a teacher after 5 years of experience is less than nearly any job after 5 years of experience.
This feeds into educational disparity even further as the good teachers want to teach in a wealthier school district as it is a much easier position to be in. More support from teachers, "smarter" kids, ability to get paid more as the taxes from that area can afford better teachers.
Contrast that with poor school districts and you get the exact opposite. You have a variety of kids from troubled homes, you don't know if they had or can afford breakfast, you don't know if they have even a remotely okay home environment to do homework in.
So sure, "anyone" with a pulse can get into college but the real issue is not that. It starts much, much earlier in the education system.
It took me a long time to figure out how fortunate I was to grow up in a wealthier school district, and the fact that I worked my ass off to get where I am now but I'd be lying if because of white privilege + a wealthier zip code I wasn't exposed to more opportunities that made busting my ass pay off more.