There are really two questions here: (1) Is 20 too late to start college? (2) Should you switch to a (possibly) better school even if no credits transfer (so you have to start from scratch)? I think there are too many unknowns to answer (2) (and I never saw anyone move from one school to another and not be able to transfer
any credits), so I won't say anything about that part.
As far as (1) goes, I had around 5000 students in my career (give or take). Some were starting college in their 20s (e.g. after working right out of high school, or having served in the military). But I had students starting school in their 30s, 40s, 50s (!), though the oldest tended to be people who were returning for a second degree (very often they wanted to teach, which is just wonderful). The majority completed their degrees. They seemed to get along with their younger peers pretty well, though that depends on personality. (I'm thinking of a couple of guys in their 50s who'd sit in the tutoring room happily studying with students who were young enough to be their grandkids.) It helps to not be too self-conscious ("I'm so much older than my classmates"), and to think of the people around you in terms of commonalities rather than differences (e.g. you're all students).
People who had been away from school for a while would probably have a bit of rust, but it was more than made up for by "life experience" - for instance, knowing what you want, understanding that you need to work at things (and being able to do it), having the ability to self-monitor (which seems to develop on average in the late teens to early 20s).
I'm not sure how much of the above is relevant to OP's situation, but I hope it helps.