Your conclusion is probably true. Though I didn't go to a "top public school," I followed a similar path and had a similar concern at about the same point in my path through college. I should've chosen otherwise, but instead I chose to finish my economics degree and find a job working for a bank. I can assure you that there are plenty of good jobs working for banks all across the country. Sure, they may not be the elite positions in investment banking on Wall Street, but lets be honest, most people who work in banking and finance don't work on Wall Street. So if you continue to pursue you existing path, you'll find there is plenty of opportunity in banking. Depending on the role and the area, entry-level pay runs between $40-$50k. After a couple years experience you can generally get $60k. It's not the mountains of riches you dream of while in college, but it's a good living.
That said, I've discovered on "the outside" (of college) that I don't really like being the analytics guy, the banker. I'd much rather build products, make things, have skills which enable me to do. Often, work in banking (be it sales or analytics or underwriting or finance or whatever) can feel pretty disconnected from the real world. For instance, I've worked in risk management, and now in credit administration, and while I'm very good at what I do, it doesn't excite me the way building products for the web does.
The key, in your position, is to determine what drives you. Do you like being the analytics/numbers/finance guy who is a whiz with spreadsheets, analytics, finance/accounting, etc? Or deep down do you harbor the desire to make things: software, web apps, mobile apps, etc.
At this point, put aside concerns about making tons of money or being "successful" after graduation. Thinking about those things will just muddy your thinking. And the truth is that there's good money to be made in either arena.
Think about whether you want to analyze or build and I think you'll have the answer to your question.
In my case, I've often wished I would've made the transition no matter the cost, so that I could be working in software development now. Since I didn't I now have to pursue a long-game of developing the skill-set on my own and eventually slowly working my way into the field.
It sounds like you expect all banking positions to pay a ton, be on Wall Street, require 100+ hours a week (and 3 emails/cold-calls a day), etc. This is certainly not the case in any of the non-Wall Street positions I've worked. I've always worked no more than 40 hours a week, received decent benefits, good salary, paid bank holidays, competent co-workers, etc.
I'm sure positions like you describe exist, I just don't think they're indicative of the vast majority of the banking and finance industry. Broaden your perspective and you might like what you find.
If not then yes, commit to change gears. Particularly if you really enjoy the software development stuff (like I always have). Don't focus on whether you do well at it, but instead on whether it is rewarding to you. If it's not rewarding to you (software development related classes) then you'll hate working as a software developer. However, if you find the programming you do in those classes rewarding then definitely make the switch, because the banking/finance world can be a very frustrating place for someone who'd rather be making things with code. You have some semi-technical skill-sets which may or may not get any respect, there can be a lot of egos run rampant, and banks are notoriously slow to adapt and innovate.
That said, if you can find meaning in the banking line of work, it's not a "bad" gig per se, it's just not quite as satisfying personally as programming is.