As someone who was in a similar position and whose world view seems to be similar, I would say that looking for a career as a programmer is probably a bad choice, but you're actually far better off than you think.
On the bad side, the career of programmer is a bad career to get into in the same way that scribe, typist, and elevator operator are terrible career options. However, if you go to a job interview and can't read, type, or use an elevator, you'll be shown the door. Programming is heading down that same road.
So, there's some good news. You didn't waste your time. You're going to be writing code until the day you die. You'll just never be called a programmer in the same way that you'll never be called a chauffeur when you drive yourself to work.
The next bit of bad news is that you probably do suck at programming. The CS AP test was a cake walk back when I took it and it DID include pointers at the time (during the brief period where they focused on C++). I was a deplorably incompetent programmer when I took the exam and I aced it without any problems.
The corresponding good news is that it largely doesn't matter that you suck as a programmer. The vast majority of people are so terrible at programming that even basic, wipe-after-you-poop competence will make you look like a superstar. To put it more succinctly, have you head of Git? I'm not asking if you've used it - have you heard of it? Congratulations! If you have, then you're in the top ten percent of crappy programmers.
I guess what I'm saying is that you shouldn't "follow your dreams" - don't become a programmer because that career is dying. On the other hand, you're pretty well off, because your dreams are going to chase you down. You'll probably never be a "programmer", but you'll spend your career programming, whether you want to or not.
You should specialize, but not in a "sysadmin" or "network guy" sort of way. These also seem to be assimilating into the basic employment competency list. Instead, ask yourself what you want your programs to DO. After all, if a couple of Silicon Valley VCs came to you and said "We need a programmer to make a database of left handed people and code areal drones to blow them up", I'm guessing that you wouldn't be that thrilled at the opportunity. So ask yourself what you'd like them write instead. If it's the next great ray-tracer for pixar, study physics and optics. If it's the next facebook, read up on sociology. If it's the new mining operation. If it's a better news site, study journalism. If it's an encrypted webmail client, study math. If it's blowing up left handed people, study aeronautical engineering.
And keep studying programming. It's a wonderful little hobby that will payout huge dividends in your career. Just don't try to be a programmer. Or an elevator operator.