-First, thanks for your teaching. It's a tough job. I suspect this particular class will get better, and the fact you're trying to do a great job for these students comes through loud and clear from your post.
-How do you personally feel about the course material? Are you personally interested in it and enthusiastic about it? If not, it'll be hard to get others interested and it's tough to fake it. If you are, are you communicating that passion?
-Have you asked for someone to observe a class or two? Or videotaped a class for your own review? There may be some subtle things that you're not even aware of that's detrimental - extra eyes or seeing yourself teach can be revealing. What feedback is your TA giving you?
-Maybe the audience actually is wrong. I'm a big believer in "no bad students, just bad teaching", too, but I suspect we're dealing with a low N here...it's conceivable you're doing everything well and your first calculus class is just not that into the material for any number of reasons. Although you didn't say, I'm assuming this freshman calc class isn't necessarily filled with math majors.
-Great job on soliciting feedback from students - I assume you did something more than the same old dry "evaluations" we hand out and sometimes just skim over.
-Are students coming to your office hours? If not, why not? In some classes, I've set up appointments with each student at least once a semester, just so I have a chance to know them and their own motivations/plans/strengths, etc. It tends to work well. It's another great time obviously to ask them each what's working and not working in a more personal setting.
Again, thanks for your teaching and your obvious desire to improve things for your students.