This is the exact reason I dropped Calc this semester. My math background already sucks and I fucked up early and fell behind fast. Makes more sense to not fail and learn on my own before trying again.
You're a freshman, I take it? If you start giving up on shit as "too hard" now, everything's going to be too hard in the future. Learning only gets harder when you get older. If you need to enroll in a more remedial class first, fine, but don't waste your and your parents' money by avoiding classes that are "too hard" for four years of college. You are what you repeatedly do.