I certainly couldn't handle it - I didn't finish any of my independent projects until I got out of college. Like you said, I couldn't really give them the concentration they required when I had to worry about classes, homework, activities, eating, friends, parties, etc. Day was just too broken up. And you can't code a startup on an hour here and an hour there.
Same problem occurs after you get a day job, but to a lesser extent. The nice thing about employment is that once you go home at night, your time is yours (or should be), so you don't have any homework or studying that you should be doing. You're still limited to 2-3 hours/day of decent worktime, and that's still not enough to really start a company, but at least you can concentrate decently on it.
I'd recommend that you finish college, get a job at a software company for 1-2 years, and then do the startup. A job will let you see the sorts of problems that people are willing to pay for (they're usually more difficult than you imagine in college), and get you in the habit of writing code all day. And it'll show you what the alternative is, for the rest of your life, if you don't make the startup fly. You might even pick up some technical skills.
If you're really really passionate about computers - enough that you just can't concentrate on class - then I'd suggest dropping out, going to work for someone now, and starting a company a couple years down the line. I have a couple friends that took that route and their careers don't seem to have suffered for it at all; in fact, they seem to be doing better than others their age (namely me ;-)) because they didn't waste 4 years on something they weren't really into. But since you phrase it as "nearly impossible to work on my startup on days that I've been in class" instead of "nearly impossible to work in class on days that I've been working on my startup", I doubt this applies to you.
Do try some smaller open-source projects though, stuff you can do in a weekend. Or work on someone else's open-source codebase. That stuff looks really good on a resume, and you'll pick up some really useful skills.