"I just feel like it's going to look like I am quitting because this project sucks... er... because IS why I am quitting."
Sounds like you answered your own question.
Would you really WANT a reference from these people?
I'm guessing no -- they're legally bound to confirm that you worked there, but your references should be people that you've worked with, trust, and respect. I NEVER list managers that have clusterfucked projects as references, I always choose people who've worked with me and that I respected technically as well as personally. (Some of whom WERE managers, but that's exceedingly rare.)