Although I have never failed a work term myself (and so cannot speak from direct experience), some of my friends have. It definitely has a major impact when searching for co-op positions. Since this is your first work term, I think it would be especially hard. You don't have the high-level courses to back up your credentials (although you are probably very bright), so future employers would look at your past co-op job performances. In most of my coop interviews, the emphasis is placed on prior job performance than on marks.

Down the road, if you decide to apply to some top-tier companies, remember that you may be tied for 1st place with another student and the failed work term will put him/her over you. I can't stress the importance of not having a failed work term. The truth is that co-op employers have heard students bullshit a lot, so even if you speak the truth, they will assume that you could not handle the rigors of your first work placement.

If I were in your position, I would try my best for 2 weeks to resolve the situation at work by talking to the appropriate people. If that doesn't work, I would take things into my own hands and work on what interests me at work. A caveat: If your employer gives you stuff to work on that you don't like, but that was clearly on the job description, you should do it.