That would cross my mind. Most potential employers I've encountered have usually asked for references from my last two jobs. On another occasion I had to undergo a background check which involved contacting my last 15 years worth of employers. So no matter how much I may have disliked an employer I would be unwilling to burn bridges until I knew I would never need to rely on them again.