Atheists and the non-religious follow the Bible because it does offer useful moral and ethical teachings (along with many other religious and non-religious texts.) and because many modern (primarily American Southern) Christians have adopted a literalist interpretation of the Bible, and it's easier to ridicule such people by pointing out how little coherence and sense the Bible actually has when read from that perspective.

Where atheists, non-Christians and the non-religious disagree with Christians is in the insistence of the latter that the Bible is the only valid source of morality, that it is perfect, inerrant and immutable (in other words, that Biblical morality cannot evolve as societies progress, despite the fact that it does) and that morality presupposes Christian theism (the oft-repeated but utterly false narrative that atheism is nihilism and atheists are incapable of morality.)