To stretch that line of reasoning, are all religious text and places of workship discriminating against athiests too?
Yes? But most of the times an atheist has used this argument it has been an ad absurdum exercise, exactly meant to show how silly the original premise is.