> what we agree on as morally correct behavior being nothing more than our arbitrary beliefs
Interesting. I am also concerned by moral arbitrariness and relativism, but fear it comes from somewhere else:
I fear that religion opens up the door to moral relativism, because you can go and say "don't eat shrimp" (Leviticus 11:10), or "don't eat pork", or "don't eat cows", or what have you, claiming "it says so right here, in this holy book full of true revelations".
That's a trivial example, but the same applies for more serious issues: morality based on (religious) revelation is hard to verify and discuss, and somewhat arbitrary.
But if we empirically agree on one real world, empiricism and evidence, we have at least a common basis and need very few, very basic moral principles (it really boils down to the golden rule, or somewhat more sophisticated versions of that, like the "veil of ignorance" in John Rawl's Theory of Justice.)
So, in purely practical terms, I think this notion of some psychopath running around claiming "there's no God, thus I can do what I want" is a non-issue.
If you furthermore seek a philosophical "Letztbegründung" (i.e. ultimate reason, where "the buck stops"), there are attempts e.g. in discourse ethics ("if we are talking, we are already implicitly subscribing to certain axioms, otherwise we would not need to talk"). But I don't think it'a a big problem.
The problem lies less in the foundation and more in certain topics, I think (think of abortion, contraception, war).