It is from the Bible and the old testiment law. A goat were prepared for sacrifice (part of a much more complex ritual that I'm skipping), but then lead out in the wilderness and released instead of killed.