Similar to what somebody else said, there are some reasons it's amoral outside of the cultural rules, at least generally throughout civilized history.
Many older cultural or religious rules had some root in logic (obviously some didn't). In some cases modern understanding makes those rules unnecessary. For example, cooking was not as consistent, and germ theory was basically unknown, so eating pork and becoming sick meant that pigs were unclean and not to be eaten. Obviously we now know that we can eat pork if it's properly cooked.
It's debatable as to the status of many sex rules. Things like diseases can become resistant to treatment, evolve to be more problematic (or create a new problem altogether), etc. Condoms generally protect against most but not all. Then there's the reproduction issue, which pulls in a host of other issues such as who's paying for the kid, abortion, etc. Some of this is covered by contraceptives, but not fully. And perhaps the one not talked about, is how it can influence people similar to gambling or substances. This is also up for public debate on how rules should apply to these things.