Routers are optimized to know where to send packets given a destination address, not to know what source addresses are valid given a packet is received.

In some cases, it's simple, one address/subnet per port, would be 'easy' to enforce; this is often the case for normal residential connections and commercial users that didn't bring their own IPs. In other cases, networks are connected to networks and what to send there and what is ok to receive may not be the same and may also be dynamic.