I don't think this should NOT be done, but I do worry people don't believe source-address spoofing is a thing. Since admin functions usually demand 2-way dataflow in TCP which means a SYN-ACK dance, it's possible the risk is lower here but if it turns out you can send data one-way and effect change in the system, this might not be enough to protect a system.
What do wiser minds say? Is this actually adequate? Isn't it better to ship with some kind of 2FA for admin?