Yeah this is the most sensible reason against it. NTP is also responsible for some of the biggest traffic amplification DDoS events.
In PTP the problem is in the synchronization protocol itself. A master in the unicast mode is basically a programmable packet generator. It sends sync/announce messages at a rate and duration specified by its slaves, and the address can be spoofed.