I wonder if you could make the 'wake up' a bit more advanced.
Run tcpdump or something similar on the Pi, with a filter set for inbound packets to the media server on ports that Plex or Time Machine uses. If it sees those packets, send the WoL magic packet.
This might not work if your switch does proper port isolation - but you could perhaps add port mirroring or something like that.