If they're operating on data supplied by the attacker, they're potentially a single hop away from executing untrusted code.
Assuming bare metal. In shared hosting / cloud / VMs it is different.
But for servers, the application-level vulnerabilities that are needed in order to get meltdown or spectre attacks to run are already devastating. Take over a game server process and you own the in game currency and the scores and the ability to ban users, and probably user level login as well. And you have your pick of privilege escalation mechanisms already, probably.