These background agents are needed for various VM recovery scenario. It's not a silent install. Very much needed.
You do lose some functionality, though.
By default, SSM Agent is preinstalled on instances created from the following Amazon Machine Images (AMIs):
Amazon Linux
Amazon Linux 2
Amazon Linux 2 ECS-Optimized Base AMIs
macOS 10.14.x (Mojave) and 10.15.x (Catalina)
Ubuntu Server 16.04, 18.04, and 20.04
Windows Server 2008-2012 R2 AMIs published in November 2016 or later
Windows Server 2016 and 2019
Of course, if the agent's verification of who it's talking to is as good as in the case of Azure, all bets are off.
---
[0] I've just checked this on an Ubuntu EC2 instance. The SSM agent is running, but it doesn't listen on any interface. No custom configuration was done it.