Would you how we could hire professionals to investigate this for us? And report it to appropriate groups..?
PS: These are dedicated servers :-/
Basic computer forensics needs a copy of the drive as unaltered as possible so you should start with that before running or installing anything. Basically don't run the server if you want to be able to get anything our of it.
If it's not a user data breach its par the course to reintsall and sweep it under the rug lol.
Next time make sure the server takes snapshots and dumps logs to an external place where they can't be deleted.
Have you ruled out an internal source that decided to use some of your "spare" capacity? Or a previously internal source that might not have had all their privileges revoked?
Is ssh only allowed by public key? (in /etc/ssh/sshd_config => PasswordAuthentication no)
Is Apache or NGINX running on the server?
Is PHP/Ruby/Node/Python running apps?
What ports are open in iptables (iptables -L)
What does /var/log/auth.log say?We searched the whole system for authorized_keys files and found one created in a /var/lib/redis/ of a staging container (with no firewall) on this host. We then came across the redis vulnerability https://kevinchen.co/blog/postmortem-server-compromised/ . A junior dev had spawned this container without help from dev-ops and hence left ports open.
What doesn't make sense to us is how this daemon (yam) was running under a statd username when the container doesn't have such a user, but the host does? Are LXC containers able to run daemons on the host?
This is because usernames don't exist, as far as the kernel's concerned. ps is resolving the process's UID to the corresponding name for the outside context, not the one inside the container.
If your (root) password is weak then I'd not be surprised if that was the source of the infection.
You might see logins via "last", or via the system logs.