What mistakes are such IT departments making? What actions are you taking to prevent it?
- Treat most servers and clients as cattle, keeping end-user data on backed up and redundant hardened specific purpose servers in small very secured networks.
- Enable detailed logging in auditd and set it immutable to change conf requires reboot. Send important alerts to multiple SIEM servers.
- Avoid sending "noise" events to syslog/SIEM to prevent alert fatigue.
- Enable spooling and TLS TCP transport in the local syslog.
- Enforce mandatory access controls such as SELinux or AppArmor. Ensure all applications are fully enforced.
- All applications, 3rd party libraries go through strict security review, change control.
- No real time connections to 3rd party networks for dependencies. Everything is local in the datacenter or Cloud account after being code and security reviewed.
- Have a manifest per code deployment of all known checksums of all files. Use an in house tool to periodically verify checksums and alert real time on anything that is unknown.
- Run a tool to verify "Full RELRO", "Stack Canary", NX/Pax, "PIE" for all running daemons. Force teams to recompile anything missing these settings.
- Disable SSH multiplexing on bastion nodes and clients outside of development. Multiplexing empowers logless phishing
- Do not allow direct access to production servers just because there is a VPN. Require a highly logged jump host. If that is too much friction for someone move them elsewhere.
- Hire 3rd party code penetration testers to find "all the things". Fix all the things.
- Hire 3rd party penetration testers to find every real world weakness in the infrastructure. Have them sit with members of every org. Tell employees to volunteer anything they see as a weakness. Fire any C-Level that objects to this. Have buy-in from board members.
- Perform extensive background checks on all FTE candidates and executives. Limit access to contractors by default.
- Perform extensive background checks on all 3rd party vendors. Be ready to explain findings to customers, auditors, regulatory bodies, etc...
- Restore an entire production environment from scratch using only backups and code in a staging environment. Have volunteer customers that signed additional NDAs test the restoration. Do this on a regular schedule and ensure a non-technical manager with the appropriate access can follow the instructions.
- Perform chaos monkey testing killing random nodes, unplugging random things and ensure teams can either self-heal or be quickly remediated by novice engineers.
- Encourage red-team penetration testers to break things in a production-like staging environment. Have executives and board members make it clear there are no reprisals, no consequences for bringing up taboo topics around security due to in-house politics.
- Require a top-down culture of treating everyone equally. This includes treating executives that leave their laptop unlocked and unattended exactly the same as an FTE.
I could keep listing best practices but I think we know most will not do 5% of these checklist items. Alternately stock up on pens, pencils, physical notepads. Have printed out lists of phone numbers and addresses.