The one you built yourself with a kernel stripped down to a TCP stack and a few essentials, a bare userspace and bootloader, and just enough libc to support whatever you're serving.
Why? Minimal attack surface.
Why not? Most people won't spend the time to learn OS/distro building. It's a steep curve, maybe one or two years to master building a minimal viable OS with just a kernel and shell, doing something like Linux From Scratch [1] or similar [2], and then learning the ins and outs of (dpkg-tools/debconf type or similar) source package management way to roll your own bare-bones.
The most secure OS would be one where you have an understanding of all it's security configuration options, have those things configured, and verify that configuration on a regular basis. With Unix-like operating systems you have the advantage of many of these things being controlled via text-based files, so you can version-control these and detect deviations of the configuration.
It also worth mentioning that the most secure OS would be supported by a network architecture that isolates it reasonably well. Firewalled subnets using the concept of Zero Trust are valuable.
On the application side, ensure whatever is running on these servers is minimal. Complexity can mean vulnerability, whether it be from the application itself or it's build dependencies. Minimalism is another strength Unix-like operating systems can deliver.
I don’t know how good they are and have never used any, but there’s tooling for building the ultimate stripped down kernel, unikernels (https://en.wikipedia.org/wiki/Unikernel)
A quick Google gives me https://nanovms.com/, https://github.com/solo-io/unik and https://github.com/rumpkernel/rumprun.
A disadvantage of such a stripped down system is that debugging such a system can be difficult. For example, if there’s only one process, you can’t open a shell to look around on a misbehaving system.
It seems to have all these upsides combined with a domain-based configuration that makes the configuration of what you need, whatever you need, quite easy, at least for the well-traveled paths.
And it's powerful enough that when you learn the language (which is small) and the necessary builtin functions (of which there are many, but there's a huge body of existing work that makes it easy to find examples) you can customize your system to your heart's desire.
The various Nix recommendations also appear to be in the same spirit of essentially starting from scratch.
If the service is part of a larger distributed system, then network isolation (for the hosts it doesn’t need to reach), application level authn/authz (on the peer instances it does need to reach), and monitoring/alerting about the host’s behavior in its interactions with other hosts can all help to protect the larger system if the host is nevertheless compromised.
For “embedded” systems (and her I am including bare-metal single-function production servers), I would choose Yocto. It was designed from the ground up to build custom Linux distributions. Every component can be modified trivially, and many different systems can be built using them same components. There are many security features that integrate seamlessly through 3rd party “layers”: secure boot with full chain of trust, secure file systems, SELinux, and so on. It’s easy to extend the system with new “recipes” and “overlays”.
I am currently working on a IoT project that likely will include several edge servers, which will replace extant cloud services. Each system will run a custom kernel (using hand-crafted configurations) and read-only root filesystem (containing only those packages required for running that service). Aside from better utilizing our compute resources, this move will save tons of bandwidth for our customers.
Best of all, we will no longer need to trust our cloud provider to protect our systems and data. Those vendors have show their willingness to push small players in front of a bus, so I consider it outright foolish for most businesses to rely on them as their primary service provider.
I decided on NixOS.
The defaults start out locked down. The firewall has only port 22 open, so give your root user a decent password, and immediately set up your user with a public/private key, test it works then turn off root login via ssh and password logins.
NixOS also only installs what you configure, and remove what you don't, so if you like utils that are considered standard, like 'which', be sure to include them in your configuration.
With NixOS I feel like I know exactly what's on my system at any given time, and nothing should be there that I didn't put there by declaring I want it there.
I have my system set up to update daily so it gets security fixes immediately.
I also have a reproducible system built on a functional build tool called Nix. I can replicate the system easily. If I want lockfile level reproducibility (like rust or npm), I can use flakes.
Are there other distributions more focused solely on optimizing security? Maybe, but the core NixOS development are extremely security conscious and I trust them more than any other group of core distribution group.
When I am in an advisory role, with my own reputation at stake, I must always recommend the hard choice known for superior long-run results over the easy thing with fast but inferior long-run results.
Doing the hard but superior thing builds competency in doing the hard but superior thing, which makes it easier to do the hard but superior thing over the long run.
Contrariwise, doing the easy but inferior thing builds competency in doing the easy but inferior thing, which makes it easier to get stuck doing the easy but inferior thing over the long run.
Pardon what may seem like hyperbole, but recommending inferior solutions to others because it's easy feels like, to me, malicious incompetence.
Can you run one of the esoteric OSes mentioned here, with a default stance that is "more secure"? Sure. But, when something breaks, can you fix it? Will you be able to maintain it? Will others? If you are building a security _appliance_, like a firewall, I would say 100% run a default secure OS. If you are running a server, pick something you are good at maintaining, and maintain it. It's not perfect, but if you maintain it you'll do pretty well.
But it depends on what you want to do and what your threat model looks like.
As others has written, knowing your system is probably more important as I believe (mis)configuration is one of the more common ways to get hacked. And even the most secure OS won’t help you if you accidentally expose your elastic cluster to the internet.
To wrap, I’d say the best thing you can do to secure yourself is “defence in depth”. Use a locked down OS. Add a firewall (or two if one is a soft one which can be bypassed or disabled accidentally). Don’t bind services to internet interfaces unless you want it. Ensure you have authentication enables. Enable auto-updates for anything exposed.
If you do this you don’t have to worry about picking the most secure OS.
If you want to manage the servers yourself there are lots of options. I would choose Debian or Rocky Linux. FreeBSD is another choice. But you will need to spend lot of time and effort to harden your infrastructure.