We are looking at taking our .NET6 monolith and "inverting" it so the entry point is our logical HTTP resources instead of the various DI monstrosities that emerge from main.
The reasoning for us is that we don't want ownership over all of the horrible things we could get wrong. If a <TLS1.2 connection some how gets established and something naughty happens because we forgot to update some startup.cs boilerplate, it's currently our pain today. Like legally. If we are just eating pre-approved/filtered/etc HTTP requests from inside our hermetically-sealed function environment, we get to cast a lot more blame than we otherwise would be able to.
I really don't get why you'd go to serverless and then shit it up with a lot of extra complexity. The whole point in my view is to inherit the compliance packages of your hyperscalar by way of directly using their native primitives in as standard a way as possible. This means you get to 'sail' through audits and engage up-market clients that are typically out of reach for everyone else.
> Heroku on an allocated server with fixed overhead at all
I don't think you're familiar with Heroku then? I'm not talking about dedicated or managed servers. That's exactly what their 'dynos' platform has been doing since ~2008, though less granular: https://www.heroku.com/dynos/scaling
"All you have to do is handle the routing inside of the lambda using regular ASP.NET Core" also applies to both fly.io and Heroku. That's what I'm getting at. If you don't have a collection of independent cloud functions to execute, and you are spinning up a little monolith for every request, the platform seems like overkill.
If you're using a managed Kubernetes (in GCP, or AWS, or Azure, etc) then the owner of the platform presumably has all this stuff going on somewhere (whether they expose it to you or not), and so is solving all these problems for you. But if you're not using managed Kubernetes — i.e. if you're ever typing `k3s deploy` or whatever equivalent — then you'd better realize that you're signing up to either also deploy and manage your own bare-metal OpenStack deployment for your k8s cluster to plug into; or to tediously glue together a grab-bag of services from your hosting provider + third-parties to achieve the same effects.
(And this is why Google's "Anthos" is an interesting offering to many enterprises: rather than running your own on-prem k8s cluster implying the need to own+manage on-prem supporting infra, you can instead have a GCP project that has all the managed supporting infra, and maybe some hosted k8s clusters; and then your on-prem k8s nodes can just think of themselves as part of that GCP project, relying on the same cloud infra that the hosted clusters are relying on — while still being on-prem nodes, and so still having data security guarantees, low-latency links to your other on-prem systems, etc.)
My opinion is as follows:
Installing kubernetes with kubeadm is easy. You can do it in few hours if you're fluent with Linux.
Kubernetes storage is where it starts to be hard. We were lucky to have OpenStack hosting which allows to use network block storage (ceph) without any additional work. Using kubernetes without any storage might severely limits apps you can run there, although it's possible. Using local storage is an option but also with many "but". Rolling out ceph: I've heard it's hard.
Load balancing: again I was lucky to have OpenStack load balancer. Otherwise, I think that it's possible, but definitely not within a few hours, unless you did it before.
Kubernetes networking might be tricky. I wasn't able to implement it in a way I like. I used calico with overlay mode in the end, but I'd prefer "flat" network. Calico overlay mode was easy.
Upgrading kubernetes manually is easy. At least until it breaks, it didn't break for me yet. kubeadm just works.
Node autoscaling is hard. There're no simple solutions, at least I didn't find any. Our cluster does not autoscale. Manual scaling is not hard, though. I can implement autoscaling with loads of bash/terraform/cloudinit scripts but I'd rather not to. I think that node autoscaling is what managed kubernetes offering do well.
Now that we have kubernetes, story is not really over. We need to collect logs for nodes, for services, for pods. We need to collect metrics. We need to set up alerts. We need to have traces. Service mesh looks really nice so you can see how services interact with each other. Those things are not for given. They take lot of time and expertise. And while they're not strictly about kubernetes, usually you need them.
When rolling your own, I think you need to invest in a battle tested SAN or just not do stateful on k8s. For the handful of stateful services I have run on k8s it would have been better to just use a managed stateful solution.