if it's agentic stuff, they likely aren't hammering a core constantly and they will maybe sit idle quite often between model requests, so it makes sense. I do wonder how much memory they allocate to each one though.

it's just very efficient use of shared cores that is required to make these kinds of workloads cost efficient

A place I worked back around 2020 was running a Grafana instance per customer that got embedded on the web dashboard. We had 110 pods per GKE (kubernetes on gcp) 4 CPU node because that was a network imposed pod limit at the time. The nodes were usually idle--could have shoved a lot more on if not for the IP limit.

I think around that time Grafana changed their license tho so you couldn't host OSS Grafana as part of your service.

how is it impressive?