> 99% of the time you'll be better off putting a docker container on a single box.
Sorry, what did you mean here? This doesn't make a lot of sense.
Sorry, what did you mean here? This doesn't make a lot of sense.
Kubernetes is so complicated and many organizations do not really think about what problems they are trying to solve before diving in. Kubernetes solves so many problems that most IT organizations don't actually have or that are perfectly solvable in a simpler way with the existing virtual-machine ecosystem.
Of course if you're not a human person it's different. Corporate people, being made up of multiple different people, care more about their human elements being able to interact with each other's work. There containers might help.
A kickass CICD setup and good developer tools probably should be done first for the typical company, otherwise the majority of the added engineering time will go into scaling the infrastructure setup.
I am pretty sure they're supposed to use a somewhat "Cooperative Namespacing" schema, that makes both the application and the OS/container abstraction layer more complex, while attempting to deliver lower overhead for said namespacing. Sorta like Cooperative Multitasking, compared to Preemptive Multitasking. And even that doesn't work for some situations; as an example, old versions of Java (7 and older) will use the incorrectly namespaced Linux file system APIs to incorrectly read whole-system limitations instead of current container limitations to setup memory and CPU parameters.
> remove a lot of pointless attack surface.
Not really, containers are for lower overhead cooperative namespacing, not security. You want bare minimum lightweight VMs for security.
Even if you don't use user or network namespaces it at least confers a desirable separation of application and its dependencies vs. underlying host at the filesystem level.
It simplifies administration and allows your app and its dependencies to move at a pace decoupled from the host kernel and its minimal dependencies required for providing ssh access and systemd.
There's a concept called COST - Configuration that Outperforms a Single Thread, which basically asks the question: "how many cores/servers does a system need to outperform a single thread/core". Often this means that to outperform a single beefy server you need to get to 100+ servers to overcome the coordination costs and actually do more processing or process more faster. Lots of workload specific details here, but the general application in data engineering is "How many nodes does your spark cluster need to outperform my laptop, or how many nodes does your cluster need to outperform a single turbo.large.beef instance?"
My belief is that most companies chase scalability wrong in data.
That said, there are some companies that handle telemetry at scale and actually need to record and analyze billions of events per second. In these cases you will trivially hit the scaling limits of a single server. 99% of companies are not here.
I could be convinced that this number is actually as low as 90% :)