Comment by anon20190326 | Hacker News Reader

Comment by anon20190326 | Hacker News Reader

Ex-Amazon SDE here.

A lot of this has changed.

First, there is a movement to build a lot of services in Native AWS instead of MAWS/Apollo.

Apollo doesn't require copying configs anymore; you can have the config exist as part of the package you are deploying. Generally, that's a best practice.

Pipelines can be configured as code too.

There is a centralized log service which requires onboarding. It does require some commandline tools, but it works. The logs get stored on S3, IIRC.

If containers suits your needs, you'd be hard-pressed to find someone telling you not to use it. Generally, though, you would want to use bare metal for Amazon's scale.

There is also a change to how NPM is being used at Amazon. It was a lot easier towards the end of my tenure, and was probably as close as it would get when working with Amazon's build systems.

Amazonians are generally conservative and don't use the latest and greatest unless it solves an actual customer need. Customer Obsession is still the defining leadership principle.

Any take on how this looks for someone who doesn't work at Amazon?

I know Apollo is their deployment tool. No idea how they plays into a Kubernetes or even container stack.

Conceptually, there is a lot of overlap. Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling. Dare I say closer to openshift? Brazil, the build tool, infrastructure and repository was similar to using nixpkgs. Brazil had some features beyond nixpkgs that were absolutely essential for an business like Amazon, but otherwise the goals and even techniques overlapped.

I onboarded very, very quickly by already knowing nixpkgs and container management. The concepts translated well. A few aspects that didn't either were things Amazon handled that other tools have yet to encounter: Cool to learn about. Or historical bits that didn't matter: fine to ignore.

> Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling

Have you looked at the codebase? Apollo and Brazil are tiny compared to Kubernetes & friends. You could reimplement them with a small script on top of traditional Linux package management (modulo the web UI).

As someone who worked on Apollo, I'd challenge this. Without divulging any details, Apollo is a non trivial system.

Yeah, I work adjacent to the Apollo org (and have worked near Pfrheak, too. Hi!). I would love to see a HackerNews commenter's idea of what it would take to re-implement Apollo. I'll bring popcorn.

> you would want to use bare metal for Amazon's scale

Containers are "bare metal".

Yeah i was confused by this too. Parent possibly thinking of virtual machines?

Fun fact: all of Google (including Google Cloud) is run in containers. IIRC even VMs are run in containers. The VMs, of course, could contain customer containers, too. It's literally containers "all the way down".

> Containers are "bare metal"

Please explain this use of jargon. Are you saying that a Docker container is "bare metal" in some culture? A VM is bare metal? What do you mean by container, and bare metal?

Containers are just namespaces for things within the Linux kernel. Unlike with VMs, you're not running separate instances of the OS, it's all run by just one kernel instance, and that kernel usually runs directly on the hardware, that is, on "bare metal". That "ubuntu" base image you can spin up does not actually run the Ubuntu kernel. As a result, bare metal containers incur none of the "virtualization penalty" that VMs do.

Cloud is, in fact, abnormal in terms of how containers are used in that it wraps them into VMs. That's in part because VMs were already there and they simplify provisioning of something that looks like "machines" from the outside, and in part because containers do not offer much in the way of security guarantees (precisely because all containers running on the same host share the kernel), whereas VMs do.

Ah, you're using it to mean "no virtualization". Got it, thanks.

I tend to use the term to mean single-tenant, user-supplied OS, non-virtualized myself but I see where you're coming from.

All of that may be true as well - if the user picks the underlying OS (i.e. kernel) to run the container engine and installs on bare metal, then literally everything you wrote applies.

My workstation runs multiple processes in multiple namespaces without containers. Is it bare metal? What if I install applications via flatpak? Still bare metal? Now what if I use docker?

It's tempting to say that containers don't have any virtualization costs, but they often do for networking. For some applications, the addition of another networking driver doesn't matter. For others, it does.

There is no "another networking driver" in containers. It's the same networking driver.

For standard Docker container deployments, there is. See: https://docs.docker.com/network/

That's not a "driver" in the traditional sense, meaning it doesn't talk to the actual hardware directly. It's just a bridge or iptables.

shrug It's additional software in the networking stack, and it has a performance penalty. I think it's fair to call it a "virtualization cost," since you're paying for the container's networking abstraction.

I think GP meant in the sense that Docker containers are just normal programs whose instructions are being executed directly by the CPU (just like how my laptop is executing Firefox as I type this) without relying on hardware virtualization features.

Reply on news.ycombinator.com