Ask HN: Do you run apps bare metal? | Hacker News Reader

Ask HN: Do you run apps bare metal?

41 pointssdevonoes73 comments

So, I am working on a side project and the way I deploy my (golang) application is basically:

- build binary

- copy binary, config files and static assets to the production server

- do blue green deployment (with nginx) to get zero-downtime deployment

- profit

(This is automated of course! I use Ansible, and I can easily rollback if needed. I can also deploy the same app to multiple machines if needed).

On my local machine I use Docker to test the Go code, but I don't really see the benefit of deploying my Go app in a container. My colleage told me "it's easier to deploy Docker containers. You just pull the image and voila!". I don't see how my approach could be "more complicated". Also, isn't my approach better in terms of performance? If my golang app runs "bare metal" instead of via a container, then sure the performance should be better, right?

View on HN

It would help to clarify some words:

"Bare metal" means that your application is running on the same OS that is also running the raw hardware (metal), aka no virtualization. Containers are not (generally) the same as virtualization.

The analog to bare metal is virtualized, where the hardware your program is seeing is not necessarily the hardware that is running on the actual host machine.

A docker container could ostensibly be considered running on bare metal. A container is really just isolation but the parent OS/kernel is in command. Here is a graphic that illustrates the differences: https://www.sdxcentral.com/wp-content/uploads/2019/05/Contai...

What you are really asking is do you need an abstraction layer or orchestration tool to manage doing this.

The short answer is that no you do not need it at all. If you can DIY this and are happy with it, that is sufficient. For example, a current deployment process for one of my clients (EC2 environment) involves stopping a custom systemd service, pulling the new binary/deps and then starting the systemd service. Really simple with a small instant of downtime, but within this environment that is not a problem.

It's also worth mentioning that even if you do not have a virtualized PC, you might still not be running on bare metal. Case in point: Java applications. They run in a language-specific virtual machine. (Also Scala, Clojure etc.) One could make an argument that this is also true for interpreted languages, but that gets into some splitting of hairs regarding the definition.

I don't consider the VM of Java to be "not bare metal" because you can still interface with the unadulterated operating system

"Bare metal" is running a program on the bare metal, without an operating system in between.

I've never heard bare metal used that way. Any time I've heard it used, it's referring to a lack of virtualization. What you're talking about is usually referred to as a unikernel.

No, what I'm talking about is usually referred to as bare metal: https://en.wikipedia.org/wiki/Bare_machine

While I have seen this definition used in some very specific circumstances (like high frequency trading where they can afford to hire people to actually write a fully customized app for the hardware), this definition is so specific as to be essentially useless as it would apply to virtually nobody.

Typical IT usage means something is running on an OS that’s installed on the hardware, as opposed to in a virtual machine.

I have never seen "bare metal" used to mean anything other than running directly on the hardware instead of through an OS, so I guess the term simply means different things in different communities. The group of people who understand "bare metal" to mean what I mean by it is much larger than "virtually nobody".

Based on other comments here, it appears this is not as clear as I thought. I can see it being used either way, but definitely understand that in the “pre-virtual-machine” era, it could be used this way.

Ah yes the "virtually" nonexistent sector of embedded development that "virtually nobody" works in.

It's not "used in some very specific circumstances", it's the very definition of bare metal.

https://en.wikipedia.org/wiki/Bare_machine

"Bare metal" traditionally means running with no OS. Your app takes the place of the entire OS.

And you can definitely run a bare metal application virtualised, just like you can run an OS virtualised.

We do it right now, though we run Node instead of Go. For us it's not 'copy a binary', but 'copy and untar', but it's still very simple.

One downside that we find surprizing is that way too many Ops people are more familiar or comfortable in cloud or Kubernetes environments these days. We find it hard to find local talent who are willing to deal with bare metal hardware.

Because of that we plan to migrate to k8s at some point later this year. I'm neither support nor oppose it, to be honest. Introducing an extra Docker build step is annoying, but the idea of adding a few hundreds of Yaml lines to get monitoring, log aggregation, tracing, etc. sounds really nice, too.

How do you do that with Node: do you ship Node binary and all packages together with each app in one tar file? Or do you manage Node versions on the server that runs the apps?

pkg "compiles" your app into a binary with a virtual file system inside the app. If you do a "require", the runtime rewrites that call and attempts to load the files from inside the sandbox virtual file system.

You can even provide assets (like say a JS app and associated HTML, CSS) that are stored inside the app. You can then build a full app as a single binary with no extra dependencies or files. There are a few gotchas like when you need native library support in your app (if you are using a sqlite driver) and then you need to ship a .node file with your app. But, generally, it is a great solution once you get the hang of it.

You compile against a specific version of node, and that runtime is included in your binary. You don't need the client to install node or choose a version, you lock that down when you build your app.

I've experimented with a lot of different nodejs packaging tools, but pkg seems to work the best with the fewest surprises.

You know there are some projects like vercel/pkg which allows you to "bundle" everything inside a single binary, you don't even need the Nose run-time, of course there is some downsides, tho it's pretty solid.

By "bare metal" you mean as a regular process in the OS instead of as a docker container?

Then it should really make no difference in terms of performance. A process in a docker container is also a process in the host OS, but with some network and user permission stuff on top.

When I read the title I thought you meant bare metal as with no OS layer, just a go app running in the processor, which I think would be pretty cool to see.

For any software that is I/O intensive, there is a large difference in performance between a container and an ordinary process, as well as a number of other sharp edges. Containers are only approximately equivalent for CPU intensive workloads.

How so? Are you running actual containers (i.e. on Linux bare-metal host) as opposed to docker VM's on a foreign OS?

How is this true?

A container _is_ an ordinary process.

https://iximiuz.com/en/posts/not-every-container-has-an-oper...

(we just discussed another article by the same author a few days ago on slimmer containers)

"The container process is isolated (namespaces) from the rest of the system and restricted from both the resource consumption (cgroups) and security (capabilities, AppArmor, Seccomp) standpoints. But in the end, this is still a regular process, same as any other process on the host system."

In a normal process (or VM in many cases) you can take direct control of the physical hardware via standard syscalls and mechanisms, which is idiomatic for a lot of high-performance infrastructure software. These appear to stop working or have unexpected (and suboptimal) behavior if the same code is executed in a container.

Note that configurations modifying syscall behavior is not unique to containers. There are other ways, only rarely encountered in the wild, to configure your Linux system to inadvertently break the same required syscall behavior.

Linux makes no guarantees about the behaviors of syscalls. Nonetheless, high-performance software is necessarily designed under the assumption of specific behaviors that are almost always true on bare metal and modern virtual machines when configured sensibly. AFAIK there is no way to configure a container to ensure the same behavior, which is the problem. VMs in the early days had many of the same issues. It may be sorted eventually (like with VMs) but for now it causes severe regressions in many codes.

I think you are either confusing virtual machines with containers, or using containers where any OS differences are transparently hidden by starting a VPS with a container in it.

Containers have no overhead, you use the system kernel to do I/O for you, just like any other process.

There is a litany of ways in which syscalls behave differently when using containers that degrade performance for high-performance software relative to bare metal (or virtual machines, ironically). Many people have observed and tried to debug these issues. There might be a way to empirically make containers work correctly for this type of software but no one has been able to discover how, and not for lack of trying.

No process serious about I/O performance uses the system kernel for I/O. Doing it in user space is integer factor higher throughput, hence why Linux has APIs to make this possible for software that needs it. These work perfectly on bare metal and virtual machines with negligible loss of performance in the case of the latter.

That aside, containers are currently (inexplicably) missing other capabilities that have been assumed and idiomatic for high-performance software for decades, such as strict hardware resource control and discovery. You can sort of work around this with gross hacks but that doesn’t scale well. Basically, containers made things that were easy for decades stop working or become complicated, and these things were highly valuable for some types of software so discarding them is not an option.

Isn't bare metal without an OS? At least this is what I've assumed when I've read the question.

[Edit] Thanks for the comments, seems I'm just plain old.

Bare metal usually refers to not running in a VM or other actual virtualization tech that boots a nested guest kernel inside the host kernel.

These days people most frequently seem to mean "without a hypervisor" when they say bare-metal. In this case I guess it's "without containers", which isn't far off. Since it's not a formally defined term I'm happy as long as it's clear what the question means, and this one is explained clearly enough imo.

It used to mean that to me; too. Language changes I guess.

I once had a small kernel patch for my ultrasparc that kept 14 of the system's 16 CPUs from being scheduled for anything but my application, just before they came up with cpu pinning and other, better interfaces for that... I didn't consider that "bare metal" at the time, but it was definitely thinly wrapped.

I've always interpreted it as running code directly on the processor with no OS. I made a "baremetal" bitcoin miner for the original raspberrypi, until it burned up the chip(and i learned why OSes are important). I guess it's been updated to mean an application running on the native OS.

> Isn't bare metal without an OS?

No. When you run with an OS, the OS doesn't intercept every single hardware instruction. It only interjects occasionally (or upon requests) - and the rest of the time, your code (or that of other processes) runs directly on the hardware.

... to be honest, that is also partially true in Virtual Machines, but there, the I/O devices and other facilities you see are mostly virtualized, so that when you interact with them you're actually triggering the VM host to run its code rather than triggering physical hardware to do something.

In modern virtualized environments, there are usually methods for doing I/O that approximately bypasses the host OS so that your VM is doing I/O against physical hardware. This wasn’t always the case several years ago but virtualized I/O performance can now be very close to bare metal if you write the code for it.

“ In computer science, bare machine (or bare metal) refers to a computer executing instructions directly on logic hardware without an intervening operating system. “

The definition may be changing, but at least Wikipedia still has it this way.

Yes.

Our contract is that a unit of deployment is a .tar.gz file, and inside that there is an executable file bin/run which starts the app. We build apps in a few languages, but they all have a start script like that.

You can use static or dynamic linking, compiled or interpreted languages, anything you like, as long as the start script sets it up correctly. For Go apps the script can be trivial.

Tarballs must be self-contained - binaries, libraries, config files, static assets. We make an exception for interpreters, because those are huge and change slowly. Those are manually installed in /opt/something. I'd like it if we did something more disciplined and reproducible, but it's good enough for now.

We don't have many machines. Maybe 20? You can get a lot done with 20 full-size physical machines.

I don't see how something like Docker would help us here. It would just be the same, but less hackable.

This sounds like neophobia or NIH syndrome. What you're doing is almost exactly the same as a docker deployment, except you're missing the ability to have some declarative leverage and tagging (unless your place invented that for themselves as well). There very few unhackable things about a Docker/container image, and typically that's for the better. The other advantage is that once you start using docker images, you now have more or less portable infrastructure. EKS, ECS, Google Cloud Run, Heroku, and on and on and on... But then again, it takes some time to realize that serverless is an advantage because if you're still young or your company isn't strapped for cash, they're fine hiring people to stay on call through the night and manage servers and upgrades and so on. Me? My fargate tasks run on servers that never need reboots or upgrades.

Yes. Go static binaries are super useful and simple to deploy. IMPO, this is one of the reasons Go has become so popular.

Docker became popular because it fixed a lot of deployment issues for dynamic scripting languages. And really, Go static binaries and/or Java jars solve that same problem. So there is no need to complicate these deployments with Docker.

I do dockerize some Go binaries to run in ECS/Fargate, but otherwise I see no real need or reason to do so.

While Java deployment has definitely become a lot easier in the past decade, I'd contend that java jars dont really solve the same problem. Uberjars get close, but you still have to either deploy the JVM yourself or ensure that your target environment is appropriately provisioned.

Yes, I frequently don't use any operating system and write bare metal C code, usually for Zynq (normally RFSoC or MPSoC class) FPGAs, although I also frequently use a Microblaze soft processor with it's entire memory in BRAM when doing initial system bring up (say for a new board).

Lots of people in here don't know what "Bare metal" means. There's no operating system - at all. Anything else is not bare metal.

To go further: Sometimes (if you're bringing up a new processor design) you don't even get cstdlib - you have to write it (or the processor's architecture won't even support C!). "Bare metal" means exactly that - it isn't the same as "no virtualization"... at all.

I think you have it right. Don't over engineer it especially early on. If you need to scale/automate with containers later, do it then. For now, just get that damn thing up and running.

I built a Go web app side project which runs smoothly on a VPS with nginx as reverse proxy, letsencrypt SSL and dead simple supervisor config to run. Boom. Forget containers. Forget docker.

Ignoring the proper use of "bare metal", you should review the pros/cons of deploying with Docker vs SCP.

Docker is helpful when you have a lot dependencies, or otherwise need to create a reproducible image (configuration, special directory structure, etc.) or when you want to control the process's resources (sandboxing).

Otherwise, because Go compiles into single binaries, if there's no other external dependencies/configuration to manage and you don't need process isolation/sandboxing, then Docker is just another moving part that adds complexity and could go wrong.

Your performance should not noticeably differ (on linux) between docker and just running as an uncontainerized process on the host. There's almost no difference as far as the kernel is concerned; it just tags your process slightly differently (different kernel namespaces and cgroups, but these apply to every process).

Bare metal usually refers to running without hardware or software virtualization (kernels within kernels), and if that were the case (docker on linux does not virtualize) it would be meaningfully faster, but that's not the case here.

The primary benefits you get from Docker may not be immediately apparent, so go for whatever works. Eventually, you'll probably find that it's substantially easier to scale out appropriately and automatically with some sort of orchestration tech, which is nearly all built on containers these days. Don't sweat it til you need it but at the same time, it's easier to learn sooner than when you're knee deep in tech debt for a dozen services.

Depending on how you setup networking for a docker container, you may see a notable network performance hit. Also, I've seen frustrating issues with the default docker bridge network that didn't play well with other services on our network, forcing us to switch to the "host" networking. The extra complexity & problems added by using a container solution may not give a positive ROI, even if the container abstraction provides more flexibility.

Bare metal generally means running on the host OS without any virtualization layer. Or used to mean that.

It means running on the hardware. bootloaders, kernels, hypervisors, dos apps, most embedded, grub invaders, efi/bios, are things which run on bare metal.

A process that runs inside an OS, is running inside an OS.

This is like when they redefined "literally" to literally mean figuratively. OK fine "language changes" but then now what word does the job that "literally" used to do? We had a word to express a concept, and now that word means the exact opposite of that concept, or worse, may still meam either one, making it meaningless and non-functional.

I once complained on HN how appearances of literally in the new sense were driving me nuts. It was explained to me that it's nowadays used as an intensifier—and has been for nearly 100 years, and my impression the change was recent was probably just the Baader–Meinhof phenomenon.

Although unconvinced, I gave up that fight and retreated to my usual protesting that a method and a methodology are two very different things, although it seems everything is a methodology these days. Longer is better?

There's nothing really wrong with what you're doing. Apps were deployed like that for decades before docker came along.

To my mind the main benefit of containerization would be that the

> binary, config files and static assets

would be bundled together inside the container, so there is less risk of you running a half-baked version where (e.g.) the binary gets updated but the other files don't. However since you're using Ansible to deploy, that risk already feels small. You could also consider using the embed feature in go1.16 to bundle all the assets inside the binary.

Arguably there might also some security benefit to running in a container, but I wouldn't want to try and make that argument without knowing a lot more about the specific details of your binary's behaviour.

There's a significant security benefit because container will run with separate namespacing of all system resources, and make it easier to drop unneeded privileges. So the attack surface of any security issue within your app will be correspondingly reduced.

This is interesting. What if OP is running their golang application under a “golang” user and group? Wouldn’t that be enough to limit the surface area of attack? I’m assuming here that the “golang” user has limited permissions in the OS.

The argument for using containers is primarily in your first two steps, far less so for deployment and profit(!). Having all of your apps code and dependencies in a self-contained, er...container, that was portable between different systems and perhaps more importantly, different developers, is arguably the main reason for using Docker.

If for now it's just you and you're not planning on spinning up in AWS ECS, CloudRun, any flavour of Kubernetes etc. then go with what works for you. All good.

Not sure if this is applicable to your case, but if the application is/can be used by an end user, then having it available as a Docker image can be really useful to the user, and reduces friction to try out your program.

Many times while looking through a project I find on GitHub, if they have a docker image available to try it out, I can try it by running a single command. But if they have more than 3 installation steps, I usually pass.

This also depends on what kind of software or if it's a hobby vs work related.

I usually do the opposite, even if there's a docker image available, to try it out I follow the build and manual setup. I need to know _how_ it works even if at the end I use the image. If something breaks, I own this and need to be able to debug it. Depending on the program and its architecture, I also need to know how to scale it.

I personally can not stand when an app comes with it's own entire surrounding operating system. I already have an operating system whose very job is to provide a standardized environment for processes to run in. I don't want 50 OS's where I used to have 50 processes. I can't stand AppImage, snaps, flatpack, docker images, etc for mere local desktop apps.

It's fine to serve as a reference, and for reproducible builds, and for appliances like a mail server. But I never opt to run an app that way even just to try it out, and I especially can not stand when an app developer only offers that as the only way they distribute an app (there are some, and not open source so you really have no choice). This prioritizes the developers convenience over the users. If it's their right to do so, then it must be exactly equally my right to say I don't like that and try to avoid using their apps.

Full agreement here. Drives me nuts when I see a Docker image for something that should be a simple zip file and some basic deploy or build instructions, but I can see why people prefer it.

Just to expand.

Container technology run your software just as it was a standard process.

There are no performance implications in running code inside or outside the container.

Here be careful, that on MacOS docker is implemented as a virtual machine. In such case there could be somehow serious performance implications. Especially around IO.

To finish up the point on performance I link this very interesting answer on so:

https://stackoverflow.com/questions/21889053/what-is-the-run...

You have it right. Now of course if your project expands and has a front-end build step or some other process, etc. then maybe it's helpful to add a container to not have to get that working in two environments.

But with just a Go binary, I think you're doing it right.

I don't think bare metal means what you think it means.

For Go applications, docker is unnecessary. Its just a single binary and systemd is enough. You don't need containers for it.

While it's not necessary, simply saying use 'systemd' is not enough.

How do you handle multiple applications not interacting or affecting the data of another if there's an exploit?

for some folks systemd is also 5th wheel in the cart

There are times when a go application needs a lot of additional files for configuration and anything else. Docker is a great way to bundle those in instead of needing to ensure those files are already on all of your compute nodes.

I think most containers also run "bare metal", they just are more isolated as far as process/memory/fs, etc.

Having worked with and without Docker for various web apps, it removes some dependency management and server setup at the cost of another layer (or two...) of complexity. It's not always worth it to go to containers.

Docker seems to make the most sense in cloud environments that scale horizontally.

What benefits would you get exactly?

Yes.

Try to deploy code without Docker and you have one problem.

Try to deploy code with Docker and you have two problems.

It's a nice joke, but IMHO it's not true... unless you don't know how Docker works

But then, by definition, the joke is true. Even if learning Docker makes sense and has a positive ROI it’a still an extra problem.

It is.

You might have a 10 second build process that does 1 MB of disk IO and it becomes a 100 second build process that does 10 GB of disk IO because you ran it in Docker.

I see people who "know how Docker works" put no care at all in the order of their image builds and end up doing 10-100x more IO than they need to.

Docker is fast compared to the time scale most people deploy servers at but is slow compared to the scale of a healthy software development process particularly if you are not running on a cloud server or are working on a gigabit connection.

If you took the time you spent trying to eliminate entropy from Docker, Kubernetes and such and actually tracked down the entropy that comes in from your environment (like the vandal who set your platform charset to not be "utf-8") you might find you don't have time to check your phone or send your co-workers stupid GIFs on slack waiting for builds.

> I see people who "know how Docker works" put no care at all in the order of their image builds and end up doing 10-100x more IO than they need to.

I don't want to go into no-true-scotsman territory here, but this really is a literal contradiction.

> You might have a 10 second build process that does 1 MB of disk IO and it becomes a 100 second build process that does 10 GB of disk IO because you ran it in Docker.

How and why? Unless you use a wrong image ( like Alpine for Python) or your Dockerfile is poorly made and cache doesn't work, nothing should change besides immutability and proper and clean dependencies ( your code won't work because you have some lib installed on your local machine by chance, and fail in prod because the lib doesn't exist or is ot a wrong version).

> Unless you use a wrong image ( like Alpine for Python) or your Dockerfile is poorly made and cache doesn't work

1. this is implementation detail.

2. it shouldn't be so easy to write a bad Dockerfile

3. if it works, it works, and Docker encourages this approach.

4. and this damn cache: Docker is only tolerated in any way at all because of this. alternatively, it might be that baking an apple pie from scratch by first creating the universe is not the sensible thing to do.

i think given the mass proliferation of Docker, i'm going to face some disagreement with this opinion: i don't think we're in a particularly good place where we have a billion deployments musl libc, busybox, nginx, and what else, all of varying versions. on top of the security implications of this, think of all the time and energy wasted in transferring all of this repeated data, on every deployment.

i think it's madness.

I never got into the Docker habit because I spent a lot of time behind a slow DSL connection.

Assuming it worked, I could wait a few hours to bring down an image for the the first time. Often I could get the job done in the time it takes to suck down the first image if "doing the job" could be done without Docker.

I never managed to use it enough that the images were in the cache when I needed them!

To make matters worse, unlike many systems that do big downloads (esp. BitTorrent) Docker is not reliable at doing large image downloads.

What people don't get is that the internet infrastructure is more expensive than they think because it is paid for in bits and pieces everywhere -- for instance it is an environment disaster that console games now come in 100 GB + downloads:

https://gizmodo.com/downloaded-games-have-a-larger-carbon-fo...

You know need to know how your server works, how docker works and how your application works.

You are introducing another layer of complexity.

But you are getting the advantage of not upgrading a system library and break your Go application because it's not anymore up-to-date with the latest and greatest dependency.

Unless you built it statically, that's another thing

Of course, but if you update your docker image, you're in the same boat.

The solution shouldn't be don't install security updates. It should be having a workflow that tests and validates security updates before deploying them to production.

And you should have this with or without docker.

If it works for you don't sweat it.

Docker is fine but if you don't need it then fine (and managing it yourself is usually a pain). But do make sure your systems are up to date (basically your linux distro)

Yes, I do this too (although you're probably running on a hypervisor unless its your own hardware, so not completely bare metal). This is one of Go's biggest advantages. Deployment is way simpler since you don't have to configure a bunch of dev ops technologies.

Reply on news.ycombinator.com