paulfurtado

Yes, all of this is definitely true and I'm not actually complaining about it, I have used arch on all my machines for 7 or 8 years and don't plan to change that anytime soon, but these problems are actually unique to using arch and other rolling distros. Ubuntu, Fedora, etc have coordinated releases with some level of testing that things are compatible with each other. On arch, these are problems one must deal with though. I don't mind dealing with it, I live for Linux stuff, but the point is that it does happen to real users and it's not for everyone.

> Partial upgrades are explicitly unsupported by the distro. Pacman allows you to do this, but you're on your own.

Definitely correct, the issue is when you're trying to install a package to get something done, and suddenly your faced with the proposition of either upgrading your whole system while trying to get work done or attempting to upgrade just the required libraries. The latter often carries less risk, but occasionally is very problematic.

> And this is why as an Arch user you will be nudged by experience to use software that gives a crap about quality.

The issue is that this is true of so many very major pieces of linux software. Upstream vendors don't necessarily test their stuff in a ton of contexts, plenty of bugs occur in the kernel/xorg/wayland/gnome/glibc/python/etc. All of these are very mainstream projects that are difficult to avoid.

I use arch on all my machines and there are really a lot of ways it has broken.

- Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.

- I've had gedit start crashing in new minor versions of gnome due to a setting being incompatible, needing to track that down and unset

- If you have python virtualenvs for development and system python is upgraded to a new major version, all your virtualenvs break

- If arch upgrades the major version of glibc during some random package install (rather than a system upgrade), and you don't upgrade the whole system at once, every app that didn't get upgraded will fail to start due to soname mismatches... and that can mean that pacman, sudo, etc are all busted (this has actually happened to me)

- If you do a full system upgrade and have AUR stuff installed, you need to be sure to upgrade the AUR stuff otherwise it could break due to being incompatible with any library that was upgraded.

- In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.

There are a lot of low level algorithms that benefit from custom allocators which allow them to take advantage of SIMD instructions.

Haven't used rust in a long time, but back in school I called out to the c++ fftw library from rust, which wants its own allocator to be used for this purpose.

That said, at least at that point in time (or due to my naivety) it was rather straightforward to hook up: https://github.com/PaulFurtado/rusty_bars/blob/master/src/ff...

That code looks like it just handled arrays, but I recall having implemented Vec at some point but I guess I probably removed it from the final code

Is UPX actually that much slower than extracting a gzipped docker layer? I didn't realize UPX decompression was much slower than other compression algorithms

The other use case is to distribute images more quickly over the network and improve image pull times, at the expense of slightly more memory use.

Similar to how saving disk is less useful to most people these days, RAM on servers in big docker/kubernetes clusters is also likely not much of a concern for binaries that are well under 100MB if you're not running a ton of copies on each server. In particular, it seems like this tradeoff is most useful for static go binaries, which don't benefit from shared system libraries and aren't likely to have many copies per server, but of course that definitely depends on the use case and I definitely do not recommend blindly using UPX without considering the tradeoffs.

It would be an amazing integration though if it were possible to integrate UPX with docker's layer compression such that the binary is extracted to the FS before running. But given how it all works, that's pretty challenging to implement and the network is only getting faster...

UPX does significantly more compression on binaries than gzip can on the layers. It's not just simple compression, it makes a lot of modifications to the binary to make it more compressible.

Definitely important to note, I wonder if upx could solve this via execing an memfd_create in some mode. Also wish upx came with a trivial extraction tool so you could use it for distribution but not runtime.

It definitely is, although a cool thing about busybox is that you can rebuild it with any applets disabled. I'm not sure just how small it can get though.

But yeah, the tradeoff of that 1MB is that you get a very functional shell environment vs just an http server. It's impressive just how small ut is, given everything that it is capable of

FWIW, if you wanted to get the static from-scratch go image a lot smaller, try running the binary through UPX. It typically improves compiled go binary sizes by absurd amounts: https://upx.github.io/

Also, https://busybox.net is a 1.0MB static binary that provides all of the standard posix-compliant tools (sh, grep, tail, etc) and actually comes with an httpd implementation. So if you wanted a more useful container at about 1MB, you could use it.

Edit: here's a previous thread on UPX that has 80 comments talking about the tradeoffs it makes. You should definitely consider these if you intend to use UPX: https://news.ycombinator.com/item?id=15456980

dnsmasq doesn't do authority checks so unfortunately an external cname to a .internal name may not resolve correctly if the upstream server gives back "additional data" containing the A record answer for the target .internal name. On the other hand, we use unbound instead of dnsmasq to solve this at work and it works great. If you just want to forward internal vs everything else differently it's great though.

If you run CoreDNS instead, they have a kubernetes mode of operation where it gives different answers based on the source IP of the query to optimize search domain lookups. So if the node hosting the VMs here did similar, you could solve this pretty easily using CoreDNS with a custom plugin but I don't love putting all my traffic through CoreDNS vs a more mature DNS server.

bind can do split horizon stuff based on source IP so that could be a potential solution, though I'm not sure if it's capable of split horizon forwarders.

By large amount of memory, I meant systems that have large amounts of memory that is nearly full. For example, a server with 256GB of memory and 255.9GB in use.

SSH is purely in memory, however, in order to allocate memory for it, linux will pull "free" memory out of whatever heavily fragmented corners it can find them in. And, it may even need to perform disk I/O to free memory that was tied up in various disk caches.

People refer to this as a "livelock", where Linux is going crazy doing lots of stuff but from userspace the system is completely frozen.

Facebook developed OOMD, a userspace oom killer to deal with this issue, their release blog post references the 30 minute livelocks they face: https://engineering.fb.com/2018/07/19/production-engineering...

They actually have gone so far as to submit kernel patches for newer PSI (pressure stall information) interfaces which they use in oomd to better detect stalls due to this thrashing

I'd imagine that in a shell context, most of your FDs would be closed on exec unless explicitly passed.

That said, why not use env vars which tell the child process what to expect on which file descriptor numbers?

Depending how crazy you want to get, you could also use unix sockets to pass around FDs instead of inheritance. In that scenario, a process would be passed a unix socket either by a socket pair file descriptor or the shell would have a named socket which it tells child processes about using an env var. You can then use the sendfd functionality of unix sockets to pass around arbitrary file descriptors with arbitrary metadata.

You could also use the unix socket to instead have the child process optionally ask the shell what the descriptor's metadata is. That way you'd keep backwards compatibility with unix tools, but the child process could sendfd the file descriptor back to its parent to ask for data about it. The parent could look it up by calling fstat on it and comparing the fstat data to the other file descriptors it sent to the child.

But regardless, when you can't add a new syscall and need to do file descriptor stuff on behalf of another process, unix sockets + sendfd can really start to look like a userspace syscall interface which you can do a lot with.

If you have a large amount of memory, are doing decent amounts of IO, and linux OOMs, the system becomes unresponsive for many minutes before killing any process. At which point, ssh sessions timeout endlessly. A serial console stands a chance.

There is then also any case where you're debugging AMI builds and need to fix grub, or the init system without waiting 20 minutes for a new AMI build each time.

Also, the existing console log feature in AWS is insultingly not real time. It doesn't typically update at all unless you're within minutes of boot or trigger a reboot and it only buffers something like 4kb so a reboot can easily fully replace the logs. This really sucks when you're trying to get the debug console output, so this feature finally solves that.

Do servers really catch on fire nearly daily at this scale?

I'm surprised it's using fuse-overlayfs instead of the overlay2 driver. Is it that you're using rootless containers in production? For rootful containers I think you'd have the same experience as docker here (or at least much closer).

A zone transfer or grep really isn't so terrible, even for someone who doesn't know much about dns it's just a quick memorized dig away. HTTP or grpc aren't really better CLI UX and if this is your infra you can just write a cli to display your dns-driven information cleanly.

Storing key-value pairs in TXT records is easily parseable in any language if you use a simple delimiter. This isn't even an abuse of DNS, the protocol has been used to serve arbitrary data forever.

All the other modern services like consul, etcd, zk, etc require keeping a quorum of servers happy and have pretty heavy clients. By contrast, it's so hard to take down DNS and spinning up a new server is as easy as copying a zone file to a new server and the new server could even be running a totally different dns implementation because zone files are so standardized. Plus, your tooling can directly parse zone files with whatever dns library you were using and have a trivial way to dump the data without any server at all.

DNS can be replicated in arbitrary configurations and everything supports DNS caching for really high HA.

When you do want dynamic discovery and don't want to implement direct zone file generation, there's always CoreDNS which has plugins for so many datasources.

And if you don't want to host it, there are tons of DNS providers with great uptime.

At this point, I really can't think of any solution for service discovery that's better than DNS for most cases. Especially since the majority of service discovery solutions end up returning hostnames instead of IP addresses so you're already taking a dependency on DNS. Other solutions only really add benefit if you need to store tons of metadata or take advantage of things like leader election, etc

If you have significant infrastructure in multiple cloud providers, colocation facilities, etc then it's very possible you want to keep your OS and configuration management systems consistent between all of them.

Amazon linux also comes pre-installed with a number of agents that a significant infrastructure may not want to have running at all times. So between releases you must determine what you want to strip from Amazon linux.

Additionally, if you build your ec2 AMIs in your CI/CD pipeline that becomes more annoying with Amazon Linux than it is for centos/debian which have clear unattended install paths.

Additionally, you may be significantly invested in your distro such that changing at any time takes a lot if effort and Amazon linux's releases are less predictable than other distros so if they go down a problematic direction for you, it could cost you a lot of effort to migrate.

Thinks like the ubuntu ec2-optimized AMIs are also no less optimized for ec2, and honestly Amazon linux's kernel these days is hardly different from any others in terms of performance. They apply a moderate patch set, but it's typically for early support for new instance types or just drivers. For example, some patches for their ARM hardware, or their ENA driver or lustrefs for their fsx product. All of these patches get upstreamed too. On top of their kernel, they have ec2 autotune which is a tiny python project that tunes like 10 sysctl that your config management system can easily handle (and may fight with).

Anyway, I agree that for small projects running in only AWS I think amazon linux is a great choice and potentially the best choice, but at larger shops I'm much less convinced. Although, even at smaller shops I think a lot of people are just more comfortable with ubuntu, there's a guide for just about anything on ubuntu, and pretty much no software that doesn't target ubuntu.

We operate about 10,000 servers in EC2 so I keep up with the Amazon kernel patches and apply the relevant ones to our internal kernel build for CentOS, but don't otherwise really look to the rest of amazon linux much

I wonder if these can easily run linux AMIs. Since OSX doesn't have native containerization, our CI system for mac builds runs linux on mac minis and runs osx VMs inside of linux since you're allowed to run osx vms on linux if linux is running on mac hardware. So in an ideal world, we'd run our own linux AMI on the baremetal mac instances and run OSX in VMs.

I'm sure it's possible of course, but I wonder how painful it would be to setup. Fighting EFI/secureboot? Some other bootloader?

If you add -XX:+AlwaysPreTouch to the JVM arguments, the JVM will pre-touch the entirety of the heap at startup time to avoid unpredictable page faults through the life of the application.

I'd imagine other HFT companies may also pay for the Azul JVM which goes even further and comes with a kernel module that the JVM coordinates with for memory allocation. The kernel module pre-reserves x% of system memory at the time it is loaded.

That article is about broken container filesystems, as in, the root filesystem of the container, but that filesystem is not where you would ever store the data of a database. In kubernetes/eks you would use a PersistentVolumeClaim with an ext4 or xfs filesystem. This would not share any of the overlay filesystem problems and StatefulSets are quite good at managing these.

We run over 900 very stateful MySQL database clusters and several hundred zookeeper clusters in kubernetes on ec2 with ebs successfully. Sure, we've hit our share of problems over the years, but storage itself is solid.

If the docker container was running with something other than --net=host, it could have been avoided easily with standard networking concepts (route tables with reverse path filtering or iptables rules), even if the attacker somehow managed to get CAP_NET_ADMIN in the container the host network namespace still would refuse the packets. Although, with --net=host you could actually add iptables rules that match based on the cgroup and limit the IPs/ports allowed. It'd also be possible to filter the container's syscalls with seccomp. I'm not entirely sure why the container had CAP_NET_ADMIN at all, which is required for tcpdump and the man in the middle. Also, using user namespaces would have limited the attacker's abilities even if they had root in the container. A lot of defense in depth techniques are possible here.

There's also: simply not leaving the gcp login backdoor open. We run our gcp instances similar to ec2: on first boot we take the ssh keys from the metadata service and lay them down and do not run the GCP agent, we have standard config management + ldap for login after the first boot. This means that a hacker gaining access to your GCP credentials can't gain a shell on an existing instance trivially.

crun is actually a project that competes with runc. runc (run container) is implemented in go, and crun is a mostly drop-in reimplementation of it in C that is more faster and simpler, it's currently gaining traction in rhel/centos/fedora and suse linux as it was first to implement newer features such as cgroup v2.

To run 64 bit VMs, you always needed to turn on hardware virtualization in the bios. To configure kvm, all you need to do is "modprobe kvm" if it isn't already loaded. At that point, everything else is user-space and 100% of the user-space dependencies are installed in the image. All the docs about libvirt on the github page are unnecessary. So full steps are really:

1. Enable hardware virtualization 2. modprobe kvm 3. docker build 4. docker run

and you have an OSX VM.

> It's not something that you can just throw into kubernetes, especially if you don't manage the kubernetes deployment yourself.

GCP and Azure support nested virtualization, so you actually could do this in a managed kubernetes cluster. It's plenty common to use privileged DaemonSets in kubernetes to load kernel modules for filesystems, storage, or iptables rules. If you're allowed to run privileged containers, it's trivial to run VMs like this in kubernetes.

It'd definitely be a harder sell for third party software to require a privileged container and /dev/kvm mounted when you run it in your environment, especially since nested virtualization is largely unavailable in AWS. It also requires that the correct kvm kernel module is loaded, etc.

However, if it was a product that required virtualization and that was recognized as a requirement, then also distributing a docker image that could do it would probably be useful for people in the "and if you don't have virtualization infrastructure, but have container orchestration and nodes that support virtualization, our service will also work in a privileged container" camp

If you have container orchestration in place, being able to use it to run VMs via qemu is actually incredibly useful and isn't really much of a yikes. Sure, you're losing the container's isolation features, but you have a VM there, which is even stronger isolation.

We do CI for our VM images in our kubernetes clusters. The build system already was in kubernetes, so putting the OS image testing in there was a big win.

The benefit of doing this is also that on a personal machine you can start playing with an OSX vm with a single docker run command with no other dependencies and many people already have docker setup, whereas standardized qemu/virtualization tooling is now much less common on developer machines

This actually exists in Racket and the DrRacket IDE for it. You could actually even use images as variable names, so back in school there was a funny lesson where the professor used the image of a sailboat as a variable name everywhere and joked "why wouldn't you be able to?"

It's pretty atypical to upgrade to the latest mainline linux kernel the day the kernel is released. Most distributions take time to pick up a new kernel and make it part of a major distribution release that is subject to more testing. The OP must be using a bleeding-edge rolling-release distro like Arch Linux or running a custom kernel. I wouldn't anticipate such problems if running a popular release based distro like Ubuntu, Fedora, Debian, etc.

Does anyone know of a generic way to detect that letsencrypt will revoke a certificates soon?

The goal would be to have our automation automatically rotate the certificates when similar issues occur in the future.

MySQL has multiple Unicode character sets. The one named "utf8" doesn't support all of utf8, but utf8mb4 supports it all.

If the language's sqlite bindings don't offer a way to load a database from a string, if you're on a modern linux kernel (3.17+) you can make use of the memfd_create syscall: it creates an anonymous memory-backed file descriptor equivalent to a tmpfs file, but no tmpfs filesystem needs to be mounted and there's no need to think about file paths.