paulfurtado

Yes, exactly. And even do things like inject virtual USB drives so you can reinstall an OS or something

Fwiw, if performance isn't paramount, you can have a lot of fun with LKL (linux kernel library) and UML (user mode Linux).

UML is full Linux running as a userspace process. Tools like guestfish use it to modify disk images. UML can only run on Linux though.

LKL lets you use the linux kernel as a library. However, only single threaded. But this allows you to use linux's tcp stack from userspace, or its filesystems, etc. LKL also works on non Linux platforms.

These are definitely a different shape than what you're looking for, but they could help you keep the real linux implementations alive from userspace far into the future.

There are really two types of KVMs... classic ones were used by consumers to plug multiple computers into one screen, keyboard, and mouse and flip back and forth between them.

Networked KVMs let you remotely access a computer or server. It's like VNC except by physically plugging into the ports. This is useful because it allows you to have remote access to a server when its OS is offline, tweak bios settings, watch it boot, reinstall the OS, etc.

Most true servers have this functionality and more built in (ipmi, idrac, redfish, etc are terminology for the feature). These plugin ones allow you to add that functionality to any machine. You're not likely to see tons of them in an actual data center, but they are incredibly useful for systems without remote management built in.

It is great for isolation. There are so many VM based containerization solutions at this point, like Kata Containers, gvisor, and Firecracker. With kata, your kubernetes pods run in isolated VMs. It also opens the door for live migration of apps between ec2 instances, making some kinds of maintenance easier when you have persistent workloads. Even if not for security, there are so many ways a workload can break a machine such that you need to reboot or replace (like detaching an ebs volume with a mounted xfs filesystem at the wrong moment).

The place I've probably wanted it the most though is in CI/CD systems: it's always been annoying to build and test system images in EC2 in a generic way.

It also allows for running other third party appliances unmodified in EC2.

But also, almost every other execution environment offers this: GCP, VMWare, KVM, etc, so it's frustrating that EC2 has only offered it on their bare metal instance types. When ec2 was using xen 10+ years ago, it made sense, but they've been on kvm since the inception of nitro.

The benefit of docker for home assistant is the packaging of it, rather than isolation. You can always run a container with host network mode and privileged mode so that it can access everything it needs to the same as if it were running directly on the host.

You don't need a fiber splicer for this. You can order a pre-terminated fiber cable online.

On fs.com you can order fiber in custom lengths. An armored pre-terminated 350ft OS2 duplex cable costs $128: https://www.fs.com/products/20720.html Non-armored would be as cheap as $40: https://www.fs.com/products/50147.html?attribute=58053&id=17...

If you don't have a conduit, you can buy direct-burial cable. Two strands at 350 feet would be $590: https://www.lanshack.com/2-Strand-CustomLine-Corning-ALTOS-O... 6 strands at 350 feet would be $687: https://www.lanshack.com/6-Strand-CustomLine-Corning-ALTOS-O...

If you have some extra length, just coil it somewhere in the wall and don't bother splicing or re-terminating it. You can also use keystone jacks or couplers at both ends too so you have flexibility later without re-running it through the conduit.

I used the VLAN "trick" for connecting my cable modem to my router for a few years in an old multi-floor apartment with pre-wired ethernet, but it's not ideal because the router is then not able to detect the link state of the modem. For example, if you unplug a cable modem and plug it back in, normally the link would go down on the router and then come back up, and when the link returns the router will attempt to fetch a new DHCP lease.

If you have a static IP, it should be fine, but this became an annoyance the couple of times the IP changed when I was living there.

This is of course not the purpose of your post, but since you're interested in this topic, I wanted to mention that you can now create memory-backed files on linux using the memfd_create syscall without using any filesystem (nor unlink) and you can also execute them without the /proc/self/fd trick by using the execveat syscall. In glibc, there is fexecve which uses execveat or falls back to the /proc trick on older kernels.

Do you have any references to specific bugs here? We depend pretty heavily on containers and I'd love to look into these and see if we are impacted and whether we should carry these patches

FWIW on AWS, all nitro instance types can boot as UEFI if the AMI has itself set to use UEFI and all arm64 instances only support UEFI. So if you stick to modern instance types, you can happily use UEFI in AWS.

GCP looks like it does UEFI too. And Azure Gen2 instances use UEFI too.

memfd is a tmpfs file descriptor, but does not use any mounted tmpfs filesystem. It works no matter what filesystems are mounted or access you have.

It's truly great for situations where APIs refuse to take anything other than files and you don't worry about cleanup. Ex: loading certs from memory into a python openssl context.

If the goal of the test is to debug a sad linux server, containers are going to severely limit what ways the server can be sad in, isn't it?

User namespaces have resulted in multiple new container breakout CVEs in the last year. Some guides actually recommend disabling user namespaces because they are still somewhat new and perilous.

If you use cri-o as the runtime along with an openshift container registry, it will actually verify signatures at the runtime layer. In addition to crio, podman and anything based on containers/image supports this too.

Really that just means a registry that sends back a header indicating it supports signatures and serves up the right signature endpoints. It's shocking this isn't more common.

But if you just want to check signatures at the cluster's point of entry, you can use an admission controller to block the pods from being created with unsigned images.

Are you from the US or another country? Heinz uses different ketchup recipes in different countries and I personally think there is a huge difference. In the US, it is by far the best ketchup, but I hate the heinz in Canada.

https://www.livingabroadincanada.com/2009/05/13/why-does-ket...

Since 2017, all new aws instance types have been "nitro", which is based on kvm https://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtual...

AWS does continue to support older instance types, however, but only legacy workloads are using them. They actually now even run the XEN based legacy instance types on nitro. https://perspectives.mvdirona.com/2021/11/xen-on-nitro-aws-n...

When standard filesystems like ext4 and xfs hit enough io errors,they unmount the filesystem. I find that this happens pretty reliably in AWS at least and I can't imagine the filesystem possibly continuing to do very much when 100% of the underlying data has disappeared.

That said, from further reading of the GCP docs, it does sound like if they detect a disk failure they will reboot the VM as part of the not-so-live migration.

Yes, under a graceful live migration with no hardware failure, the data is seamlessly moved to a new machine. The problem of moving local data is ultimately no different that live migrating the actual RAM in the machine. The performance does degrade briefly during the migration, but typically this is a very short time window.

You can read more about GCP live migrations here: https://cloud.google.com/compute/docs/instances/live-migrati...

I can actually say that not supporting this really does hinder crossplane adoption. At work we operate large shared kubernetes clusters. A team attempted to use crossplane and we immediately had to remove it from our clusters immediately because it made kubectl and a variety of other tools completely unusable due to all the rate limiting and how frequently it refreshes cached discovery data. They wanted to use crossplane for like 10-20 of its supported object types. Instead, they had to actually run crossplane inside of a vcluster because there is no option to filter the number of CRDs it creates.

So while I can get behind this sentiment philosophically, until something changes upstream in kubernetes, this makes it really difficult to use crossplane in a cluster used for anything else and it probably makes sense to offer a workaround until then. Also, in practice, any security conscious users running crossplane in production are probably going to give it AWS credentials scoped to only the resources they want to allow it to manage, so even if you do install all of the CRDs in the cluster, 90% of them won't work due to their AWS credentials anyway.

It's not exactly normal, but it's probably a common enough experience in the US with 120v circuits and 15A breakers. Especially if in an old building with imperfect wiring causing excess resistance.

Many devices also operate less efficiently in high heat. If the AC unit is on a circuit with other devices, it is possible that the influx current when starting the AC unit trips the breaker. One might even be able to get away with 2 AC units on a 15A breaker as long as both compressors never start at the exact same time, but cause a trip when then kick in together.

The disks are physically attached to the host. The VM running on that host moves from one host to another. GCP live-migrates every single VM running on GCP roughly once per week, so live migration is definitely seamless. Standard OSS hypervisors support live migration.

When hardware fails, the instance is migrated to another machine and behaves like the power cord was ripped out. It's possible they go down this path for failed disks too, but it's feasible that it is implemented as the disk magically starting to work again but being empty.

You can read more about GCP live migrations here: https://cloud.google.com/compute/docs/instances/live-migrati...

When a local disk fails in an instance, you end up with an empty disk upon live migration. The disk won't disappear, but you'll get IO errors, and then the IO errors will go away once the migration completes but your disk will be empty.

Both GCP and AWS provide super fast and cheap, but fallible local storage. If running an HA database, the solution is to mitigate disk failures by clustering at the database level. I've never operated scylladb before, but it claims to support high-availability and various consistency levels so the normal way to deal with this problem is to use 3 servers with fast local storage and replace an entire server when the disk fails.

At a huge price, EBS can finally get you near-local-nvme performance. If you use an io2 drive attached to a sufficiently sized r5b instance (and I think a few other instance types), you can achieve 260,000 IOPS and 7,500 MB/s throughput.

But up until the last year or two, you couldn't get anywhere near that with EBS and I'm sure as hardware advances, EBS will once again lag and you'll need to come up with similar solutions to remedy this.

Also, I guess AWS would fight them a little less here: the lack of live migrations at least means that a local failed disk is a failed disk and you can keep using the others.

I'm not sure if it comes out ahead on price, but I find Waka instant coffee to be great. Similar to your experience with Starbucks Via, it's the first brand that convinced me that instant coffee could be good. You can buy it on Amazon in little single-cup packets, or a bag of it.

It's definitely sad that there isn't built-in context support, but if you're looking for a solution to this, it is actually very easy to attach context to logs using threadlocal and a logging filter.

When running a webserver though, an even simpler trick is to add middleware that sets the current request's info as the current thread's name, and then including threadName in your log format.

With 10,000 nodes running kube-proxy it is a bit expensive and, more importantly: error prone. A problem on a single node that wasn't even talking to the app could stop that app from exiting indefinitely if acks were required and clusters this size already do gigabits of traffic in endpoints watches.

Additionally, there's no acks possible for clients of headless services, so just kube-proxy handling this doesn't go far enough.

But yeah, maybe accept that as a tradeoff for clusterip services, but more deeply integrate the real load balancer options.

With kubernetes you can at least add a preStop hook that sleeps for 60-120sec if the app is not designed to handle SIGTERM. The pod enters the terminating state just prior to executing the preStop hook.

Are you aware of the kernel keyring and the keyctl API? I've always been curious why more programs don't use it, I guess it's not the most ergonomic API and doesn't have many language bindings. https://man7.org/linux/man-pages/man2/keyctl.2.html

gvisor is awesome and works for particularly untrusted applications, but it's not a performance hit we'd be willing to take across the board and effectively only protects you from security bugs rather than other kernel issues. We run thousands of production database workloads, hundreds of load balancers, thousands web apps, ML jobs, batch processing, etc in kubernetes, most of which require as much performance as possible.

When an EBS volume for a pod goes impaired, if it's using xfs you can basically count the whole server as dead no matter how many xfs + block io timeouts you set. xfs will stop being able to mount/unmount any other filesystems once hung in an unmount call for one. With a proper VM, you'd passthrough the nvme device with pcie passthrough and the host would be totally unimpacted.

Also, gvisor's better mode requires kvm, but it's cool that it effectively functions with ptrace when you can't use kvm.