paulfurtado

This isn't actually effective. When there are instance shortages and someone launches an on-demand instance, no matter how high you set your spot bid price your spot instance will still be terminated in favor of the on-demand instance. We run hundreds of spot instances with their max price set higher than on-demand and still see terminations on popular instance types: there are few day-to-day, but periodically nearly all of them are terminated at once. During those situations, if you went to replace the terminated spot instance with an on-demand instance, you'll even get an error stating that they're out of on-demand capacity for the instance type in that zone, so it's pretty clear that it happens when they've completely run out of capacity for some instance type in a zone

When a spot instance gets interrupted, they've supported terminate, stop, and hibernate for a while. Hibernate is effectively just stop, but has an additional hook inside the instance to tell linux to save memory to disk while shutting down.

This post is different: with on-demand instances you can stop them whenever you like and only be charged for storage while they're in the stopped state and then start them up when you need them again (keeping all the same settings, like the IP, etc). A stop/start is also the way to effectively move your instance to new hardware if there is an instance with the underlying host. Annoyingly, the stop/start API calls were rejected for spot instances. This announcement means this is now finally possible.

That said, as an operator of a number of fairly large kubernetes clusters (100-500 nodes), I might prefer java's memory tuneability to Go's garbage collector simply not caring about the max memory a system can offer.

At scale, we need to heavily oversize the kubernetes masters to deal with the spikiness of go memory consumption. It's possible for a kube apiserver to average 2GB of memory use and then jump to 18GB after suddenly handling more requests than usual and get OOM-killed. I'd much rather it simply slow down a bit than behave so erratically.

This is a common thing across all Go programs that handle data: since Go's garbage collector doesn't try to keep memory use within some upper bound, if you allocate and throw away a bunch of objects you'll quickly run out of memory on any low-memory system or container even though there's a ton of memory ready to be reclaimed.

Even a major library like the official AWS SDK's S3 client had this wrong as recently as 2017 (solution was to use sync.pool to avoid throwing away buffers): https://github.com/aws/aws-sdk-go/pull/1784

Python should perform much better from a memory perspective in these situations: its weakness would be the deserialized size that objects blow up to in memory, but its use would be bounded. Java would also handle this pretty effortlessly, though with a high enough rate of garbage production might hit a few short GC pauses that last far less time than a process restart.

Really wish go were better about this. Imagine that a simple Go cronjob that uploads a backup to s3 at under a 160MB/sec could risk you OOM killing your server if you don't set cgroup memory caps on all your go processes.

The use case here is turning an outlet or light on/off, so latency matters, but it requires nearly zero bandwidth (assuming that you're not talking tcp). Granted, x10 security cameras exist.

x10 speaks a protocol designed for this purpose and requires a base station.

There are classic home automation systems like x10 that potentially predate wifi and offer wired solutions that use the power lines themselves for wired communication, you might want to check these out

haproxy 1.9 added support for logging to stdout/stderr/files, this is a different feature.

This feature looks like it is intended for cases where you're logging to a remote syslog server without any local log files and want to attach to the process and see the live recent logs which it stores in a ring buffer.

In Boston, in the mission hill neighborhood, I was really surprised that the Bank of America ATM handed out $10 bills instead of $20 ~5-6 years ago and that was the only ATM I've encountered that does that.

That said, it seems like all of the Bank of America and TD Bank ATMs I use in Boston are starting to let you choose the bills you want on screen. Was recently in Montreal and had the same experience there too so I'm guessing that's what ATMs are trending towards.

You can make it dump monthly billing CSVs to s3 daily. You'll get a new CSV each day that contains the month to date so it gets bigger every day until it resets the next month.

The purpose of using a dynamic DNS service is because you don't have a static IP, isn't it?

Services that specifically provide free DDNS may not stick around long, but professional DNS providers with REST APIs change very rarely. I don't think cloudflare's DNS service will change very soon.

The goal is to lookup your own public IP address to store it in a DNS record. If you're behind a carrier grade NAT, there isn't much use in that exercise, right?

FWIW audi does do this rather effectively in my 2011 audi A4 and it also comes with a huge battery compared to other cars I've owned. You can tell if your battery is near EOL by the fact that the entertainment system starts taking 30s to come on after starting the car, but it also shuts off most other electronics as the battery dips lower as well.

When a battery was on the edge of failure, I was also amazed at how many days it continued to be able to start for, driving a mere 3 miles per say and starting it twice. The voltage gauge on an accessory I had would show the battery under 10 volts, I'd push the ignition button and silence would follow, then seconds later the faintest sound of an engine would start. Every day, for nearly an entire month of winter.

The way kubernetes uses this streaming is to keep datastructures up to date in memory. A node needs a list of all its pods, so it makes a list call at startup and then starts a watch from that resource version onwards to keep the in-memory list in sync.

The whole architecture is designed such that the etcd storage backend could be swapped out completely and the only thing that would care is the api servers. Much like the transition from etcdv2 over http with json objects to grpc with etcdv3 and protobufs.

You can also create alternative implementations of kubelet, kube-proxy, the scheduler, the controller-manager, etc because they all access the data via the api server's well-defined public facing API and anyone using the API can easily watch objects in any programming language using the same semantics as the rest of the API. It also works from browsers, etc.

Additionally, kubernetes supports RBAC for nodes themselves such that they can only see updates for objects related to pods running on them - you wouldn't want any node to be able to watch all secrets in the cluster needlessly.

Overall, I think we'd lose a lot if kubernetes switched to having all of its components access the data store directly. Every operator ultimately needs the same things the controller-manager, scheduler and the other kubernetes components need

What issue did you face around network namespace awareness in this setup?

cri-o and containerd run standard docker images seamlessly. They're mostly drop-in replacements for dockerd. So even if you're using standard dockerd at build time, that's perfectly fine.

Things like buildah, kaniko, img, and whatnot can build standard Dockerfiles fine and push to standard docker registries. I don't have much experience with them though.

Kata is great if you're already using docker, it also is increased isolation versus lxd.

If you don't already have a use case for docker, kubernetes, the image repository features, etc then of course kata wouldn't be useful to you over lxc.

Also, in case you're dealing with zombie processes in docker again and can't ditch docker, running a container with `docker run --init` gets you a minimal init process that reaps the zombies. Adding that flag to dockerd instead does this for all containers by default. It's insane that this isn't turned on by default since it's such a common problem. Additionally, if you want to run systemd inside the container similar to lxc, you can install the systemd-oci-hook and it will do the necessary setup for systemd to be happy.

Kata works pretty well. You set it as a runtime for docker and it gives a fantastic illusion that the container isn't different from any other docker container but uses a KVM VM with minimal hardware to run the container. It starts in under 200ms (and faster if you make it use firecracker), etc.

The problem your parent is pointing out is that AWS instances don't support nested virtualization (and it sounds like neither does Digital Ocean) so on both of these cloud providers you can't use something like kata containers or any nested VMs. GCP does support nested virtualization in any VM and aws `.metal` instance types do as well, but are rather expensive. It's really a shame that even with the new EC2 kvm-based hypervisor they still didn't enable nested virtualization for most instance types, otherwise we'd definitely be making heavy use of kata containers.

I wish Oracle hadn't bought Ravello Systems and that Ravello open sourced their binary translation stuff that made nested VMs possible in EC2 without the full overhead of software virtualization. Unfortunately, there are no open source implementations of similar software that I know of. Their blog is now hosted on oracle's site: https://blogs.oracle.com/ravello/nested-virtualization-with-...

In this case, this is a library for Go, not a standalone program, so it seems worthwhile to include the language in the description. If there's something technical about Go that makes pure-go implementations of things interesting it's that it's uncommon/frowned upon to use cgo to import c dependencies whereas in python/ruby/etc you can trivially wrap a C library for many things and call it a day.

Also, you mention you don't see similar posts for other languages, however, similar complaints are constantly lodged against Rust posts, maybe even more so than Go. A recent comment in one of those threads also mentioned similar musings were common against python earlier in its history.

Regardless, Go and Rust are popular languages on HN right now because they're new and interesting and being used to build interesting new software. In practice, other older languages are more popular for work, but as the library ecosystems improve, maybe that will change or maybe it won't.

Random cheap things I've had stolen: - old spice deoderant 6 pack

- listerine

- pepto bismol

- ibuprofen

- $20 azamax plant pesticide desperately needed while a houseplant was being destroyed by fungus gnats. Enough value I suppose but so random to steal.

- aquaphor (the tubes of stuff for dry lips)

- a box of pencils

I could honestly go on for much longer if I went through my purchase history. I live in a nice neighborhood in Boston and have a locked lobby where they drop off packages that I guess people tailgate into with high success. Or I suppose ringing the bell to a random apartment and saying "UPS" probably gets you buzzed in on most days.

The official base Ubuntu image is really not particularly large, I'm assuming you were using an image with additional things installed? But regardless, once pulled, it is cached locally. Be sure to reference the image by specific tag/sha rather than latest to ensure you're not doing excessive pulls.

Overlayfs 1 had many issues. Overlay 2 was buggy for a long time and we needed to patch in aufs in production for stability and lock around image pulls to prevent kernel deadlocks. But at this point, overlay2 is extremely stable in the 4.x kernel series in production. Though, there is still the copy-up quirk with hard links but that affects very few applications in the wild.

If you don't like overlay filesystems, you can use the devicemapper storage driver, and if you set it to direct-lvm mode it should be pretty equivalent to VM based volumes. When it comes to mounting in shared directories from the host though, I trust docker's bind mounts much more than VM based filesystem solutions.

Anyway, if you work alone or on a small team, VMs certainly suffice and the appeal of docker may be limited, but much of the convenience of docker comes from the ecosystem and immutability of images. I've always found it frustrating that such an ecosystem was built around docker when it could have been done with VMs all along. If your company is building docker image artifacts of your software as part of the CI system and that software has many dependencies, executing a production build becomes as easy as "docker run X" and reduces the need for developers to standardize their workstations on one linux distribution. That said, while I find this incredibly useful, it's rare that I do proper development with local docker images unless I'm briefly touching something with painful dependencies (like getting the frontend stack working to make a quick UI change as a backend developer)

So, for some companies, failing over between providers is actually viable and planned for in advance. But it is known in advance that it is time consuming and requires human effort.

The other case is really soft failures for multi-region companies. We degrade gracefully, but once that happens, the question becomes what other stuff can you bring back online. For example, this outage did not impact our infrastructure in GCP Frankfurt, however, it prevented internal traffic in GCP from reaching AWS in Virginia because we peer with GCP there. Also couldn't access the Google cloud API to fall back to VPN over public internet. In other cases, you might realize that your failover works, but timeouts are tuned poorly under the specific circumstances, or that disabling some feature brings the remainder of the product back online.

Additionally, you have people on standby to get everything back in order as soon as possible when the provider recover. Also, you may need to bring more of your support team online to deal with increased support calls during the outage.

My argument is that the kernel gives us namespaces, seccomp, selinux, apparmor, etc for isolation and attempting to implement all of the path resolution and permission checking from a privileged context outside of the container defeats all of that and requires reimplementing all of those guards from userspace, which feels futile. By using tar, you're left with serialized path strings and file contents rather than file descriptors, and it should be far easier to sanitize those strings than deal with the linux filesystem API.

I definitely recognize that the container process could ptrace the tar process, and with kubectl cp, it's even directly using whatever tar binary is in the container so tar could easily be malicious from the start, but what it can never do is break out of the container onto the node when the tar file is not being extracted onto the node using the docker daemon's prvileges, which is extremely important for multi-tenant environments.

If you executed your example command as root on the node, then yes, a vulnerability in the node's tar implementation could allow a malicious tar file to take over the node at extraction time, but tar does guard against this by default, as do standard posix user permissions: the tar extraction can happen in a completely unprivileged context.

I do view tar's extraction as a valid attack surface since modern tar implementations are complex, however, that would require a tar CVE and there's no reason that `docker cp`'s output target handling is any less vulnerable to the same problems. I really think the most important thing to guard against is at input time.

To be clear, docker exec is not currently vulnerable to this attack, right?

Currently, docker exec (somehow safely) fully enters the container's namespaces and cgroups, then exec's a command inside the container. My suggestion was basically to have a statically compiled C binary that executes in the fully untrusted context, which things can ptrace and manipulate all they want. The thought was that the C binary would open the file descriptor from inside the untrusted context so that it is incapable of doing anything privileged and then send the file descriptor back over the inherited unix socket via sendfd. I'd imagine the only way this could be vulnerable is if sendfd is vulnerable somehow since this means 100% of the path resolution happens from a fully isolated context.

The performance argument makes plenty of sense, but it sounds like it'd be solvable by just doing a classic tar pipe where tar (or similar) is running in the fully untrusted context and writing its output to a pipe (with no unix sockets involved). You'd just need to get that statically compiled tar binary into the container, similar to how `/dev/init` is done. Would this be unreasonable? `kubectl cp` is already doing an actual tar pipe via docker exec, the missing bit is that it fails if tar does not exist inside the container, so you'd need to inject it in. This would fully removed the complexity of chrootarchive and any path checking, and you'd be able to rely entirely on the security constraints of docker exec.

So, `docker exec <container id> cat /path/to/file` is definitely something that works, is as safe as `docker exec`, and doesn't involve running any runtime code inside the container. The expense is that you have cat buffering the data and then the runtime reading the data from a pipe and re-buffering that data. Is that really ludicrously expensive? It doesn't seem any worse than SCP, which is just SSH executing the SCP binary as the command and communicating over the stdin/stdout channels.

Doing it with cat is just an example, since cat may not be present inside the container. Instead of cat, and in the absence of fork-without-exec in go, you could execute a small C program injected similarly to `/dev/init` (or via memfd_create) that uses sendfd over a unix socket created by socketpair to pass back a file descriptor to the runtime so that the data is only buffered once.

It's certainly possible to call nsenter in Go but even so, unlike other namespaces, mount namespaces affect the entire process rather than threads, and the thread must not have launched any threads yet at the time that nsenter is called.

That said, docker does support `docker exec` which gives you a shell inside the container's namespaces and cgroups. I'd imagine they could do something similar and just not call exec once they've entered the container. This would be similar to calling `docker exec $containerid cat /path/to/file`

This is extremely common at major bars in most states in the US, especially if you're anywhere near a college. It's there to detect fakes. I'd imagine fake IDs are a much larger problem in the US, however, given that the drinking age is 21.

That said, at least years ago, the ID scanners would operate on data stored on the card and not make any network connection to a database. So then more expensive fake IDs began to exist that could scan fine. Then, I think some bars started getting more expensive ID scanners which actually do talk to some database.

That's due to no shortage of effort by the maintainers, however: one small bug can easily lead to memory safety problems.

Rust is great in this sense because you don't have to think about those issues.

That said, I'd pick rust over C for most projects simply because it's a nicer language to write IMHO. It feels like a modern high-level language with things like iterators, traits, etc, and it has a great package manager too. In comparison, I find writing C to be a huge chore, especially if you're doing much string processing

Ah, interesting. I wrongly assumed the SoftLayer bare metal servers might be closer to EC2's bare metal offerings where, from the outside, you're given the same set of management tools as a VM, and from the inside, a fairly limited hardware surface area is exposed to you.

Wasn't clear to me from the article whether the firmware was modified via the BMC's network interface or whether it was done locally over some hardware interface exposed to the customer's OS. Was this really done over the network?

I'm a little curious why the BMC hardware is even exposed to the customer at all. Is it provided as a feature or is this just an oversight? Can't they just hide all of this hardware from the customer's OS?

> For instance how you are charged for S3 traffic your buckets receive even if you don't have any files there. Or how traffic between zones that is outside of your control is still charged.

Can you say a bit more about these two things? When do you get charged for s3 traffic on a bucket with no files? What do you define as uncontrollable cross zone traffic here?