kevin_nisbet

That's a good one. One of my personal favorites was a device that truncated long passwords in the set password function. Didn't take too long to figure out what happened but I was worried for a moment I wouldn't be able to unbrick that device.

Is there any possibility there is a problem with the 429 response? So something is converting to say a bad gateway error because it doesn’t like the 429?

Not trying to criticize if this was already checked. Just something I’d try to double check out of being overly cautious.

Maybe a noise implementation? http://www.noiseprotocol.org/

Iirc this is what wireguard is based on and wireguard would be a great project to learn the crypto choices from.

Trying to bring an open mind, I could see a number of plausible scenarios where an engineer could do this, with various degrees of legitimacy.

It's certainly a complicated subject, but I think in general companies are really good, especially big ones, at getting people to work on things they might not be comfortable with otherwise. This thread has been talking the extremes like immigration status, but there are all kinds of subtle pressures as well. Some people might not believe they have the political capital to outright refuse a project (especially a pet project of the CEO) vs choose to accept and try to nudge the project onto more solid footing. And I suspect many engineers are terrified of being labelled as not a team player, which aids in the creation of group think, but makes it very difficult to foster a healthy culture of discussion that would bring forward the serious concerns of this work. And there is almost always some room of uncertainty as the last convincer... is it unethical to work on the project if the consumer is fully informed and offers consent to the invasion of privacy?

If there is an extreme where it's justifiable, for any reasonable engineer to accept the project, then it get's really muddy on where exactly the line is, and when it should be drawn.

I also suspect many of us envision ourselves having much more fortitude than we really do as well, imagining the heroic efforts we'd put in to changing a companies mind from a bad idea... where the more likely outcome for most of us is to fall silently into the background.

I hate to dispute with someone like Brendan Gregg, but I'm hoping vendors in this space take a more holistic approach to investigating the complete failure chain. I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure. It may be true, but if we don't do the analysis we could leave ourselves open to blindspots. There may also be plenty of alternative approaches that should be considered and appropriately discarded.

I think the part I specifically dispute is the only negative outcome is wasted CPU cycles. That's likely the case for the class of bug, but there are plenty of failure modes where a bad ruleset could badly brick a system and make it hard to recover.

That's not to say eBPF based security modules isn't the right choice for many vendors, just that let's understand what risks they do and do not avoid, and what part of the failure chain they particularly address.

I'm not sure this is correct, I suspect this works on a similar principal as something like gvisor where as I understand it syscalls are redirected to another userspace program. In gvisors case the kernel basically get's re-implemented in user-space to provide the secure container like implementation.

Also, as I recall some of the kernel based syscall based sandboxing have had a number of issues with dealing with some of the syscalls.

I haven't personally seen this particular case either but I have no doubt it could happen. I've seen orgs where a blameless type culture isn't natural, and I've had to explain to the leadership that publicly humiliating (in jest) someone for getting caught by the phishing tests or posting private data to a pastebin type service is a bad idea.

And I've interacted with plenty of people who externalize everything that goes wrong to them, naturally some of these folks will be in leadership positions.

We did this in the Gravity Kubernetes Distribution (which development is shut down), but we had to for the use case. Since the distribution was used to take kubernetes applications behind the firewall with no internet access we needed the registry... and it was dead simple just running the docker-distribution registry on some of the nodes.

In theory it wouldn't be hard to just take docker-distribution and run it as a pod in the cluster with an attached volume if you wanted a registry in the cluster. So it's probably somewhere between trivial and takes a bit of effort if you're really motivated to have something in cluster.

It's cool to see it, I was interested in trying something similar a couple years ago but priorities changed.

My interest was mainly around a hardening stand point. The base idea was the release system through IAM permissions would be the only system with any write access to the underlying S3 bucket. All the public / internet facing components could then be limited to read only access as part of the hardening.

This would of course be in addition to signing the images, but I don't think many of the customers at the time knew anything about or configured any of the signature verification mechanisms.

That combined with the scheduling layers... on the CI service the company I'm working for uses it takes like 2 minutes to just start the job.

Thanks for sharing, you're post really resonated with me and I want to learn more about Creo now.

While I don't have the shared experience, I think I've also landed on economic thinking as part of my primary model of how I think an eng team should function and been trying to push the concept with limited successes (although I've termed it differently). And I've never observed a team that I think has executed on it really well.

> So far my impression is that there are a whole lot of other things I need to worry about in Postgres before I spend time considering serial vs. random UUID vs. ordered UUID. Am I wrong here and this is something that really matters and you should invest more time in?

Like any sort of optimization I believe this will depend on your workload and what's important.

For me when I switched to UUIDv7 a few months ago, it was basically no effort to switch v4 to v7 on a relatively new system. I was observing much higher batch insertion latencies than I expected, and producing inserts that touch less of the btree on an index created a very noticeable reduction in insertion latencies. But my workloads and latencies may look nothing like yours. On amazon RDS instances with EBS volumes and relatively low memory, insertion latency stood out, so using strategies that reduce the number of disk blocks that are needed has an outsized performance impact.

This of course would produce different results on different hardware / system sizing.

whois is returning AWS and I don't see any of the normal cloudfront headers, but I do see a server header of nginx. So it doesn't look like cloudflare to me, I'd guess they're just running some ec2 instances with nginx configured to give the exact behaviour they need (as I recall they return cached pages to non logged in users, which is why you can sometimes log out and get the page to load when they're having issues). I also see awsdns in their ns records, so it looks to be like they're just doing Geo-dns in route53 to route to the closest instance they're running.

Yea I did alot of debugging of the networks hardest problems, and the crossover is especially true with telecom systems since all the inter-connectivity comes from the standards bodies, which share a lot of the same base and history. But some stuff differed as well, it was difficult for a bunch of the hardened circuit switched folks to move to an all IP world, and I only had a cursory view into the circuit switched world which was on it's way out when I was in industry.

Nothing like this story of getting bullied to do something, but I once talked a director out of rebooting the cell phone switch that handled all cellular phone calls for I think it was 4 provinces in Canada. This is something I don't think we had ever done before, and was expected to be a 100% outage for all cell phones calls + 911 for like 20 - 25 minutes or something.

We were having elevated error rates due to an earlier outage, and had just run the next closest hail mary to recovery the switch by telling it to resync all temporary state. The problem was that resync didn't work the way anyone on the team thought it did and the director didn't give enough time for it to be effective. We later found out the resync for a particular record didn't happen until the cell phone itself checked in with the network for a periodic update, so we had to wait for however long that timer was to expire on all cell phones. Luckily there was an active 911 call on the switch which caused the team to delay long enough for me to get the right data (the real world failure rate was far below 1% and starting to trend towards normal after the resync command) and talk the director out of the reboot.

The funny thing is that equipment wasn't even mine, I worked on the internet side of cell phones, and I don't even remember why I was pulled onto that outage call since it didn't have anything to do with my part of the network.

I agree it shouldn't be parroted as though it's the word of the lord, like any advice it will be more or less applicable on specific situations. I've been on both sides of the let's redesign a thing plenty of times.

The balance is somewhere in the middle, it's valuable advice for what can go wrong when you don't think through the implications of the decision making process, understand how and why the system works the way it does, and what risks exist for what can go wrong with a big redesign. But like anything, if the risks are understood, then those risks can be accepted, mitigated, or rejected if appropriate, or provide guidance on why the redesign investment isn't worth the associated risks.

I believe luck is a much larger component than most people realize, but I don't see it as the "luck" of when you were born.

There is all sorts of business luck that occurs. As one example, in a startup you get some early customers in a vertical and the government changes the rules to make you're tech necessary for the vertical. And then no one else recognizes or competes with you on that vertical for awhile. So there is a giant don't f it up component as another comment pointed to, but you also don't get run out of business by a slightly better execution from a bunch of competitors.

I don't know anything about this... I'm curious how small could the monitoring station be. Could it be something that could be mobile, like in a trailer or cargo container format, and dropped off at a site temporarily and moved.

This is sort of fair, but working very slowly occurs in more than one dimension.

Swap on means applications may be slower due to page faults requiring swapped out memory. Swap off may mean file system operations are slower due to less opportunity to file system cache frequently used objects.

There is still a trade off here on performance balanced against use case.

A few times in me career I've had a narrow use case to disable swap on some systems. The most notable is trying to limit interactions between garbage collected runtimes and the latency introduced by the page faults.

The TLDR is GC pauses were high enough to be causing problems in a particular service, at the time the swap was on rotating media so latencies on a page fault were high, and this was especially a problem during mark and sweep operations leading to long pauses. Due to other GC tuning and the overall IO workload, disabling the swap made sense for the particular servers.

But if I had to do it today the situation might be very different, I might have NVME drives to drop swap onto with much better access latencies, or be able to use mlock to lock critical pages and preventing swapping, etc.

Also, there are some very clear problems introduced with disabling swap, especially on NUMA systems. Again, the particular times I disabled swap, we were able to lock processes onto their relevant numa nodes and do other tuning to make this worthwhile.

So as a general rule, especially with modern hardware, I would agree that it isn't recommended. However, you can probably find narrow use cases where amongst a number of other tunings it makes sense to drop swap. Also, there are plenty of other things you likely want to tune before removing swap.

I'm totally with you, I find drawing things out, especially when dealing with someone who is clearly upset and acting out on their feelings is an incredibly important de-escalation technique.

I'm not really sure what to make of this article, it says itself it's a study commissioned by promoters of another methodology. If they're consultants trying to pitch their consulting services, they need these types of things to sell.

But my fundamental reaction is, what does failure mean. Because in my world view failure should be expected and accepted and learned from. It's entirely possible to spend a bunch of time avoiding every possible failure mode, and really not delivery much value at all... but we've successfully avoided the work being considered a failure.

Radio Access Network - basically think the cell phone towers and the equipment for those towers.

I was going to add the same thing, this isn't only something the telecom providers do, the equipment providers include lawful intercept features as part of the equipment, the telecom just has to setup access. I don't have a chance to check right now, but I think this is even part of the published standards.

One company I worked for did the enterprise version through a git submodule. So main repo was the OSS version (with it's own main), and the git submodule was a different program entrypoint that contained / activated the enterprise code. So devs would have both the OSS/Enterprise module checked out and could work on both together.

In this example as I recall most of the enterprise code was just activation code, so a properly motivated individual could probably figure out how to turn on some of the enterprise features, but most people won't bother to figure out how to do their own builds, and it's more work then finding the secret env variables in an open repo.

It's a little bit painful at times because changes that cover both code bases need two commits, and it isn't uncommon for someone to forget or commit the wrong submodule pointer. But depending on the team, it can work.

Not for everyone, just including it as an option I haven't seen mentioned in other comments.

Seems like a misleading chart since there are both a left side and right side scale. I can’t tell which is which but my guess is it’s to make Texas stand out more.

Early in my career I had to deal with similar issues, but with cellular networking equipment. I do think it's one of those things where if you have a way to detect the bit flips it happens way more than you think... but most people can't detect them and mostly get lucky enough that they never see an impact when it does happen.

But because we had some equipment that would checksum itself we were able to see the bit flips happen, probably every 1-3 weeks or so I'd find a bit flip.

I have the same question in my mind... seems like there is one of these reported every month or two and it's just asking for a more serious mistake to be made.

I don't think ion thrusters would work, for the balloon to maintain orbit you'd still be somewhere with enough atmospheric drag to be a problem. Ion thrusters only produce millinewtons worth of thrust, so they're very much relying on a no drag environment and the ability to run for continuous use for weeks to years.

This looks awesome, congrats on the launch.

I've been pondering this type of issue for awhile, not from an application standpoint, but from a dev environment where I want more resources than my laptop has, but it'll be very bursty workloads and I want them to just pause when I'm not using them. So I'll be excited to see how this tech develops... good luck!