Tell HN: AWS appears to be down again

us-west-1:

7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region.

8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery.

8:10 AM PST We have resolved the issue affecting Internet connectivity to the US-WEST-1 Region. Connectivity within the region was not affected by this event. The issue has been resolved and the service is operating normally.

us-west-2:

7:43 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region.

8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-2 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery.

8:14 AM PST We have resolved the issue affecting Internet connectivity to the US-WEST-2 Region. Connectivity within the region was not affected by this event. The issue has been resolved and the service is operating normally.

Why not? The big tech companies use each other all the time.

For example, set up a new firewall on macOS and you can see how many times Apple pulls data from Amazon or Azure or other competitors' APIs and services.

Pretty funny actually.

1: https://auth0.com/blog/auth0-architecture-running-in-multipl... 2: https://twitter.com/auth0/status/1471159935597793290

We've taken a massive turn away from a "decentralized" internet.

https://news.ycombinator.com/item?id=29516482

First of all, how dare you!

Second, shoulda used rust ¯\_(ツ)_/¯

also, building access systems should be hosted in the building they reside in for security reasons anyways.

Cloud-based badges make sense if you have locations with small staffs and no HR people or managers. Like if you're controlling access to a microwave tower on the top of a mountain.

But badges-in-the-cloud for an office building full of people who are being supervised by supposedly trusted managers, and all of whom has been vetted for security and by HR, is just being cheap.

Like the 1980's AT&T commercials used to say: "You get what you pay for."

If you plan to replicate all of AWS I'd agree with you. But if all you need is a handful of servers, you could end up with better uptime doing it in-house just because you don't have all the moving parts that make AWS tick, reducing the chance for something to go wrong.

My bare-metal servers stayed up during both of the recent outages, not because I'm some kind of genius that's better than the AWS engineers but just because it's a dead simple stack that has zero moving parts and my project doesn't require anything more complex.

The trade offs aren't quite that simple. Those specialists are necessary because they're building and maintaining infrastructure that's extremely complex since it has a crazy scale and has to be all things to all people. When you're running in-house, your infrastructure is simpler because it's custom tailored to your specific requirements and scale.

There are tradeoffs that make cloud vs local make sense in different contexts and there's no one right answer.

Edit: Hitting refresh a bunch finally got it open.

Hard to setup is relative. It all depends on what you're doing and how much reliability you need. For a side project or a dev server you can just start with Debian, stick to packaged software (most language runtimes and services such as Postgres or Redis are available) as much as possible and call it a day. You can even enable auto-updates on such a stable distro.

The knowledge you'll gain by dealing with bare-metal is also going to be useful in the cloud even in container environments.

I mean they're not particularly. Unless the use is extremely minimal, it'll always be cheaper to buy a small server even for a side project.

I use cloud VMs for projects that can live on $5/mo VMs because at that usage rate I'll never break even to buy a machine.

But as soon as your AWS bill is even like $50/mo, worth to start looking at alternatives.

edit: Yup. https://news.ycombinator.com/item?id=28479595

Really only 3?

A lot of web systems work this way - DNS records for example use a "TTL" which means "time to live." If the TTL is 60, then you throw it out of the cache after 60 seconds even if you have room in the cache, and you have no reason to believe it's invalid. This lets independent entities (like a DNS authority) make a change and get it rolled out everywhere.

I think the reason this is common is that proving cache invalidity is so hard, especially with the typical "dumb" cache appliances that are widely used. They just do stuff like cache the response bytes for a particular URL; they might not even understand HTTP beyond interpreting the request's headers, and certainly don't really understand the response.

See discussions from the last outage about the VP signoff needed to admit, I mean announce, an outage.

I know nothing but that still seems too generalized that it doesn’t have an exception somewhere in the world.

Yes, that is a lot of time to go hungry and testing just one item. And then you still don't know what actually gives you nutrition vs just not killing you (for example leaves that you can break down such as leaf lettuce, vs eating grass).

For instance, I manage a team that does "full stack" development, where full stack means I regularly interact with mechanical and manufacturing, operations, electrical engineers, battery and radio people, embedded developers, mobile, and most aspects of backend engineering. We had an issue where one of our chip suppliers changed their FW, didn't tell us, and we literally were taking apart units to get to the bottom of why units off the line weren't working properly. We go pretty deep. Still, at some point we throw our hands in the air and say "Hardware is hard, it's in the name."

So you are disagreeing with this statement and saying that, in fact, you are the person who knows how the whole thing works?

I knew tech work produced some large egos, but sheesh.

The problem is that this is really only possible with a toy model of a computer and very simple programs. Modern chips with their branch prediction and caching and threading and advanced vectorized operations and so on are vastly more complex. The 6502[2] was perhaps the last chip that one person could fully grok. Maybe a chip designer at Intel or AMD could understand the whole circuit in detail but no one else has the time - it would literally be a full time job. The same thing is true for operating systems - even if you're Raymond Chen, you can know a lot about Windows, but you can't know everything.

We learn just enough about the other parts of the system to convince ourselves that we understand the principles. We build the basic mental model we need to interact with other systems but all we can really do focus on our own specialized areas and hope that everyone else is doing their job. This works well enough until something like Spectre[3] or Meltdown[4] crops up and that's when we realize that we've been building castles in the sand.

[1]: https://www.nand2tetris.org/

[2]: https://en.wikipedia.org/wiki/MOS_Technology_6502

[3]: https://en.wikipedia.org/wiki/Spectre_(security_vulnerabilit...

[4]: https://en.wikipedia.org/wiki/Meltdown_(security_vulnerabili...

We really depend on three things - knowledge (stored collectively and in various media eg books), materials (tools and manufactured precursor goods, available via active supply chains or existing stores), and most importantly having our basic needs met trivially so that all our time is not sucked up addressing them.

A scenario where someone has a 'wasteland' to pick over for their basic needs, knowledge and materials looks quite different to a return to primitive living where what nature provides is all their is to work with. 'if you want to bake an apple pie, first you must invent the universe' or however it goes...

Then of course there's the question of why someone would have any interest in obtaining computing power were either of those scenarios to occur. Much like the 'how do we warn future civilisations about our nuclear waste' problem perhaps it is acceptable to not bother, they'll figure it out again on their own eventually given enough time.

This stuff is fun to think about at 6am when hay fever is preventing my sleep :)

I eventually got a fully-functioning 32-bit cpu with instruction pipelining, two levels of cache, DMA input/output, an asynchronous bus, a custom assembly language with an assembler written in python, and got the Game of Life running on it.

It ran about 2kHz with 8kb of memory or so.

From my perspective, we require abstractions in order to free our intellectual capacity up for the next layer of complexity.

Commit and push often.

We had solar panels and a generator we used only when absolutely necessary. We were never without power, but we lived with the constant anxiety of optimizing our energy consumption. Some stuff we could only do during the day and at night we only used devices with batteries.

For a couple of weeks we didn't have running water in the cabin because we were rebuilding our water deposit tower. We used buckets for everything.

That was almost a decade ago and I still feel grateful at having unlimited energy or running water on demand.

I also feel guilty at times when doing power hungry stuff like playing video games, knowing electricity production is by far the biggest driver of climate change.

What's probably closer to truth is that many humans were forced to join farming communities. Stronger individuals or tribes probably enslaved others, and then forced them to build and produce.

The patterns of inequity and the march toward hyper-specialization we still see today make sense in that context.

As a tangent, if anyone is interested in that "cavemanness" deep in our DNA, check out the idea of primitive camping. That was my first experience camping, and I expected an idealized tv-ad experience. The trip was not framed as "primitive camping" to me.

I was dealing with intense burnout, stress, ADHD symptoms, immune problems, trouble sleeping... And I was thrown into the desert in the summer with a tent and some beer. It fucking sucked sooo bad. It fucking sucked sooo bad that I forgot every stupid problem I had, because I spent the entire time in survival mode. Setting up camp. Hauling equipment up and down dunes. Staying hydrated in the 100f+ heat. Making food. Making sure my wife and friends were ok. Strategizing how to defend our camp from bugs and psychos.

I really have not had such an existentially-dense experience as that one. And no, I didn't take any mushrooms, as the rest of the group did. I wanted to be lookout. Maybe I come from a long line of hyperaware sentries.

[I only liked the first four books, but that's enough to cover the original story arc]

https://topatoco.com/products/qw-cheatsheet

And the T-shirt's companion bandana and spin-off book:

https://www.popularmechanics.com/culture/a23286104/how-to-in...

If I were a bit more clever, or maybe if I was 50 years older and had played with this kind of stuff growing up, I'd probably try to make a spark-gap transmitter. That seems to be in a sweet spot of not requiring too many super clever bits, and having obvious applications.