kevan

Manage identity centrally is probably referring to using an identity management system like Okta, Microsoft Identity, or hosting your own IdP and using strong hardware 2FA. You don't want people creating their own accounts manually for everything or shared accounts that everyone knows the password for (or is on a shared spreadsheet that the entire company has access to).

> III. Professional Obligations

> 1. Engineers shall be guided in all their relations by the highest standards of honesty and integrity.

> ...

> b. Engineers shall advise their clients or employers when they believe a project will not be successful.

There's nuance around how hard you should push back on bad requests and where ownership/accountability and decision-making responsibility ultimately lie but providing professional judgement/advise/opinions is definitely in bounds for engineers.

[1] https://www.nspe.org/resources/ethics/code-ethics

There's still differences. If you're running it in prod then the functionality has at least gone through code review and you have higher confidence what's running is what you think it is. If you run things from personal boxes there's always the risk of them not having the latest code, having made a local change and not checking it in, or the worst case of a bad actor doing whatever they want with the privileged role. But if code review isn't required or engineers have unrestricted SSH access to prod hosts then it's pretty much equivalent.

I remember news saying the FAA added supplemental type certs allowing G100UL for basically every engine in 2022. What gaps are there?

Commercial mowers do exist, for example Exmark[1] (some not launched yet) and Toro[2]. Based on the pricing I'd guess the breakeven vs gas is still >3yrs without tax credits.

[1] https://www.exmark.com/electric [2] https://www.toro.com/en/professional-contractor/commercial-m...

And make sure your project plan includes milestones for explicitly aligning all stakeholders on definitions otherwise you'll have a big hot potato game the first time the outputs show something unfavorable.

For an example, my team owns a dozen services and they have hundreds of direct and transient dependencies. Of those, maybe a dozen or two to need work to support the new version but that's a dozen different teams that have to put the work on their roadmap and prioritize it. When the entitlement is 'devs want to use shiny feature X with hard to quantify productivity benefit' it's difficult to prioritize. When there's an efficiency benefit then things move fast because a 10% efficiency improvement means 10% lower server costs and that's easy math.

This is just the data serialization format, you have to build any other functionality yourself. We do have a pattern on a few of our APIs where there's a big fixed schema (i.e. it's just a struct and you can't do GraphQL things like following references and hydrating them into objects) and clients select the subset of attributes they want and we only return that. It's useful for reducing response sizes but the main benefit is we can pretty easily track which attributes are actually used over time. That helps us deprecate attributes with a lot less pain.

The SDKs use Smithy[1] which is tailored for defining+generating services and SDKs, Ion is more of a pure data serialization format. It's definitely niche but my org uses it in a few places and it has some nice properties that fit our case pretty well (rapidly evolving schema, most clients only care about a small subset of attributes, ability to apply multiple and different schemas based on regions or businesses).

It's the sort of thing where I'd advise exploring other options first and only using it if the whys[2] really resonate with you because it definitely comes with some overhead.

[1] https://smithy.io/2.0/index.html [2] https://amazon-ion.github.io/ion-docs/guides/why.html

>and it's way better than Amazon internal tooling (is that 'industry standard' enough?).

It's slowly changing but I wouldn't consider Amazon's tooling to be industry standard by any definition.

The key to sleeping at night is to add metrics and alarms near those cliffs or 'edges of the map' so they aren't invisible anymore. It's hard to anticipate every angle to this but in any case where you're making this kind of assumption or tradeoff it's a good idea.

A simple example is setting an alarm on your max load test traffic. When you get that alarm you know your system is now operating in unknown territory and it's probably time to do another load test or at least take a close look at scaling.

Everything in moderation I suppose. I've experimented with marking large blocks (2-4hrs) as tentative for IC work a couple days/week recurring and it does help avoid the Swiss cheese of 30m meetings spread across the entire day.

Lately for a 1-3 day horizon I've been adding everything I'm working on as blocked. That definitely helps me visualize my priorities for the next couple days and whether I have room to fit more meetings or tasks. The main downside is I'm used to a more free-flowing schedule where I pick a task based on how I'm feeling.

Funnily enough floods (GCP) and fires (OVH) are two of the 3 things AWS explicitly mentions in the Well Architected docs. For a lot of companies an AZ going down is an annoyance or bad day but a whole region going down could be a real continuity risk.

> Each Availability Zone is separated by a meaningful physical distance from other zones to avoid correlated failure scenarios due to environmental hazards like fires, floods, and tornadoes.

https://docs.aws.amazon.com/wellarchitected/latest/reliabili...

>slightly easier

As a company grows sooner or later most of these features become pretty desirable from an operations perspective. Feature developers likely don't and shouldn't need to care. It probably starts with things like Auth and basic load balancing. As the company grows to dozens of teams and services then you'll start feeling pain around service discovery and wish you didn't need to implement yet another custom auth scheme to integrate with another department's service.

After a few retry storm outages people will start paying more attention to load shedding, autoscaling, circuit breakers, rate limiting.

More mature companies or ones with compliance obligations start thinking about zero-trust, TLS everywhere, auditing, and centralized telemetry.

Is there complexity? Absolutely. Is it worth it? That depends where your company is in its lifecycle. Sometimes yes, other times you're probably better off just building things and living with the fact that your load shedding strategy is "just tip over".

I'm a huge advocate for CI/CD pipelines and my team owns a lot of them. We're confident enough to deploy anytime but we choose to limit deploys to our team's business hours and not on Fridays. Why? Because we think the return going from deploying 4 days/week to 5 days/week is outweighed by the stress and morale hit of ruined weekend plans if something weird happens. There's probably situations where that extra speed makes a difference but for us deploying to all regions safely can take a full day anyways so it's pretty normal to have multiple changes flowing at the same time.

On #4, Coffeezilla got past the smokescreen and got SBF to admit there was "fungibility created during those withdrawals between assets" and that it had always operated this way.

[1] https://www.youtube.com/watch?v=4o_jPzBZSIo (~15:30 in)

In a similar vein, in the early days of launching Relay[1] I ended up spending a couple weeks in total squashing timestamp bugs. We were integrating with a few existing systems that used epoch seconds or milliseconds for timestamps and they usually didn't have a hint in the field name to tell what it was so it was really easy to miss in code reviews.

Our problems were caused by a mix of serialization format (JSON numbers) and not always converting into the language's date/time types at the boundary (sometimes raw epoch seconds/millis were passed around layers of code and only parsed into a date for display. That created opportunities for misinterpretation at every function call.

My general rules for non-performance critical code are

1. Always Parse into a first-class date/time/duration type at the serialization boundary.

2. Always use an unambiguous format (e.g. ISO-8601) for serialization

It's not the most efficient but lets you rely on the type system for everything in your code and only deal with conversion at one place.

[1] https://relay.amazon.com/

The philosophy is "1 > 2 > 0". This means ideally you'd prefer a single thing but single things often bottleneck delivery from different orgs as they have differing requirements. In that case you'll accept overlap/duplication if it keeps those orgs decoupled and both delivering faster.

I ran into this a couple years ago with partially duplicating a service that existed under another SVP org. It didn't feel good at the time but in hindsight it was probably less effort and higher chance of success than trying to align priorities across SVP orgs and making their thing work for our slightly different use cases.

>Somewhere your model of the code is broken, and you need to test find the breakage (model needs correction, or program needs correction).

Most of the harder bugs I run into fall into this category. Great list of techniques to use. For when I'm really stumped I go into a more formal scientific method to the splitting the problem space approach. Formulate a hypothesis that if answered will reduce the search space, either by ruling in or ruling out things. Write it down - this is the important part, test it, repeat.

It's slow but usually helps me keep forward momentum. Worst case it at least creates a list of things that need better logging to figure out what's going on

Not being able to check out customers is a really bad customer experience. It's a double whammy of wasting their time and they don't even get what they needed so it's worth investing to make that less likely. Things are probably better now but when I worked at a Sears our network connection to HQ wasn't reliable enough to depend on completely for checkout operations.

It's counterintuitive but when you're dealing with distributed systems lots of things are: https://aws.amazon.com/builders-library/avoiding-fallback-in...

I've found this really effective too. I've been using Anki instead but same general idea. Ramping up on new topics still takes decent time and effort but I've been really surprised how easy it is to retain things after the initial learn.

>It looked like shit the last time I used it (~2019) especially as your classes get more and more functions built into it. I also can't imagine how shitty it looks for codebases that have a significant coupling problem.

That's the point, right? Visually representing the complexity of the system. I've used IntelliJ to do this before to show why modifying certain behavior was so slow and error-prone. In that case there were 3-4 classes with heavily overlapping functionality because, surprise, in the past there were multiple teams contributing to the same codebase that all did their own thing.

Have you checked out CDK or Pulumi? CDK still has a few warts but overall I've been pretty happy with it. Haven't used Pulumi but from chatting with friends it sounds like it has a similar feel as CDK but supports multi-cloud like Terraform.

Security patches stopping after 3 years made me switch. The phone (Pixel 2 XL) was perfectly fine performance-wise and still competitive with camera and screen quality but it wasn't safe to use anymore. I could've flashed a different OS but getting security at work to accept that to enable email and calendar access is unlikely.

Do any relational DBs let you lock statistics or the query planner directly? Something like how ML pipelines are split between training and inference phases. Let me train the optimizer on a representative load (prod replay or synthetic) and then lock the query planner for my production traffic. Still take advantage of the black magic but control it so query plans don't unexpectedly change.

The product quality is awesome. The company has some bad history with chemical dumping and caused at least one superfund site. And used the standard tobacco playbook after finding health concerns with PFAS chemicals in the 1970s to delay the rest of the world from finding out how bad it was. They only started phasing them out in 2000 and paid off at least one academic researcher for another decade to prevent or slow the release of studies showing negative affects.[1]

[1] https://www.smh.com.au/lifestyle/health-and-wellness/toxic-s...

It's not just limited to startups. When my business unit in Amazon spun up we hired way too fast and ended up having a harder time shipping v1 of the product than if we would've started lean and grown organically. We ended up spending a lot of time on a modularized platform to enable ~5 matrixed feature teams to contribute across iOS and Android apps. Looking back, a small dedicated team for iOS and another team for Android probably could've shipped v1 in less than half the time with a lot less tech debt.

Disclaimer: I'm at Amazon but not in AWS

Fair, I was thinking of this in the non-cloud perspective where efficiency improvements can push you to upgrade even if the hardware still works. In cloud provider mode it makes sense to keep it around as long as it still works and it's not too annoying to run. It doesn't really matter how (in)efficient the hardware is because you set the pricing to keep it profitable as long as someone's willing to buy it.