dastbe

the complexity is in different places. For these systems the sheer scale of throughput makes reasoning about them challenging.

the OP mentioned 1B requests/day, where there are systems handling 1B requests a second.

are these separate services? Are you going to integrate how services preempt into overall system design?

My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.

this was always the nice thing at amazon, because you never needed an atomic migration with version sets.

it was definitely a case of making many things worse to do than in a monorepo, but more things viable to do.

you can increase profits through volume or margin.

right now we are in a whole economy experiment to see how long you can increase margin through price increases and shrinkflation.

> That can be easily done with comments. If the PR are orthogonal, they could have been split.

Comments are ad-hoc and don't scale, relying on the author to interpret and adhere to the extent of the reviewers approval.

> And if they're not, I would really like to know how the part that I'm reviewing interacts with the rest of the changes.

you are free to look up, down, and around the stack; nobody is hiding the code from you. But in many cases this is just unnecessary.

> What's the benefit of this type of stacked PRs over a well-curated set of commits, and reviewing per commit?

For the people who work with stacked diffs (in phab/otherwise) this is exactly what they'd consider reviewing a well-curated set of commits one-by-one.

One distinction is that cognitively a unit of review (a PR, a diff) remains a single bound change. Comments are focused on that change and the PR does not grow with size of the feature

Another distinction is the ability to focus each part of the stack to a particular audience. One change may require review from an external team, another may be just your team mate, a third might be the consuming team. By focusing the stack to the different reviewers you can avoid ambiguity about "what a person is signing off on" in the stack.

aside: one thing that would be great for github reviews is the adoption of change ids such that comments persist across reviews with a rebase workflow.

unless their business is going to be putting raw physical infra in dcs, they are about 5 years away from getting to something that could compete with AWS or GCP

"The average SAT score among legacy students was 1543, while it was 1515 for non-legacy students."

the last number is for the quarter whereas the rest are annual. if they maintain revenue it will be 3.88m/spooner

because they’re not; anthropic is pushing enterprises to switch to API/token pricing

How does this work with a tool like jujutsu that provides native support for stacking and preferably don't have a separate tool mucking about with state?

one thing to add for passive healthchecking and clientside loadbalancing is that throughput and dilution of signal really matters.

there are obviously plenty of low/sparse call volume services where passive healthchecks would take forever to get signal, or signal is so infrequently collected its meaningless. and even with decent RPS, say 1m RPS distributed between 1000 caller replicas and 1000 callee replicas, that means that any one caller-callee pair is only seeing 1rps. Depending on your noise threshold, a centralized active healthcheck can respond much faster.

There are some ways to improve signal in the latter case using subsetting and aggregating/reporting controllers, but that all comes with added complexity.

From a dataplane perspective, it does mean your healthchecks are running from a different location than your proxy. So there are risks where routability is impacted for proxy -> dest but not for healthchecker -> dest.

For general reliability, you can create partitions of checkers and use quorum across partitions to determine what the health state is for a given dest. This also enables centralized monitoring to detect systemic issues with bad healthcheck configuration changes (i.e. are healthchecks failing because the service is unhealthy or because of a bad healthchecker?)

In industry, I personnaly know AWS has one or two health-check-as-a-service systems that they are using internally for LBs and DNS. Uber runs its own health-check-as-a-service system which it integrates with its managed proxy fleet as well as p2p discovery. IIRC Meta also has a system like this for at least some things? But maybe I'm misremembering.

kind of right, kind of wrong

* for client-side load balancing, it's entirely possible to move active healthchecking into a dedicated service and have its results be vended along with discovery. In fact, more managed server-side load balancers are also moving healthchecking out of band so they can scale the forwarding plane independently of probes.

* for server-side load balancing, it's entirely possible to shard forwarders to avoid SPOFs, typically by creating isolated increments and then using shuffle sharding by caller/callee to minimize overlap between workloads. I think Alibaba's canalmesh whitepaper covers such an approach.

As for scale, I think for almost everybody it's completely overblown to go with a p2p model. I think a reasonable estimate for a centralized proxy fleet is about 1% of infrastructure costs. If you want to save that, you need to have a team that can build/maintain your centralized proxy's capabilities in all the languages/frameworks your company uses, and you likely need to be build the proxy anyways for the long-tail. Whereas you can fund a much smaller team to focus on e2e ownership of your forwarding plane.

Add on top that you need a safe deployment strategy for updating the critical logic in all of these combinations, and continuous deployment to ensure your fixes roll out to the fleet in a timely fashion. This is itself a hard scaling problem.

They invented a language to avoid you imperatively updating infrastructure, but that's not what CDKTF does; it just makes it easier to materialize that declarative output.

It also makes it easier to reason about that output as you can avoid awkward iteration in your declarative spec.

my expectation is that they would either sell crucial RAM at such a low volume and/or such a high price that it would do more damage to the brand than sunsetting it and returning to it when the slowdown occurs.

Yes. the acknowledgement you're getting in your application code is that the data was persisted in sqlite on that host. There's no mechanism to delay acknowledgement until the write has been asynchronously persisted elsewhere.

litestream makes very few consistency guarantees compared to other datastores, and so I would expect most any issues found would be "working as intended".

at the end of the day with litestream, when you respond back to a client with a successful write you are only guaranteeing a replication factor of 1.

> the last place couldn’t because datadog apparently bills sidecar containers as additional hosts so using sidecar proxy would have doubled our datadog bill.

that seems like the tail wagging the dog

the problem is that they want to apply a number of stateful/lookaside load balancing strategies, which become more difficult to do in a fully decentralized system. it’s generally easier to asynchronously aggregate information and either decide routing updates centrally or redistribute that aggregate to inform local decisions.

My cheeky answer to "how should this be regulated?" is that sports betting isn't materially different from other high-risk private investments, so it should only be available to accredited investors. Imagine if fanduels/draftkings had to verify assets and income before taking a single bet?!

I was interested in this, so perusing I found https://govinfo.library.unt.edu/ngisc/reports/2.pdf which estimates in the late 90s

"Estimates of the scope of illegal sports betting in the United States range anywhere from $80 billion to $380 billion annually, making sports betting the most widespread and popular form of gambling in America."

which seems surprising even at the low end.

similarly from https://www.americangaming.org/new-aga-report-shows-american... in 2022

"AGA’s report estimates that Americans wager $63.8 billion with illegal bookies and offshore sites at a cost of $3.8 billion in gaming revenue and $700 million in state taxes. With Americans projected to place $100 billion in legal sports bets this year, these findings imply that illegal sportsbook operators are capturing nearly 40 percent of the U.S. sports betting market."

I think what would be more interesting to me is estimates on the unique number of citizens betting. Is it up? If so, how appreciably?

Go? I haven't see this particular unit before.

the article is a bit breathless, which seems par for the course for security blogs these days. And while "containers are not a security boundary" is evergreen and something AWS has been trumpeting since the beginning, they IMO should also try and make it a bit harder for your to get access to the host credentials.

I do know the ECS team highly indexes on maintaining backwards compatibility and minimizing migrations wherever possible, but this seems like a case where it's warranted.

why would they have access?

At this point, you might as well use no cloud provider because at some point someone may be able to be leveraged? Whether that's by your country, another country, or some other nefarious entity.

this would really be helped by demonstrating the wrong way to solve this, which is trying to think of the first balloon in the range and then combine with the subproblems. i expect this to be where people who know about dp are most likely to get hung up, and may not realize they can think about it the opposite way.

yeah, i would expect a candidate who hasn’t seen it before to have at best a 50/50 shot of realizing that the traversal is “in reverse” so to speak, and need a hint to move forward.

though once you know that i’d expect a candidate to bang it out fairly quickly. it’s not that many lines of code.

we’re talking about wiping out most of the stored wealth of roughly a quarter of all homeowners here. and they cannot take that home with them when a new job opportunity comes up or worse get fired/sick.

this sounds more like a suicide pact than a plan.

> In non-recourse states, you'd expect to see defaults as people leave the keys in the mail to reduce their housing costs by moving next door at the reduced price or rent. More broadly, people don't like seeing their wealth go down.

At the trade-off of never being able to get a mortgage again, unless that's the bailout these homeowners get. Almost everyone will either sit tight or rent/sell at a loss. That being said, you will lose out on the public and private support of everyone who bought a house since roughly 2020. It doesn't matter if you've got a 3% rate if you're not getting your down payment out of the house.

The plan that makes the most sense to me is to keep housing prices constant/barely increasing while letting 3% inflation and gradual lowering of interest rates do its thing. Eventually the houses won't seem that expensive and those who locked in at high rates and high prices have an offramp through refinancing.