1,497 karma · joined July 22, 2014
tl;dr Modern art existed before the CIA and even CIA predecessors did not originate it, but they did push it abroad to show that America is great because we put up with modern art made by socialists depite most americans disliking both the art and the artists
In the library scenario, I know tooling like bazel ensures that’s one changeset, not 3+. Tests run for both the changed library and all consumers of it in the same pass. You’re right that it might loop in others for review who weren’t expecting it, but i think that’s the same in a polyrepo approach.
if a library is used by multiple services, and gets an important bug fix, each service using the library needs to update to get the fix
these are sequences of changes, not literally at the same time or requiring deployment coordination, but when this happens a lot people start asking about monorepos
I know it’s a bit cliche at this point, but this harkens to “programming as theory building”[0] which I agree is easy to lose out on when embracing agentic coding today.
I built and operated an application that used 1-2000 L4 GPUs in production for a couple of years. Long-term GCP reservations running in a GKE cluster. At that scale we had a few GPU failures per week, and once saw 3 in a day. Nearly every failure required an engineer to manually intervene to get rid of the bad node, and then file a ticket with GCP support as they requested.
NVIDIA GPUs throw an “XID” code when they fail, which can be seen from serial port logs for GCP compute nodes (not through k8s!). If you’re lucky, it fires right as your application starts to fail, but there’s often a delay of several minutes. Even when you get an XID, by default GKE only responds to one or a few of them. You can expand the list via configuration, but the reconciliation loop is so slow that might still take 10-15 minutes during which a pod is puking errors and someone might be getting paged.
They’re working on it, and we never saw a single XID after we migrated to H100s, so the situation is improving. I imagine other clouds are even worse, though iirc Azure was leading some effort to improve k8s node problem detector to include accelerator problems so maybe they have a better story.
Training workloads are rather more sensitive to a node failure given that many modern training runs (including SFT etc) need multiple nodes where topology matters, and they might not have another e.g. 8xH100 box in the right place when one fails.
I suspect the overlap of those that can afford it, and those that have the talent to manage it, is a fairly thin slice of the Venn diagram. Even the large corps are gonna be getting it from the inference vendors, or more likely Bedrock and friends.
I think that could be simplified, so the tool can tell that a commit is reverting a breaking change and thus the version should be decremented, but at least there's an escape hatch.
I love it, knowing about these little details. Also fun to share with friends that inquire about the various LEGO on display in the house. This, and all the fancy mechanics (e.g. typewriter, nintendo), engender a ton of respect and awe for the designers.
Went on for about a year, worse each week, before i left.
That could be an incentive to keep companies small, but high-scale companies do have unique benefits to society.
I noticed it was especially bad for on-call and incident response; these managers get pulled in to all the incidents because of their status and supposed involvement, but are not particularly useful in those rooms, adding even more cooks to the already crowded kitchen.
how can this be? isn’t this a trivial metric to pull from any clouds monitoring service?
to get the good ones (H100+) you generally have to reserve them, a fixed cost you pay monthly and can’t pretend to not know
Data Engineers took over the plumbing once they moved on from Scala and Spark. ML Engineers took over the modeling (and LLMs are now killing this job too, as it’s rare to need model training outside of big labs). Data analysts have to know SQL and python these days, and most DS are now just this, but with a nicer title and higher pay.
Once upon a time I thought DS would be much more about deeper statistics and causal inference, but those have proven to be rare, niche needs outside soft science academia.