aschleck

Atherton closed their station in 2020 and, through a great twist of irony, it will become a museum for the line that's still alive next to it: https://inmenlo.com/2025/06/16/community-interest-meeting-on...

It's been a while since I thought about this but isn't the reason providers advertise only 3.2tbps because that's the limit of a single node's connection to the IB network? DGX is spec'ed to pair each H100 with a Connect-X 7 NIC and those cap out at 400gbps. 8 gpus * 400gbps / gpu = 3.2tbps.

Quiz 2 is confusingly worded but is, iiuc, referring to intranode GPU connections rather than internode networking.

Bazel makes easy things hard and impossible things possible

I had this problem and complained to AT&T support. I know it doesn't make sense to replace the modem because this is a software, not hardware, issue but they very quickly offered to replace the modem and now my DNS isn't getting hijacked. Would recommend trying it!

This is both a good thing and a bad thing, but Pulumi is way more flexible than Terraform. I wanted to have a cloud-provider-specific submodule that created resources (like EKS and GKE) that then exported output values (think kubeconfig), and then I wanted the parent module to pass those in as inputs to a cloud-provider-independent submodule. Terraform couldn't do it without needing to duplicate a ton of code, or without something heinous like Terragrunt (not sure it even would have worked.) Pulumi makes it trivial and in a language I like writing.

Additionally, our applications consume our cloud configuration (eg something that launches pods on heterogenous GPUs needs to know which clusters support which GPUs, our colo cluster has H100s but our Google cluster has A100s etc.) Writing in the same language in the same monorepo makes it very easy to share that state.

I found that using "helm template" to convert every Helm chart into yaml, and then using Pulumi to track changes and update my clusters (with Python transformation functions to get per-cluster configuration) made my life so much better than using Helm. Watching Pulumi or Terraform watch Helm watch Kubernetes update a deployment felt pointlessly complicated.

1979 16 bit flops on an H100 is with sparsity. See footnote 2 on https://www.nvidia.com/en-us/data-center/h100/. You should be halving it for non-sparse flops.

It's worth noting that just because an H100 has a higher flops number doesn't mean your program is actually hitting that number of flops. Modern TPUs are surprisingly competitive with Nvidia on a perf/$ metric, if you're doing cloud ML they are absolutely worth a look. We have been keeping costs down by racking our own GPUs but TPUs are so cost effective that we need to do some thinking about changing our approach.

I'm not certain but I think part of this is that XLA (for example) is a mountain of chip-specific optimizations between your code and the actual operations. So comparing your throughput between GPU and TPU is not just flops-to-flops.

Cool stuff! Do you have any thoughts about how this compares to https://github.com/fabianlindfors/reshape?

Luiz called large meetings "all-heads" instead of "all-hands" meetings. "Why? Because we're not sailors!"

I never got to interact with him directly but he was a leader worth following. This is a sad day.

Ahh this is a good idea, my friend in fact has been asking me to add it forever. I've been hesitating since the data seems so sparse but I think you're both right. Thank you for the feedback!

A website making trail data from OpenStreetMap easily browsable: https://trailcatalog.org/

Myself and a friend have been working on this for quite a while. It's frustrating because it has come so far (showing trail data worldwide interactively on a tiny budget is challenging), and yet it is so far from being something super useful like AllTrails due to low data quality and a lack of relevant features (photos, reviews, etc etc.)

Because they've limited the dynamism of Python so you can't do as much crazy runtime nonsense, their (vaporware) tech stack is able to (theoretically) compile more of your program into GPU kernels.

I built a cross of React + Wiz (a fantastic frontend framework at Google): https://github.com/aschleck/trailcatalog/tree/main/js/corgi . Totally irresponsible and probably full of bugs, but I was so tired of writing business logic in the same place as my view logic with React and now I'm free of it!

Atomic AI | Software Engineering | onsite South San Francisco, CA | Full-time

jobs: https://atomic.ai/careers#opportunities, company: https://atomic.ai

Atomic AI is fusing cutting-edge machine learning and structural biology to unlock RNA drug discovery.

We're looking for sharp and thoughtful engineers excited to help us design therapeutics to treat untreatable disease. There are two positions currently open.

* Senior Software Engineer, Data - design and build solutions with our in-house experimental biology team to analyze, store, and serve experimental data and enable new analysis workflows (https://boards.greenhouse.io/atomai/jobs/4726839004)

* (junior - mid-level) Software Engineer, Infrastructure - integrate and build broad-spanning systems to create a holistic research platform (https://boards.greenhouse.io/atomai/jobs/4531035004)

I'm the hiring manager for both, please apply via the links above or email me at april@atomic.ai if you have any questions!

I wrote this post to try and give candidates more insight into what it feels like to be a technical coding interviewer, what we're looking for, and in the hope that more candidates might see the interview as a conversation (versus simply a written test.) I'd be very curious to hear what others think.

It's worth noting that Google is also working on JAX these days and that's picking up steam, and one reason is because its API is super clean.

(chiming in here as a founding engineer at Atomic)

So I spent more than 8 years as a SWE at Google, and now work here with both experimental biologists and machine learning scientists. And yes, a lot of the concerns mentioned in this thread are also things I have had anxiety about.

Most obvious to me, being a software engineer at Google felt like being the center of the universe. Coming here, the focus is the scientific research. And yes, the scientists all managed to complete their PhDs so they don't necessarily need me to unblock them every second of their day. But contrary to my expectations, this has been remarkably freeing. I think one particularly important part of our company that makes this work is that, even on the science side, we're multidisciplinary (at a high level, emphasizing both experimental biology and ML.) And so engineering feeling like another arm of that multi-discipline nature is fairly... natural.

The reason I feel it's freeing, and the reason I enjoy working here, is also the greatest challenge. Because the scientists are focused on the science, because they respect me and trust me to figure it out, and because they aren't constantly blocked by me, my job is mostly about dreaming extremely expansively about what I can do to reduce toil and make the scientists more productive. Of course they have feedback and input, but how I use my time and what I build is ultimately my decision because I am the engineer. And I have been able to do some things I am very proud of, like rolling out Bazel and Kubernetes and finding ways to seamlessly bring them into the cloud (we're even multi-cloud now without them even noticing!) On the other hand, it's very challenging because when you work on a product, say Google Photos, as a SWE, you always have some direct tether to the product ("what should we build next? ahhhh, well I guess we could just embed stable difficusion and a million people would immediately play with it".) At Atomic, my tether is very ambiguous. If I do my job successfully, they'll be able to do research more quickly (? effectively?), and eventually we'll be able to produce a therapeutic that hopefully changes the world. Identifying what I can do today to speed up that far outcome in the future is very challenging, but it is a far more interesting challenge than gluing some pre-existing software into my UI or running A/B tests to turn a red button blue.

If, like me, you enjoy being given ownership over incredibly ambiguous problems, please do reach out!

This role focuses on directly partnering with the biologists: https://boards.greenhouse.io/atomai/jobs/4726839004

This role is expansive cloud infra: https://boards.greenhouse.io/atomai/jobs/4531035004

And this role is directly partnering with the ML scientists: https://boards.greenhouse.io/atomai/jobs/4191285004

This is sort of a confusing article because it assumes the premise of "you have a fixed hardware profile" and then argues within that context ("Most scientific computing runs on queues. These queues can be months long for the biggest supercomputers".) Of course if you're getting 100% utilization then you'll find better raw pricing (and this article conveniently leaves out staffing costs), but this model misses one of the most powerful parts of cloud providers: autoscaling. Why would you want to waste scientist time by making them wait in a queue when you can just instead autoscale as high as needed? Giving scientists a tight iteration loop will likely be the biggest cost reduction and also the biggest benefit. And if you're doing that on prem then you need to provision for the peak load, which drives your utilization down and makes on prem far less cost effective.

The permissions management (user/group but with cloud roles) is what I miss most. NFS style approaches assume that I will give every user an unprivileged user on their local machine and deny root access so they can't just create a user with a different ID. S3 style ACLs per-object are just very confusing and no one is going to set those. S3 access policies per role are obviously not a solution because that requires centralizing lists of what the permissions are. I just want to be able to assign a chmod and a chown on a file for a cloud user and group... And don't forget about how fantastic the group tooling was.

I also miss being able to write files with /ttl=72h/ anywhere in the path and knowing they will be cleaned up automatically without me doing any more work. In contrast S3 just rots. Need to find an intern to write an SQS pipeline to schedule file deletions :)

Bazel works well when you're in Google's situation: many teams working on separate but connected projects, in multiple languages, infinite money to fund infrastructure teams and near constant migrations, a mandate that all code (*approximately) must be developed using it, and a culture of vendoring every dependency. Meanwhile the typical OSS project is a group of folks working on a monolith in one or two languages, and the predominant development culture avoids vendoring (for better and worse.)

I'm a strong apologist for Bazel, but I can absolutely see why projects avoid it. The more you deviate from what Google does the more paper cuts you'll get, but the less work you'll have to do. Bazel is great but it's certainly not a clear win in the general case.

You're right that using canvas/WebGL will be much faster. It seems the embeddable Google Maps JS library recently moved in this direction, though it appears they kept the layers of stacked divs (almost certainly for backwards compatibility with existing users who have HTML layers that need to be kept in sync.)

But something fun to notice with Felt is that the rendering library (LeafletJS) is using transform3d to do 2d translations (instead of just using translate), so you may wonder "why?" At least in Chrome (and it seems in Safari) if you use transform3d the browser is more likely to keep the div in its own layer. This will reduce a lot of the paint/compositing time, and make the frame rate dramatically better. Of course in this case the micro-optimizations are irrelevant in comparison to Felt's JS performance problems (which on my machine appears to be due to projecting 2000 points from lat/lng space to pixel space every frame.) Choosing to project points every frame is super confusing because the whole point of the translate/transform3d optimization is to avoid having to recalculate pixel space during the latency-sensitive pan interaction. Odd.

Sure! My email is in my profile, not sure how to contact you.

Unfortunately I can't share code at this time but I just described the archiving in more detail in a sibling comment.

The Python interpreter is also quite simple. There's several ways you can do it, but the simplest thing to imagine is if you make a launcher.py script that just invokes Bash as a foreground subprocess. The pstree is kind of funky (bash -> python -> bash), but inside that shell PYTHONPATH will be set approximately correctly. There are reasons to prefer an approach that works with sourcing (eg so you can set PS1), but it's a little harder to describe. You can do some acrobatics to make runfiles (mostly) work, and my recollection is that PATH mostly works but that may require some more work. We do the same thing for Jupyterlab and IPython.

Sure! So the overall goal is to prebuild a runfiles tree containing all the external dependencies into a Docker container, and then when the user wants to run something we build a runfiles tree with all the non-external code. Then in the cluster we want to extract the user's runfiles tree on top of the prebuilt runfiles tree, and then execute the user's code.

* I have an archive Starlark function that I use for both this and containers. It sets up a folder structure similar to <target>.runfiles with everything symlinked to the actual location, then it tars the whole thing following symlinks. It has a parameter to include files that start with external/ or not.

* This archive function is used by my Bazel container rules, so I simply made a runner.py target that depends on every possible external Python dependency and made a Docker image with it.

* I then made a Bazel rule that uses the archive function to archive a given executable without external/ and uploads it to a shared location.

* At runtime runner.py is given the location as an argument, downloads it, extracts it, and then execv's it.

I really can't emphasize enough how much I love using Bazel. The ability to tell a less technical user "just run `bazel run //amazing/server`" regardless of language and know that everything will magically work (toolchain installation, future toolchain upgrades, incremental rebuilds) is really freeing. The actions graph with rules and aspects is quite powerful, so you can do things like add Java nullability checks or Python type checking remarkably easily. Recently I put together a simple build rule that strips external dependencies, archives the rest, and uploads it to a cache. Then we can easily run that archive against a pre-built container (which contains the external dependencies) in our cluster, enabling a very fast ML iteration loop on beefy cluster machines. I've also done a lot of work to enable middle-ground environments, so my users can run Python scripts like they're used to (`python script.py`) while inside of a Bazel environment, which makes it easy for them to develop quickly and then create a BUILD file when they're ready.

The major downside I've experienced is that any time you're trying to do something in a less-than-Bazel way (for example relying on binaries built outside of Bazel) things can get really hairy. My containers often need various things from apt repositories, so I had to give up on rules_docker and made my own rules for Podman. I think you need someone who understands aspects and rules before adopting it, or else the sharp edges of Bazel will keep cutting you until you drop it.

Atomic AI | Senior Software Engineer | ONSITE, South San Francisco CA | Full-Time

Atomic AI, Inc. (https://atomic.ai) is a well-funded, early-stage biotech company transforming the rational design of molecules and medicines through the cutting-edge fusion of artificial intelligence and structural biology. Atomic’s unique R&D platform, based on our research featured on the cover of Science (https://doi.org/10.1126/science.abe5650), provides new strategies to treat or cure previously undruggable diseases by targeting RNA structure. We are an interdisciplinary team working across computational and experimental biology and believe that our strongest asset is our people.

I am hiring a generalist Senior Software Engineer interested in truly full-stack work: from AWS resources, Kubernetes, building classical and supporting ML pipelines, all the way to client-side frontend development (though if you're in bio or ML check out the roles posted at https://boards.greenhouse.io/atomai! We haven't yet posted this SWE role.) This Senior Software Engineer role is fairly unique, we're in the early stages of trying to put together a modern cloud environment to support researchers and scientists. One of my current problems is figuring out how to build a secure environment while also giving folks the flexibility to iterate quickly on their code, maybe you know the solution :)

If you enjoy taking end-to-end ownership of problems in ambiguous areas, thinking about requirements, designing solutions, and implementing them, please reach out! april@atomic.ai