HNHacker News
TopNewBestAskShowJobs

jsolson

2,421 karma · joined February 21, 2007

Google principal engineer based out of Seattle. I am the technical lead for delivering the next generations of AI GPU Supercomputers to Google Cloud. Previously, I worked on the hypervisor that powers Google Compute Engine.

jonolson at google dot com

Opinions are my own, not those of my employer.

submissionscomments
jsolson··on Tesla FSD data is getting worse, according to beta tester self-reports
If it's anything like the original Autopilot was: yes.

I had one of the first Model S vehicles that was autopilot-capable. Early enough that autopilot itself was added later. The early versions were... intense. Not just in the "if you take your hands of the wheel it might try to kill you" intense, but also in the "even using this as adaptive cruise with lane-keeping, sometimes it will suddenly try to veer off the road and murder you" sense. Even when working "as intended" it would routinely dip into exit ramps if you were in the right lane. As a result, I didn't use it all that often, but over not a lot of time it improved pretty dramatically.

At this point my wife and I are on newer Model 3s, and our experience with autopilot (not FSD) as a frequently-used driver assist has been... uneventful? Given the _original_ autopilot experience, though, neither of us is particularly eager to try out FSD yet. Autopilot strikes a good balance for us in terms of being a useful second pair of eyes and hands while unambiguously requiring us to drive the damn car.

jsolson··on Waymo's collision avoidance testing
If you're buying puts you have bounded risk (the amount you invested), albeit with a more cliff-like risk profile than other strategies.
jsolson··on The factory that only builds white Toyota Land Cruisers
> My car, for instance, turns off the engine when stopped, that can't be good for the engine and battery.

I understand the intuition behind this, but one could also see a place where starting the engine while it's still hot is better than idling (less wear, fewer opportunities for deposits to build up from wasted combustion), and where the impact to the battery is minimal as the hot engine makes it easier to turn over for the very few cycles it takes to get it going again.

jsolson··on Mcmaster.com is the best e-commerce site I've ever used
As someone who software engineers and builds physical shit from time to time, yes, but money helps making this true.
jsolson··on AWS vs. GCP reliability is wildly different
Today I learned! I'll admit I didn't know this functionality existed, and I've instead had used instances.insert following by querying the VM resource.

This is nicer!

jsolson··on AWS vs. GCP reliability is wildly different
As far as I know, the `instances.insert` API only allows individual VMs, although the CLI can issue a bulk set of API calls[0], and MIGs (see below) allow you to request many identical VMs with a single API call if that's for some reason important.

You can also batch API calls[1], which also gives you a response for each VM in the batch while allowing for a single HTTP request/response.

That said, if you want to create a set of effectively identical VMs all matching a template (i.e., cattle not pets), though, or you want to issue a single API call, we'd generally point you to managed instance groups[2] (which can be manually or automatically scaled up or down) wherein you supply an instance template and an instance count. The MIG is named (like nearly all GCP resources), as are the instances, with a name derived from the MIG name. After creation you can also have the group abandon the instances and then delete the group if you really wanted a bunch of unmanaged VMs created through a single API call, although I'll admit I can't think of a use-case for this (the abandon API is generally intended for pulling VMs out of a group for debugging purposes or similar).

For cases where for whatever reason you don't want a MIG (e.g., because your VMs don't share a common template). You can still group those together for monitoring purposes[3], although it's an after-creation operation.

The MIG approach sets a _goal_ for the instance count and will attempt to achieve (and maintain) that goal even in the face of limited machine stock, hardware failures, etc. The top-level API will reject (stock-out) in the event that we're out of capacity, or in the batch/bulk case will start rejecting once we run out of capacity. I don't know how AWS's RunInstances behaves if it can only partially fulfill a request in a given zone.

[0]: https://cloud.google.com/compute/docs/instances/multiple/cre...

[1]: https://cloud.google.com/compute/docs/api/how-tos/batch

[2]: https://cloud.google.com/compute/docs/instance-groups

[3]: https://cloud.google.com/compute/docs/instance-groups/creati...

jsolson··on AWS vs. GCP reliability is wildly different
Sure, except I think that at a macro level we got it more right than AWS, despite some choices that I believe we'd make differently today.
jsolson··on AWS vs. GCP reliability is wildly different
You're entitled to that takeaway, but I disagree. I believe GCP's tendency to use caller-supplied names for resources is one of the single best features of the platform, particularly when compared against AWS's random hex identifiers.

Note that whether this creates collisions is entirely under the customer's control. There's no requirement for global uniqueness, just a requirement that you not try to create two VMs with the same name in the same project in the same zone.

jsolson··on AWS vs. GCP reliability is wildly different
> When try to create the same resource twice, the second should report success instead of failing.

Before I quibble with the idempotency point: I agree with this, entirely, but it is what it is and a lot of software has been written against the current behavior. So I'll cite Hyrum's law here: https://www.hyrumslaw.com/

> GCP control plane is generally not idempotent.

The GCE API occupies an odd space here, imo. The resource being created is, in practice, an operation to cause the named VM to exist. The operation has its own name, but the name of the VM in the insert operation is the name of the ultimate resource.

Net, the API is idempotent at a macro level in terms of the end-to-end creation or deletion of uniquely named resources. Which is a long winded way of saying that you're right, but that from a practical perspective it accomplishes enough of the goals of a truly idempotent API to be _useful_ for avoiding the same things that the AWS mechanism avoids: creation of unexpected duplicate VMs.

The more "modern" way to do this would be to have a truly idempotent description of the target state of the actual resource with a separate resource for the current live state, but we live with the sum of our past choices.

jsolson··on AWS vs. GCP reliability is wildly different
I mean, it's kinda nice to know that if you reissue a request for an instance that could costs thousands of dollars per month due to a network glitch that you won't accidentally create two of them?

More practically, though, the instance name here is literally the name of the instance as it appears in the RESTful URL used for future queries about it. The 409 here is rejecting an attempt to create the same explicitly named resource twice.

jsolson··on AWS vs. GCP reliability is wildly different
This is mostly not true in cases where resources are actually available (and in GCE if they're not the API rejects the VM outright, in general). To the extent that it is true for Borg when the job schedules immediately, it's largely due to package (~container layers, ish) loading. This is less relevant today (because reasons), and also mostly doesn't apply to GCE as the relevant packages are almost universally proactively made available on relevant hosts.

The origin for the info that jobs take "minutes" likely involves jobs that were pending available resources. This is a valid state in Borg, but GCE has additional admission control mechanisms aimed at avoiding extended residency in pending.

As dekhn notes, there are many factors that contribute to VM startup time. GPUs are their own variety of special (and, yes, sometimes slow), with factors that mostly don't apply to more pedestrian VM shapes.

jsolson··on AWS vs. GCP reliability is wildly different
Idempotency.

You're inserting a VM with a specific name. If you try to create the same resource twice, the GCE control plane reports that as a conflict.

What they're doing here would be roughly equivalent to supplying the time to the AWS RunInstances API as an idempotency token.

(I work on GCE, and asked an industry friend at AWS about how they guarantee idempotency for RunInstances).

jsolson··on Patagonia founder gives away the company
Depends on the trust?

If your ideas are abstract and roughly "climate isn't fucked" they seem pretty perennial. They're also open to interpretation.

That said, sure, lots of periods in history would've produced trusts that are truly appalling. Something to be addressed case by case, for now, though, and collective social will can always disolve what is ultimately a social contract.

jsolson··on Why no Roman industrial revolution?
> Without demand, there is no supply.

An economics professor of mine put it as "markets are created by demand, not supply." That was nearly 20 years ago, but it's really stuck with me.

As the linked article notes, the precursors for technological innovations were present in many places, but many of those places lacked one or more components of demand flow to finance the required innovation. Only after demand for industrial innovation is present do gaps in the supply chain come into play.

jsolson··on Things I Won't Work With: Azidoazide Azides, More or Less (2013)
The video makes it quite clear that he has a degree, soooo, clearly he's not doing stupid shit on YouTube.
jsolson··on AWS Just Walk Out Technology
I can't really fathom how you can include "scan, scan, scan, weight, scan, scan, scan, checkout, touch my credit card, go" in your comment and consider that comparable to "leave." For routine grocery shopping the technology is incredible.

That said, I'll grant that it's also creepy AF. The bigger issue for me, though, is that Amazon is so fucking awful at physical retail inventory management (both the selection of items that should be there and the selection of items actually available for purchase) that the Go Grocery in Seattle went from being "this is how I buy groceries" to "there is zero chance they'll have most, let alone all, of the items on my incredibly boring grocery list."

Amazon still wins, though... I end up walking the other direction and going to Whole Foods...

jsolson··on Severance is the workplace thriller we’ve needed
> You need an Apple device to watch Apple TV+

Watched it on a Sony TV running Android through Apple's official app.

jsolson··on Ask HN: Has anyone here worked on the Windows kernel?
When your job is to build infrastructure, "in a data-center" does not necessarily mean it's a production server :)
jsolson··on Ask HN: Has anyone here worked on the Windows kernel?
In at least one instance of breaking my tools with my tools, the machines in question were in a data-center. Ordinarily one might be able to grab debugging output over an NC-SI link, but that only works if the breakage doesn't also unexpectedly crash your NIC firmware...
jsolson··on Ask HN: Has anyone here worked on the Windows kernel?
I take it the Firecrakcer folks haven't built out support for KVM_GUESTDBG_SINGLESTEP yet :)
jsolson··on Ask HN: Has anyone here worked on the Windows kernel?
I think I can count the number of kernel changes I've submitted on one hand, but I work on core virtualization that involves a lot of pretending to be hardware and (these days) a lot of poking directly at hardware registers.

I would say James Mickens sums things up nicely in "The Night Watch[0]." For example, you mention debugging with logs and metrics -- this snippet came to mind:

     “Yeah, that sounds bad. Have you checked the log files for
     errors?” I said, “Indeed, I would do that if I hadn’t broken every
     component that a logging system needs to log data. I have a
     network file system, and I have broken the network, and I have
     broken the file system, and my machines crash when I make
     eye contact with them. I HAVE NO TOOLS BECAUSE I’VE
     DESTROYED MY TOOLS WITH MY TOOLS. My only logging
     option is to hire monks to transcribe the subjective experience
     of watching my machines die as I weep tears of blood.”
Mind you, I absolutely _love_ working on low-level stuff, and I wouldn't trade the time I get to spend actually doing that for anything. That said, the complexity of modern operating systems, CPU architectures, interconnects, and peripherals creates opportunities for frustration and confusion that honor no bounds of reasonability or decency.

[0]: https://www.usenix.org/system/files/1311_05-08_mickens.pdf

jsolson··on The Path Is Set for PCI-Express 7.0 in 2025
In some settings they already have -- the 3M twinax cable the GP mentions is sold in a multi-pair format where, rather than a crimp-style connector, the ends of the cable are soldered onto PCBs that are part of connector assemblies (from, for example, TE's Sliver line: https://www.te.com/usa-en/products/connectors/pcb-connectors...).

Note that on balance I'll take something like an OSFP assembly (https://www.te.com/usa-en/products/connectors/pluggable-conn...) with a fancy jacket that pulls all of the individual twinax cables together, but it's also substantially more expensive than a simple flat tape-wrapped ribbon.

Other places where you'll see coax cables ganged together inside a jacket: high-speed USB cables. For example, here's a type C cable cross-section: https://twitter.com/tubetimeus/status/1125926941469462528

jsolson··on The Path Is Set for PCI-Express 7.0 in 2025
If it were truly a bus, these signaling rates would be totally impossible. Too many reflections.

In practice it's a bunch of point-to-point links, but even then the path across a PCB is pretty brutal in terms of loss. Cables are (surprisingly to me, initially!) better.

Overall though, sending some number of additional bits for error correction is often cheaper than driving the line harder to reduce the bit error rate.

Put differently: we could manage lower noise, but it would either result in a lower bitrate or (much) higher power. Instead, pick a target goodput and design for power and error correction from both sides.

jsolson··on A blog that is a single executable binary
Why not just use https://pkg.go.dev/embed ?
jsolson··on Intel Virtualization and Apple Silicon
Quibbling a bit with the details here.

> This is not at all specific to ESXi... you can do PCIe and USB passthrough with qemu running on Linux.

This is true.

> The operating system doesn't get in the way.

This isn't, really, at least not qemu does quite a bit of fiddling under the hoot -- without VFIO or legacy KVM pass-through, the operating system very much gets in the way. Linux now provides facilities that allow qemu to pass through devices -- that is, it provides the APIs necessary to move it back out of the way.

I don't know to what extent ESXi looks like pure-kernel setup of passthrough vruss KVM_ASSIGN_DEVICE versus VFIO -- that would be quite interesting.

jsolson··on Lanai, the mystery CPU architecture in LLVM
You should not have posted this, least of all because it's extraordinarily misleading.

We use Lanai elsewhere, and it's in hardware powering Google's server networking networking (inclusive of GCP), but Lanai sadly does not play a substantial role in Andromeda networking.

Feel free to ping me on corp (username in my profile here) to discuss.

jsolson··on Lanai, the mystery CPU architecture in LLVM
Ah, no, the fairly was merely that we omitted details and some things have changed. That said, the paper completely captures the use of Lanai in GCP's networking data plane.
jsolson··on Lanai, the mystery CPU architecture in LLVM
Indeed.
jsolson··on Lanai, the mystery CPU architecture in LLVM
We've been fairly open about how Google Cloud Networking works -- https://www.usenix.org/conference/nsdi18/presentation/dalton

There's also some more recent coverage on IPUs that I don't have a handy link to.

jsolson··on Light mode, Dark mode, and Gen-Z mode?
> Older, wiser and more bitter professionals, fustily defogging their glasses while seated defeatedly at their architect-style slanted drafting desks (with optional standing desk accessory), would oftentimes mutter to themselves, alone at night in their downtown 23rd-floor apartments, lit only by the synthetic warm-LED glow of their ironically chosen "Banker's lamps", a different name for this cultural watershed: "The End-times' Madness." But nobody listened to them anyways, and they didn't much care.

Well, I feel seen, except that it's a Tolomeo desk lamp and I keep a flat desk (with standing desk accessory) as I sometimes use solder in anger.

← PreviousPage 3 of 24Next →