HNHacker News
TopNewBestAskShowJobs

jsolson

2,421 karma · joined February 21, 2007

Google principal engineer based out of Seattle. I am the technical lead for delivering the next generations of AI GPU Supercomputers to Google Cloud. Previously, I worked on the hypervisor that powers Google Compute Engine.

jonolson at google dot com

Opinions are my own, not those of my employer.

submissionscomments
jsolson··on Andromeda 2.1 reduces GCP’s intra-zone latency by 40%
They're different approaches to solving the same problem (improved throughput with lower latency and jitter). The major thing they have in common is that they both dedicate hardware to the problem.

With respect to AWS, in the historical "enhanced networking" case Amazon dedicated hardware by offering SR-IOV capable NICs. SR-IOV is a well understood and effective technique for approaching bare metal performance for virtualized environments, but it tends to lock you into a particular vendor, if not specific model, of hardware. I gather ENA does something a bit different, but I don't know the details.

In Google's case, we dedicate hardware to the Andromeda switch in the form of processor cores (the "SDN" block in the linked post). This allows us to be flexible in terms of NIC hardware while presenting a uniform virtual device to guests, in addition to simplifying universal rollout of new networking features to all zones/instance types.

Both approaches have tradeoffs, although I think even with ENA AWS hits ~70µs typical round-trip-times while GCE gets down to ~40µs. Amazon's largest VMs in some families do advertise higher bandwidth than GCE does currently.

(I was the tech lead for the hypervisor side of this launch — Jake, the post's author, leads the fast-path team for the Andromeda software switch)

jsolson··on Andromeda 2.1 reduces GCP’s intra-zone latency by 40%
Yes, any kernel >= 3.8 includes the relevant offload features. The improvements here primarily come from reduced overhead getting packets out of the VM (not mentioned in the post is that Andromeda 2.1 also eliminates VM exits when packets are sent and, where supported by Intel, when interrupts are delivered).

edit: Realized you might have meant Linux VMs running outside of GCE -- the improvements here are fairly GCE-specific, although as wmf points out, vhost is a similar technology in the open source world. Performance specifics down at this level (tens of microseconds and below) tend to be hardware dependent.

jsolson··on Andromeda 2.1 reduces GCP’s intra-zone latency by 40%
Yes, this is enabled for all VMs running on Compute Engine (which includes GKE VMs), however the in-guest iptables &c. bits add non-trivial overhead (I don't have numbers handy, apologies).
jsolson··on Andromeda 2.1 reduces GCP’s intra-zone latency by 40%
At the time Andromeda was originally introduced it took a fairly recent Linux kernel to get support for multi-queue networking and offloads with virtio-net. Today anything even moderately recent has support baked in -- specifically Linux 3.8 and above include multi-queue support (as well as the offloads we support).

In terms of specific offloads, the big ones are TCP segmentation offload (TSO) and TCP large receive offload (LRO). These substantially reduce the compute burden on the guest. Less impactful (although still important) are checksum calculation and verification offload.

(I was the tech lead for the hypervisor side of this launch — Jake, the post's author, leads the fast-path team for the Andromeda software switch)

jsolson··on Apple Reports Fourth Quarter Results
Wait, what about an escape key for EMACS?

My heavy emacs use is the reason I didn't mind getting the touch bar. It's vim users I'd expect to be annoyed...

jsolson··on Robert Blakeley, Designer of Fallout Shelter Sign, Dies at 95
I had to search to see what you meant by "magenta yellow". My immediate assocation was with the two of them having identical color schemes (although also both being immediately recognizable). I haven't seen the actual radiation symbol often in practice other than in medical facilities or on trucks/trains.

Faded fallout shelter signs were ubiquitous on buildings in my childhood (the 80s), so in that regard there's little room for confusion.

jsolson··on Introducing Amazon EC2 P3 Instances
Out of professional curiosity, what are you looking for from ENA?

(I'm an engineer on Google Compute Engine with a deep interest in customer networking use stories, particularly heavy utilization customers, even if they're not my customers :)

jsolson··on How Seattle Got More People to Ride the Bus
Hey Ronnie -- long time! We should catch up -- drop me a line (personal email is hn username at gmail).
jsolson··on Danish numbers
This is fantastic. I'm an American who's been learning Danish casually, and as a native English speaker the pronunciation is... challenging. Even a year or so in, developing an ear for it is still an ongoing process.
jsolson··on How Seattle Got More People to Ride the Bus
It tends to be annoyingly cool when it rains for about six months out of the year. Take today for example -- 50 and raining. It won't kill you if you're otherwise healthy, but being caught out in it without proper attire feels "shitty". Good rain gear that you can wear every day for months doesn't come cheap, either.
jsolson··on How Seattle Got More People to Ride the Bus
I would describe the bus drivers in Seattle as assertive. They make their own way if nobody yields.
jsolson··on The Family Making Billions from the Opioid Crisis
Yes, but it's tied up in a bunch of other bits like not eating pork and worshipping a particular deity.

The secular quip raises the same questions about human motivations without the other baggage.

jsolson··on “How Amazon Took Seattle's Soul”
Wellllll, I can't edit it anymore, but yeah, got that totally wrong (original source was a real estate developer who's involved with a couple of the projects).

I suspect he took "about 60% more than any other city" and radically misinterpreted it (and I went and parroted him -- oops). Original conversation was right around when this article was published: https://www.seattletimes.com/business/real-estate/seattle-ha...

Apologies for the error!

jsolson··on “How Amazon Took Seattle's Soul”
Seattle currently has more than 50% of the cranes nation wide.

That said, they're almost universally building apartments (exceptions being Insignia, LUMA, and Nexus, that I know of).

jsolson··on Pixel Visual Core: Google’s first custom-designed co-processor
You're begging the question here. Apple's acquisition of PA Semi filled a void. You assume a similar void.
jsolson··on A Little Story About the `yes` Unix Command
Ah, so, this raises an interesting question: what do people use yes for?

One answer (a bad answer) is "generating artificial load".

This makes the I/O strategy oddly relevant -- different strategies result in different numbers of syscalls and (potentially) kernel lock acquisitions.

Couple this with someone trying to benchmark something's behavior with an antagonist load, and the story gets downright painful to contemplate in terms of confusing results.

Source: in my younger years, I did this. In more recent years, I've seen other engineers do it.

jsolson··on A Closer Look at the $13B Premium Eyewear Market
I love Prodesign, but yeah they're not exactly competing on price.

Also a big fan of Etnia Barcelona -- they're in a similar price bracket (as far as I know they're independent, but I'll admit to not having researched it)

jsolson··on Compute Engine machine types with up to 96 vCPUs and 624GB of memory
Preemptible VMs exist in part to avoid leaving capacity sitting idle.

We hate idle resources. :)

jsolson··on Coffee Lake Review: Initial Numbers on the Core i7-8700K and Core i5-8400
Yes.

At least for their high-end server parts, Intel has docs on how many turbo bins you get. AVX can also slow things down all on its own (it reduces the available turbo). A co-worker of mine is (hoping to) speak at KVM forum about this as relates virtualization.

Ah, found some docs for the Core i5: https://www.intel.com/content/www/us/en/support/processors/0...

There's a nicer version of these somewhere with graphs, but I'm on my phone and it's early :)

jsolson··on Ask HN: Where do I get started on ASICs, FPGA, RTL, Verilog et. al?
Sure, but if really want to avoid trusting trust (and you're of the mind to build your own hardware), FPGAs and µcs offer a lot of room for snooping.

Given the GPs suggested use, it seemed trusting trust was not on the table.

Certainly even a tiny FPGA can fit pretty naïve versions of common crypto primitives, as can any modern micro-controller. Assuming you only need to do a handful of ops for whatever you're looking to assert/verify, that is by far simpler than building a gate-level representation :)

jsolson··on Ask HN: Where do I get started on ASICs, FPGA, RTL, Verilog et. al?
Not for an ASIC without spending a LOT on tooling, and really $10k is awfully optimistic even if you had all of that tooling (I probably should've just said tens of thousands).

For <100k, yes, you can absolutely do a small run in that range.

Honestly, you might be better off just buying functional ICs (multi-gate chips, flip flops, shift registers, muxes, etc.) and making a PCB, though. Most crypto stuff is small enough that you can do a slow/iterative solution in fairly small gate counts plus a little SRAM.

jsolson··on Ask HN: Where do I get started on ASICs, FPGA, RTL, Verilog et. al?
There are basically two companies in this business, Altera (Intel) and Xilinx.

I would not worry about vendor lock-in for now -- there are some quite affordable dev boards (like the Arty (http://store.digilentinc.com/arty-a7-artix-7-fpga-developmen...) I've mentioned elsewhere), and no matter what you pick there's a ton of tooling. The concepts from the tools will translate between vendors, though, even if the commands and exact flows change.

jsolson··on Ask HN: Where do I get started on ASICs, FPGA, RTL, Verilog et. al?
I'd start with learning a hardware description language and describing some hardware. Get started with Verilog itself. I'm a fan of the [Embedded Micro tutorials](https://embeddedmicro.com/tutorials/mojo) -- see the links under Verilog Tutorials on the left (they're also building their own HDL, which unless you own a Mojo board isn't likely of interest). Install Icarus Verilog and run through the tutorials making sure you can build things that compile. Once you get to test benches, install Gtkwave and look at how your hardware behaves over time.

You can think of "IP cores" as bundled up (often encrypted/obfuscated) chunks of Verilog or VHDL that you can license/purchase. Modern tools for FPGAs and ASICs allow integrating these (often visually) by tying wires together -- in practice you can typically also just write some Verilog to do this (this will be obvious if you play around with an HDL enough to get to modular design).

Just writing and simulating some Verilog doesn't really give you an appreciation for hardware, though, particularly as Verilog can be not-particularly-neatly divided into things that can be synthesized and things that can't, which means it's possible to write Verilog that (seems to) simulate just fine but gets optimized away into nothing when you try to put it on an FPGA (usually because you got some reset or clocking condition wrong, in my experience). For this I recommend buying an FPGA board and playing with it. There are several cheap options out there -- I'm a fan of the [Arty](http://store.digilentinc.com/arty-a7-artix-7-fpga-developmen...) series from Digilent. These will let you play with non-trivial designs (including small processors), and they've got lots of peripherals, roughly Arduino-style.

If you get that far, you'll have discovered that's a lot of tooling, and the tooling has a lot of options, and there's a lot that it does during synthesis and implementation that's not at all obvious. Googling around for each of the phases in the log file helps a lot here, but given what your stated interest is, you might be interested in the [VLSI: Logic to Layout](https://www.coursera.org/learn/vlsi-cad-logic) course series on Coursera. This talks about all of the logic analysis/optimization those tools are doing, and then in the second course discusses how that translates into laying out actual hardware.

Once you've covered that ground it becomes a lot easier to talk about FPGAs versus ASICs and what does/doesn't apply to each of them (FPGAs are more like EEPROM arrays than gate arrays, and for standard-cell approaches, ASICs look suspiciously like typesetting with gates you'd recognize from an undergrad intro-ECE class and then figuring out how to wire all of the right inputs to all of the right outputs).

Worth noting: getting into ASICs as a hobby is prohibitively expensive. The tooling that most foundries require starts in the tens-of-thousands-per-seat range and goes up from there (although if anyone knows a fab that will accept netlists generated by qflow I'd love to find out about it). An actual prototype ASIC run once you've gotten to packaging, etc. will be in the thousands to tens of thousands at large (>120nm) process sizes.

jsolson··on Abseil: C++ Common Libraries
Yes.

It's a little complicated as I did it for a personal project, but I'm also pulling it into Google for my 20% project. It is by FAR easier/faster for me to release under Google's copyright, but there's a bit of "paperwork" I need to go through to get it staged and approved for the Google GitHub.

Since it seems there's interest I'll take a stab at that on Monday. It usually takes a week or two, but no promises. I don't anticipate any blocking issues, though, as the bazel bits don't expose anything about Google.

One caveat -- as released it won't quite be bitwise hermetic, as it depends on an external Xilinx toolchain, and Xilinx does non-hermetic things to their bits that are awkward to work around (you can, but my code to do that is a much less mature shell script full of dd and cat commands to rip apart bitstreams and glue them back together -- I've since found open source tooling for accomplishing the same task, but haven't adopted it yet).

Drop me an email (addr in profile) and I'll tell you when it's up.

jsolson··on Extending per-second billing in Google Cloud
Certainly!

I love what I do, and I love talking about it to anyone who will listen. I find Google's infrastructure is fantastically exciting -- sometimes too exciting, rarely boring. Most of the time I wish I could share more. Always happy to hear someone has found what I can share interesting!

jsolson··on A High-End Mover Dishes on Truckstop Hierarchy, Rich People, and Moby Dick
If you want to propose punitive legal remedies for the actions of others, yes, you should do your research.
jsolson··on Abseil: C++ Common Libraries
Aren't embedded systems already there? What is a firmware blob if not a monolithic static build artifact? Bazel is perfect for this. I recently taught it how to build boot images for Xilinx FPGAs including a bitstream, first stage bootloader, kernel, and initial ramdisk, all hermetically built and bottled up into a boot.bin. I can reproduce my firmware from source control and get out the same bits. How neat is that?
jsolson··on Extending per-second billing in Google Cloud
Today's control plane is fairly smart about which routes are required for a given config change event (there's a lot one could speculate about here, especially around the word "required") -- fwiw, I also don't work on the control plane -- I have spent a lot of time on the GCE hypervisor network dataplane. So a fair amount of hand waving follows -- just assume details are missing because I don't know what they are :)

We aim for global convergence of network state as sort of an ongoing goal, but it's a distributed system with failure domain isolation, so that goal is necessarily flexible. There are different rates of convergence, and Solomon is certainly right that routes to first party services are some of the easiest to converge. Internet connectivity is to a certain extent the hardest thing to converge, as in our premium tier (which up until recently was our only tier) we aim to keep data on Google's network for as much of its journey as possible. At the extreme, this means a lot of edge nodes learning that a given external IP belongs to your VM.

Part of it is also just the mundane business of reconciling what a given configuration event means and propagating that to interested parties. With respect to your firewall example, it's really just another config change event with some set of implications for routes that are added or removed.

Anyway, that's my hand wavy explanation. Hopefully it's helpful!

jsolson··on A High-End Mover Dishes on Truckstop Hierarchy, Rich People, and Moby Dick
> I'd love to do that. Unfortunately, even at a slow speed, it can be _challenging_ (to say the least) to detect a safe turnout area (e.g. in winding roads on the way to Tahoe) soon enough that one can actually use it.

I hate to say this, but... slow down more if you see a potential candidate approaching.

I'll hate you for it right up until I see you pull aside, once I realize what's up you'll probably even get a courtesy wave!

jsolson··on A High-End Mover Dishes on Truckstop Hierarchy, Rich People, and Moby Dick
Sure, sorry, I wasn't aiming to criticize the driver in the overall post -- my comment was solely aimed at the drivers described in the quoted passage from the prior comment :)

For most vehicles, on most roads, opportunities to allow others to pass safely appear with reasonable regularity. Despite this law, in my experience most slow drivers choose to entirely ignore those opportunities.

← PreviousPage 9 of 24Next →