jsolson

Hell, I sometimes forget which end of the Ethernet cable plugs in where.

Piper is awesome, but it's not unreasonable to think of it as Perforce if you designed and built Perforce to scale to handling all of Google in one repo. It has basically the same set of nouns and verbs. citc is the really magical bit, UX wise, as it's like you had a Perforce view set up for our entire repo but never actually had to sync it manually. Just navigate to a file and start hacking.

The paper that corresponds to the talk includes numbers on that (I forget if they're in the video).

http://m.cacm.acm.org/magazines/2016/7/204032-why-google-sto...

(note: I work at Google and absolutely love our SCM and other dev tooling)

So, I've made it from Burlington, Washington to the Canadian border without turning off Autopilot in Friday evening traffic (about 50 miles, including some pleasant winding interstate). As described by Subaru LKA "...can gently guide the vehicle back to the center of its lane if sway is detected". It sounds like it only corrects for small deviations rather than actually completely steering for you? Is Subaru under-selling it in their press release (which would honestly be sort of amazing)?

ioeventfd for PIO exits is still handled in the kernel, but that one is easy since it's a dedicated VMEXIT type.

We do very little that typically requires trapping MMIO, particularly in places that are performance sensitive (VIRTIO Net and VIRTIO SCSI do not, and honestly there's not too much that guests do inside GCE that isn't either disk or networking :).

> I am surprised you don't use ioeventfd though!

We do in some cases, for both networking and storage. Since our devices are (mostly) VIRTIO (of pre-1.0 vintage), we're using it for OUTs into BAR0 (which again of course get their own VMEXIT and don't require emulation).

By and large we try to elide the exits entirely if we can, naturally, although in today's GCE production environment serialized request/response type workloads will see exits on every packet. Streaming workloads fare better, as we do make use of EVENT_IDX and aggressively trying to find more work before advancing the used.avail_idx field.

We do not. That it's not vhost you can infer (correctly) from the topology of our VIRTIO Net device; specifics on that front will have to wait for another day, though :)

Adding to Andy's point here -- if you consider the services that back GCE in terms of the features offered by our virtual networks, virtual storage devices, etc. they necessarily have tight dependencies on "How Google is Built" to hit the performance targets we aim for.

I, personally, am optimistic that we'll be able to do it for the core at some point. Something not mentioned in the original article is that architecturally our VMM allows us to easily change the set of available functionality (virtual devices, backing implementations of those, etc.). So, for example, the fuzzing Andy described as depending on our custom VMM isn't even linked into the VMM run in GCE, nor are custom devices we keep around for testing and development. I'd very much like it if we could layer the VMM in such a way that we could release a useful core that forms the basis of the VMM we consume internally. Today, though, the layering just isn't quite there (among other technical and business reasons, as Andy says), and there are Googley specifics in places that would make even the VMM core (minus devices, etc.) unviable outside of Google.

Yeah, it still gets hit now and then. It should not get hit often in the typical steady state, though, which is why you can punt it to userspace with little performance penalty.

(I work on the custom VMM we run)

> No doubt it's simpler than QEMU but I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable.

There are some folks who have this view, but doing it from scratch also has the advantage that it integrates much more cleanly with Google's global shared codebase. There's a huge body of existing work that I can leverage more or less trivially. This includes things like Google's internal metrics and monitoring framework, our RPC framework, etc. Yes, you could bolt these onto the side of qemu, but qemu is a C codebase and most of Google (including the custom VMM described in the article) is not.

Additionally, when software is built using the same style, tools, and best practices as the rest of Google's codebase, it makes it easy for other engineers in the company to contribute. We benefit from Google-wide code cleanups, *SAN analysis tools, codebase-wide refactorings that make code easier to reason about the correctness of, etc.

Several years ago I think the question would've been a lot more difficult to answer, but today I the advantages of the route taken are unambiguous.

(my team owns the virtual network devices visible to GCE VM and the first chunk of the on-host dataplane, one virtual hardware component of the custom VMM we run :)

When I took it we were encouraged to use Weka for the algorithm implementations themselves. This certainly allowed me (and I'd never so much as touched machine learning prior to taking the class -- I took it on a bit of a lark that wasn't related to my research at all) to focus on understanding the behavior of the algorithms rather than worrying about hacking them together.

I'd agree that Tech has too few TAs for too many students, generally, for its graduate courses, but I don't know that other schools do a better job. A brief survey of the folks around my desk elicited howls of laughter at the notion of useful or accessible TAs in grad school.

> I agree that Isbell is a joy to listen to - he is charismatic, entertaining, and I too enjoyed his anecdotes. However, I felt like you would only get something out of his lectures if you already knew what you were talking about.

I think this assertion is, at best, too strong. A better assertion might be that his lectures depended on coming in with sufficient background.

As I said, I came into the course with no experience with machine learning at all. On the other hand, I did have a fairly strong theoretical computer science, stats, and linear algebra background. I will admit that may have made me blind to things he was simply assuming with respect to educational background that were not actually safe to assume. That said, I still refer back to his primer on information theory (http://www.cc.gatech.edu/~isbell/tutorials/InfoTheory.fm.pdf) when discussing work relying on it, so he certainly made some effort to fill in gaps as he discovered they were common.

> When I think about the quality of the class, I think about how responsive the class is to the individual needs and progress of the student.

For a graduate level course I feel a class clears this bar when it accurately and thoroughly documents the prerequisites. Now, I'm not saying Charles's class necessarily does this. As I said, I came in with a pretty strong background in what turned out to be more than sufficient, but with that background I personally felt his lectures were quite tractable, even assuming complete ignorance of ML itself.

Fair enough, and I should have been a bit clearer in my original post.

In my experience with the MSCS program (nearly ten years ago at this point) the core required classes were mostly well structured and would serve people well continuing onto a PhD or growing their skill set for industry. The core constituted a relatively small chunk of the overall credits required, though, and the elective courses tended to be more along the lines of what I described.

I'm glad to hear that the Analytics program has a more dedicated focus on practical matters. It might be interesting to produce a series of similar (but narrower) curricula that amount to curated collections of CS classes making up degrees in Machine Learning, Systems Programming, etc.

I personally really enjoyed my dartboard-oriented approach to class registration. I learned more than I've never needed to know about approximation algorithms, cryptographic theory, and compilers. Even if much of what I learned there hasn't proven itself directly useful yet, I really enjoyed learning it for learning's sake, and I think I'd have had a hard time picking up some of the gems I pulled out of that since. I also still have a hobby of proving problems NP-complete on demand as a bit of a parlor trick (within the limited scope of problems for which you can apply the small handful of patterns I've burned into my brain over the years :).

I did both my BSCS and MSCS at Georgia Tech. While I have many complaints about the school, the quality of the classes is not one of them, for either the undergrad or grad programs.

That said, with a couple of notable exceptions, the graduate classes are there for PhD students as first and second year background material so they have some starting points for their research. This naturally leads to a format where the semester can effectively be described as a long reading list of papers and lectures to spur discussion on the content of the paper. I was planning on pursuing a PhD when I started into my MS, so this format worked quite well for me at the time. In the years subsequent to that, the grounding from those classes has given me starting points for deep dives into problems I encountered at work[0].

It's interesting that you brought up machine learning. Charles Isbell's Intro ML class was a significant exception to the pattern I described above. In addition to high quality, pre-prepared lectures peppered with entertaining anecdotes, the had high quality projects that worked with pratcial tooling. It was also probably the highlight of my graduate career[1].

[0]: In particular, the material covered in my graduate systems classes has been invaluable for not reinventing the wheel for the thousandth time. The material from the couple compilers classes I took on a whim has been a huge boon when talking about software correctness. I work on the hypervisor underneath GCE. Correctness is near and dear to my heart, but performance is right there with it :)

[1]: For undergrad that dubious honor has to go to Olin Shivers, not only because of his eclectic teaching style, but also because his class completely altered the way I think about problems in computer science. In particular, my mindset shifted to one of models of computation and decomposition of problems into subproblems for which the simplest model could apply. I have an example I'd like to write up, but it's a bit long for a footnote.

In my experience it most often manifests in the form of columns that are constrained not NOT NULL into which you suddenly can't insert, say, a user who has no last name.

I find that number... Odd.

I've spent a fair amount of time pulling together King County Recorder's office data, and I don't believe that number unless they're all doing business as... strings that look an awful lot like names.

I don't agree on the free trial. I think it lets you play around a lot more than the AWS free tier -- as I said in my comment up the thread, go ahead and move the slider to the right and scale up a bit (a little -- the free trial is still quota limited to prevent abuse), see how things perform, slide it back to the left and have only used a tiny fraction of your credit.

I think you get a lot more "kick the tires" flexibility with a moderately large up-front credit.

(affiliation: I'm an engineer on Compute Engine)

Depending on how you use the services the AWS free tier is actually worth less than GCP's $300 free-trial credit: https://cloud.google.com/free-trial/

I will give Amazon that theirs extends out to a year, but summing up the costs of everything included in the free tier if you use all of it, I think it may still come out to less than $300 (or the equivalents on GCP would, anyway). For example, running an f1-micro for a year will cost you a bit under $60. If you add in another for Google Cloud SQL you're up to a total of about $150 over the course of a year. What they offer you in S3 is basically free (<$2 over the course of a year).

It's possible the other services are a better deal than compute and storage, if you have a use for them, but the GCP free trial lets you allocate that $300 however you like. You can scale up more in that 60 days than you can within the AWS free tier. To me, personally, this strikes me as more valuable if I'm trying to sketch out a new product — I'd rather not try to figure out how to fit inside the AWS free tier resource envelope and instead understand how my costs are scaling with the resources I'm using while not being on the hook for those costs (up to the $300 size of the credit, obviously) for the first couple months. Especially to absorb things like shaking out automation -- go ahead, spin up a GKE cluster, scale it out to 5x the size you currently need, run some quick load tests, and then scale it back down 30 minutes later (and only pay for that 30 minutes).

(affiliation: I'm an engineer on Compute Engine)

I work on GCE — when I read that post I also thought the 50% was a bit generous, but your other comment made me curious. I ran some numbers and posted a reply over there: https://news.ycombinator.com/item?id=13078157

tl; dr: Even matching instance types more closely, for on-demand pricing the 50% number held up remarkably well, although I'm worried that I got something wrong with my math on provisioned IOPS pricing, as a 10x difference seems unbelievably high.

Note... I work on GCE. That said, numbers I'm posting here for GCE are from plugging custom instance and volume types into the GCE console on my personal account rather than anything formal (I'm on my personal laptop). While you can nitpick the OP's points, our custom instance types make it possible to do something closer to an apples-to-apples comparison, and by my math below (which may be wrong; the risk of posting at 1am) the 50% number still holds up if you ignore reserved instances, assuming the OP's prices for AWS were correct.

> Does anyone know why GCE doesn't have a eu-west-1a? only b,c,d? I'd be curious to know the story there... not trolling -- just curious.

Just guessing (I've worked on GCE for a bit over three and a half years, but I don't remember the history here) -- I suspect that it was a zone that existed prior to GCE going generally available that was turned down and there was a desire to not re-use the name.

> c4.large has 2 CPUs... n1-standard-1 has 1... comparison on price is strange?

Well, ok, but a custom 2 vCPU instance with 3.75 GB of RAM is $45/month (including a 10 GB PD root volume), so still nearly 50% lower than the c4.large price.

Similarly, if I build a c4.4xlarge equivalent custom instance type (16 vCPUs and 30 GB RAM) it's $358/month, again putting it at just about 50% of the rack rate for a c4.4xlarge.

I'm not sure where the OPs claims were coming from for networking. Certainly you can get 1 Gbps out of relatively small EC2 instance types, although at a virtual NIC level GCE is considerably more generous, providing virtual 2 Gbps/s/vCPU link speeds (so up to 8 Gbps for a 4 vCPU instance versus 750 Mbps for a c4.xlarge). Bandwidth-to-the-edge will be lower. That said, when trying to find an upper bound on bandwidth it's really best to test (regardless of which platform you're evaluating) with multiple streams ideally to multiple peers. Single stream can end up limited by a number of factors including RTT and congestion window scaling bounds as well as getting unlucky in terms of ECMP path selection for multi-path links.

> Local SSD #s: Use PIOPS volumes, tunable size, lower cost, can attach to any instance type.

PIOPS volumes are EBS-SSD, not local, aren't they (looking at https://aws.amazon.com/ebs/pricing/)? Our analog would be PD-SSD. GCE PD (including PD-SSD) IOPS scale with volume size — let's say you needed 20 GB of storage at 10000 IOPS. On EC2 you'd be paying $0.125×20+$0.065×10000 = $652.5. On GCE you'd need to buy a bigger volume, at 30 IOPS/GB for PD-SSD you'd need a ~334 GB volume. However, at $.17/GB you'd get the same 10000 IOPS at $57/month — for this example EC2 is more than 10x as expensive, and if we scale the storage up so you get the same 334 GB on io1 EBS PIOPS SSD it's more than 12x as expensive.

I picked these numbers somewhat arbitrarily; EC2 would've come out looking better for very large volumes with very low performance requirements, but with PD-SSD priced at $.17/GB and io1 volumes priced at $.125/GB + .065×1IOPS = $0.19, io1 volumes are still more expensive even at the lowest IOPS bound (above 1GB they get cheaper, but only below 3 IOPS). General purpose SSD is cheaper than PD-SSD, so I suppose for users who can live without a performance SLA it might be a good choice, although at that point I'd be tempted to benchmark it against our vanilla PD offering.

You can also attach GCE's truly local SSD (as opposed to PD-SSD) to any instance type (with really fantastic IO performance -- up to 680,000 IOPS read perf with multiple volumes), although I'll grant you that the 375 GB granularity is a little... coarse.

> source? details? KVM came out in 2007. Xen came out in 2003.

To be fair, GCE doesn't exactly run on stock KVM (nor, I assume, is EC2 running on stock Xen), and our userland is not qemu. I don't think it's meaningful to try and assess these technologies from the outside[0].

> Steve Yegge quit Google. Famously... on stage...

He didn't, he just switched projects... he's still at Google. He went on to say that Google couldn't do platforms; one of my favorite things about working here is the company's ability to be introspective and recognize when a rant like Steve's has more than a grain of truth to it. Google adapts. I think comparing the GCP APIs to Google's historical API efforts makes it abundantly clear that Google has improved by leaps and bounds at building a cohesive platforms.

[0]: I do lead the team that owns our virtual NIC and a chunk of our host dataplane in addition to prior involvement in broader aspects of our hypervisor architecture, so I'll admit to more than a little bias on this front :)

I should really find a better way to phrase that comment. I wasn't actually there for that long (a bit over two years), but I still regret not leaving sooner. I was hired in as an SDE2, which at the time was probably the right level due to lack of experience with large companies, but I outgrew it pretty quickly. The SDE3 promo process involved a staggering amount of hoop jumping specifically so my manager could put together my packet. My manager was fairly transparent about the checklist nature of the process, but it clearly involved an even more staggering amount of work for him preparing a packet for me.

When I say I should've left sooner it's because for all the effort put into it I ended up taking a substantial pay raise and a level drop when I finally did leave five months later (and went through a much lighter weight promo process shortly after joining my new gig that I would've been able to pursue regardless). It's possible the promo put me in a better negotiating position, but given that I didn't disclose my current compensation during negotiation I can't say that with any confidence.

So, the great irony of the leadership principles was the lip service they were paid at Amazon and how much I miss them since leaving.

They're a seriously good bit of Amazon culture that's been perverted into something you can use to submarine any meeting. If treated honestly, they seem (to me, anyway), like a fantastic set of rules for governing a company's actions. Sadly, it's difficult to align incentives with actually honoring and encouraging those principles :(

> they sell it as a service and operate more servers than google

Citation needed.

Not really.

Principal, maybe, but SDE3 is a position you can be easily hired into with enough experience. As a promo it's a bitch, though, and absolutely not worth it. Jump ships for something level equivalent at a company with better comp and get promoted there.

I say this having stuck it out at Amazon for SDE3 before jumping ship. My only regret is that I stayed too long.

If you've recently set up iOS devices you'll have seen it ask for the passcode for another device before you can access iCloud data on the new one.

When I moved to Seattle I sold my car because everything was walkable :)

Mind you, I was coming from Atlanta, GA.

I have one, but honestly I crossed about a dozen times each way last year for skiing and never spent more than a minute talking to an agent.

I got Nexus because the line to get to the booth is sometimes longer than I'd like (and TSA pre-check is nice).

Note:

GCE - Google Compute Engine. VMs in the Cloud.

GKE - Google Container Engine. Hosted Kubernetes. Runs on top of GCE.

(I work on GCE)

Seems like there's a market for a tiny hypervisor.

It can quite difficult to get a large diesel generator permit for a data center in the Atlanta area due in part to automotive emissions but also (ironically) the high greenhouse gas emissions caused by Hartsfield-Jackson Atlanta International Airport.

Mind you, I'm not saying they don't have one, just that it can be difficult to get -- there can also be limits on the number of hours you can run them.

You might ask some people in the American South about their views on States' Rights.

Sure, but you can get similar things from Google Cloud SQL, Google Cloud Storage, Google Cloud Storage (Nearline), BigQuery, and Google Cloud DNS.