Hacker News Reader

Comment by fxtentacle | Hacker News Reader

Google's Cloud TPU makes AI training like 10x cheaper than the cheapest GPU instances that AWS and Azure offer. Plus Cloud TPU is perfectly integrated into TensorFlow, due to both being Google products.

View on HN

It’s not 10x more expensive. For example, Cloud TPU costs $12 for ImageNet training. AWS is $42. Lambda GPU Cloud is $19.

You also need to take into account the cost of changing and tuning your code to run on a TPU.

https://dawn.cs.stanford.edu/benchmark/#imagenet-train-cost

In my case it was 10x. We're training networks to estimate optical flow on Sintel and KITTI using TensorFlow. On GPU, we hit the 12GB limit which caused TensorFlow to log a message that convolution performance would be limited due to not having enough cache RAM. On TPU, we didn't hit the RAM limit.

Oh and amazingly enough, I took the exact same Python file that I built on GPU on my workstation, ran it in Google Collab with a TPU connected, and I really only had to change the configuration to make it work. There was no need to change or tune anything. I was pleasantly surprised by how well that worked.

And let me rant a bit longer, when we started this project I actually first tried to use AWS because our company already had billing set up with them. But AWS was using CUDA 10.0 for SageMaker, which has a known crash- and nan-inducing bug for 2D convolution. AWS support being non-existent as always, we couldn't make things work on AWS and then went bare metal for our GPU training with hetzner.com - which simultaneously also shaved like $30,000 monthly off our bill. AWS's EC2 GPU instances are really expensive when compared to a dedicated root server.

I’m not arguing either way. But that’s still not where the money or growth is. Cloud computing only makes up 5% of enterprise (big Fortune 500 companies) computer spend.

MS recognized moving enterprises with boring workloads to Azure was where the money was along with hybrid computing. That’s how it was able to succeed so quickly.

Besides,Microsoft has always been focused on the dark matter developers doing boring stuff. Their tooling is far better than Google’s and their IDEs that developers are already using integrate with their cloud offerings.

From watching NVIDIA's presentations, it sure sounds like AI is growing exponentially. Plus Google needs AI to advance for Waymo to become safe and profitable. Looking at other companies in my area, all of them have plans for AI, too, even if it is trivial stuff like automating the most repetitive 5% of support inquiries. So if Google can beat AWS out of the AI waters early on, that might become very valuable in a few years.

It’s easy to grow “exponentially” from a small base.

Also, if the only advantage that GCP has is price, that’s not defensible. It’s not like it takes a great engineering effort to change a price.

The market leader can usually afford to be a “fast follower”. That was MS’s playbook for decades.

I posted a link earlier from a podcast interview of someone who works at GCP. She said that only a few small internal projects at Google run on GCP.

I may be wrong, but I recall reading Azure grew quickly because they already had MS contracts in place with many enterprise companies. They simply said, we’ll also hold your hand in the entire cloud process.

This was critical because by then if you weren’t on AWS, you probably didn’t have the DevOps team to do so - MS was now willing to lend you enough consultants to do it for you.

The number of spend shouldn’t be astronomically high. Most workflows in cloud computing are commoditized low-margin. It also explains why Google wants to do high-margin AI stuff, and why AWS is happy with low-margin areas.

That’s kind of my point and what I’ve been saying. Microsoft doesn’t tell a company that they should be using their new cool technology because we are “smart people” (tm) who know what’s best.

Microsoft would say:

“If you have a workload that still requires Windows 2000 and SQL Server 2003. We will help you migrate it to Azure as is and we will extend your support contract.”

While Google will talk about migrating your entire workload to K8s.

Plus, Microsoft also has K8s support, so not only can they can "lift and shift" the existing dark matter applications, but they can also offer a conversion path/greenfield path.

That’s not where the money savings come from being in the cloud. K8s is not the be all end all of being cloud native.

Sure, but it adds an illusion of portability, which is partly what GCP is taking advantage of in focusing it and definitely what Microsoft here is taking advantage of in promoting K8s to dark matter developers:

Part of Microsoft's implied message in heavily marketing K8s is that they know GCP is heavily K8s-based, and know GCP is heavily competing with them on price, and they are counting on some of that "dark matter conversion work" wanting to move towards the "more portable solution" in order to "maybe when the moons are right eventually shift over to GCP or something cheaper than Azure". By offering that conversion path at all, and knowing Enterprise's weakness for sunk costs development over from-scratch rewrites, they are playing a very different game than what GCP can offer.

Portability is always an illusion once you have any type of scale. Everyone who has ever done any type of large implementation or migration knows that.

Even if you did everything in a “cloud agnostic” way, once you have even a couple of dozen VMs, your networking infrastructure and security rules, your user permissions, your DNS entries, your site to site VPNs, your client to site VPNs, all your data in databases, security and compliance audits etc. migrating your infrastructure is going to be a months long process, involving project plans, regression tests, and still have little to show for it.

I was involved in a much smaller migration as a dev lead when all we had to do was migrate to WorkDay for HR, payroll, and time tracking. That involved tons of integration work and hiring Workday consultants. Not to mention consolidating all of the other SAAS solutions that we had to integrate when we merged with another company.

Which is exactly part of my point in how shrewd Microsoft's marketing is here. K8s exactly has the marketing "smell" of "cloud agnostic", without being in fact provably so. Microsoft can compete with Google better in that arena than Google can precisely because they have other services. It's a "yes, plus" situation. "Yes, we do K8s too, but also we can lift and shift your existing operation in the meantime while you build it." That has a lot of hidden lock-in power and I presume Microsoft knows that very well, even better because they know after that "lift and shift" many dark matter companies will just stop there because while they could migrate, migrations aren't free or cheap and they can kick that can down the road possibly forever. Managers every day are persuaded by the possibility that they could migrate to K8s if they wanted to, whether or not that is ever the real plan. It's why "lift and shift" really is a fascinating game changer that Azure has that neither GCP or AWS can quite so directly compete with.

It's why Azure's K8s strategy is so fascinating: precisely because the more they make it "cloud agnostic" on paper and are good K8s/Docker citizens the greater power they have for Manager mentality lock-in to the non-cloud agnostic stuff (especially "lift and shift").

Reply on news.ycombinator.com