Hacker News Reader

Comment by klingonopera | Hacker News Reader

Ooof, yes, now I see where the $10K/day is coming from...

As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?

View on HN

Google Colab gives each user a dedicated Nvidia Tesla K80 GPU for 12 hours for free, which is super cool and presumably why the project is on Colab. But as each user spins up their own Colab instance it pulls down the 6GB of GPT-2 model weights, incurring 30-40 cents of data egress charges against the GCP Storage Bucket that the data is stored in.

60k users yesterday * 6GB each -> 360TB of data egress!

Normally, a scenario like this wouldn't involve bandwidth costs because GCP -> GCP same-region bandwidth is free, but Colab is technically not part of GCP, so the bandwidth charge is being assessed as egress to the public internet, which is pricy for that much data. Though it's probably still a lot cheaper than paying for the GPU-hours for that many users.

This is sort of a killer example for the layperson of what ML can do, so hopefully Google will recognize this and comp most/all the data egress since it's a drop in the bucket for them, but every person that uses it can still go "Wow, Google's services allows for some amazing stuff."

Oh, it does. NOT.FOR.FREE, though; that's the entire reason it exists at all.

In other words, yes, this is an amazing amount of raw power - with a corresponding price tag.

But you could set up your own file-serving node and then data transfer to end-user Colab instances would be free, right? ("Free" — costing only as much your CDN costs for 360TB/day, which is still quite pricey, I think, but not as much as $10K/day. I.e. Google wouldn't charge you for data transfer here.)

Yup. I think I saw yesterday that they were looking to move the model to BitTorrent.

The $10K/day was actually coming from the large egress fees they were getting for transferring the models and agents from Google Cloud Storage to the Colab notebooks. I think if you were to serve the game as a web app you definitely wouldn't need one instance of a 12 GB GPU for each user. But the thing about Colab is that you need a Google account to use it, and you run your own notebook, independent from the author's account.

$10k/day is just for file transfer, not for GPU.

Ok, ok, let me see if I now got this right: The AI needs a VM (because users rarely have 12GB GPUs at home), and this VM then downloads about 6GB from GitHub each time a user opens up a new session instead of sourcing that from a locally cached copy?

EDIT: Almost, the VM runs on Colab, which only works with Google, and Google's charging for the upload to Colab? ...more like collaborator, amirite? dodges rotten eggs

Yeah, now you almost got it right! Colab is a service provided by Google.

The funny thing is the high egress fees Google charges for transferring data between two of its services (GCS -> Colab).

Funny, cheap and cheerful, little hidden costs in the fine-print. Oh my Google, how could we ever be mad at you? opens a can of laughter

Reply on news.ycombinator.com