UC Berkeley launches SkyPilot to help navigate soaring cloud costs

Most people don’t bother refactoring their application and leave whatever IIS garbage they have running on windows 20xx in a cloud instance (to say nothing about the windows licensing cost).

Also universities don’t have the same cost models as companies. Their labs have less money - except those institutions with large endowments - but they all have access to near limitless “slave” labor in the form of grad students and post docs.

"Simplify to succeed and Complicate to profit"

-- (likely) Tim Ferris.

I sat out of the cloud hoopla and it cost me opportunities - Now, I am dragging my feet into it to be employable.. but the Anti-Cloud (River?) is going to be a concept. 39 signals already did a blog post and change in strategy.

I look forward to the economically lean years of 2023 and 2024 and HN front page articles about how moving away from Cloud and owning your own server is saving 'millions'.

Reducing cost was always about 1) getting rid of the hardware infrastructure team and 2) getting rid of the tax of having to think about the hardware as a CIO

Savings is a really complicated thing to measure, especially when you're trying to accurately account for internal costs which aren't cleanly broken down per-project. It's very easy to get an itemized bill from a cloud provider which looks much larger than your internal service because the latter numbers don't include staffing or overhead costs (e.g. how much idle hardware are you paying for because the alternative is telling someone they can't have a VM until you complete a new procurement?). What I've found is that almost every time someone tries to get those internal numbers it goes the other direction by a significant margin because they weren't accounting for things like how much staff time they spend managing systems, especially for the compliance type stuff.

One really big factor here is opportunity cost: if it takes 6 months (not hyperbole in .edu) to get a new server from the enterprise IT department from request to being in service and correctly configured, there's a significant cost which you don't receive on an invoice. This is especially true if you're looking for any service more advanced than a bare Linux/Windows install where the staffing & internal barriers are going to dominate the true cost.

Network bandwidth is definitely the most egregious point to complain about with commercial cloud providers and it'll cancel out any other savings for some applications. You can negotiate significant discounts with most of them and for something like UC Berkeley I'd also assume they might be able to take advantage of things like AWS' cheaper peering using Internet 2, but it's still something I'd mention to your sales rep as a deterrent any chance you get.

At my institute, we've been steadily expanding our HPC facilities and adding newer hardware for the SOTA research being done by faculty and students.

Also gives people exposure on how to manage their instances and tooling required around their projects, which I reckon helps in the long run.

Does it support spot instances ?

Underneath, Skypilot uses a Ray quite a bit. You can read is paper here: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...

A lot of Berkeley researchers use the NERSC facilities at the nearby Berkeley National Lab: https://www.nersc.gov/

At several universities I've been at, HPC groups have been utterly unprepared (and disinterested in becoming prepared) to handle PII or any sort of health or confidential data.

Berkeley CS (and in particular the systems research labs like AmpLAB, RISE lab, etc.) gets enough funding from AWS, GCE, Azure, etc for it to be uneconomical to have a data center administered by the campus.

That being said, there are HPC facilities shared with LBNL, as well as smaller clusters operated by the department.

It really depends on the workload. GPU clusters are usually cheaper to run in house since Nvidia let you use regular GPUs for research which end up cheaper than cloud GPUs. And often Universities will charge less overhead for capital expenses on a grant which can artificially reduce the cost of running it yourself.

The big downside of institutional HPC is it can be difficult to get stuff running on the ancient distributions they're stuck on. Nowadays perhaps Docker makes things easier but even running a Docker container was a challenge when I worked at a university some 5-10 years ago. As a software engineer who had not used HPC previously I found it much more difficult than just firing up jobs on AWS which had documentation (though for many researchers it was the opposite.)

I also kind of object to the term HPC... with the exception of a small number of shared memory clusters used for physics simulations they're usually just a bunch of standard servers often with incredibly slow network storage. Nothing high performance about them.

I too have been surprised by the poor state of research compute at American universities. Of course it's hard: that's why it needs some smart and expensive people who do research on computing to run it (but that's what universities are all about). But maybe it's a cultural thing: in the US organizations including universities like to rely on commercial services when they can instead of seeing the value of doing it in-house.

Berkeley has at least one cluster, probably more.

https://docs-research-it.berkeley.edu/services/high-performa...

University HPC clusters are typically managed and owned departmentally. My former employer had two clusters in two different departments. I worked directly with a few other universities who also had departmental HPC clusters, and I’ve read a boatload of HPC docs from different unis and labs. Sharing clusters for many types of workloads seems to happen more at the regional level (like Archer cluster in Edinburgh, or PNNL in WA).

Ours is a fairly small cluster of maybe 60 or 80 job nodes. All compute networking travels over Infiniband—-I think we bought HDR for the new cluster, but price may held us to EDR—-which is probably why the cluster was so expensive to build (around $2M). Our storage cluster was another $1-$2M project.

All that is to preface this: cloud usage doesn’t mean they’re only using cloud resources. Our goal, for instance, was to build a hybrid cluster (and their continued goal, as far as I know). The first step was to offload low priority work with low resource requirements and wall times, and comparatively long timelines to the cheapest possible, compliant cloud provider (some of our researchers have specific data sharing and privacy requirements).

Let’s say a job is submitted on Monday morning at 9 am. It needs 2 CPUs and 4 GB of RAM, and a wall time of five minutes. The researcher can wait until next Monday at 10 am for the results. There’s at least a chance that running this job in the on-prem cluster is less cost efficient than offloading the job to another resource, whether the cost is direct (i.e. 5 minutes on-prem costs $0.10, and the cloud costs $0.05 or whatever real values would look like).

Ideally, we would have some method by which you could compare the cost to run a job in the cluster vs another resource, and it would automatically offload jobs of up to $X to the cloud, whether as a very low priority queue, or as overflow in times of full utilization. There are several other conditions that would need to be met, as well, but you get the idea (just for example, one consideration is if the work be interrupted. If the job has to run start to finish, the cloud is likely not suitable for that work).

(SkyPilot dev) Can't agree more on autostop / autodown. Subjective to measure but save our own costs massively.

I believe the Ray suite of ML + distributed computing tools is also from AMP descendants:

https://docs.ray.io/

https://vast.ai?

Single ML job can’t be distributed very well. More often than not they hit network limit, even on single zone 1 GB/s network speed that we normally get. Most of the distributed workloads use something like NVLink.

Datacenter GPUs have >1000gbps network connections between nodes, which is necessary to actually utilize GPUs with current training techniques. It's possible that a furthering of techniques used in GPT-JT[1] might make it feasible to use home computers, but even GPT-JT requires at least a 1gbps connection.

[1]: https://www.together.xyz/blog/releasing-v1-of-gpt-jt-powered...

why....that sounds like Web3! (I kid I kid...)

For infrequent, unpredictable workloads using something like spot instances on AWS and scaling as needed (including to zero) will likely be cheaper.

I think Hetzner and other budget cloud providers are excellent choices for use cases that require always on, reasonably predictable workloads like webservers.

They usually are, unless you get interruptible instances. You can compare pricing for most cloud providers here:

https://cloudoptimizer.io

(my hobby project)

Does DO or Vultr even offer the HPC instances AWS does? 96 CPU, 384GB, etc? Let's not even mention storage or networking.

Looks great! Do you anticipate any pushback from cloud providers, who might one day decide to restrict access to stop people getting a better price elsewhere?

I’m curious how this affects network and data storage costs. Maintaining data storage and private fiber to all the clouds has its own costs.

Conisder adding CoreWeave for GPUs. The A100 there is about half the cost of V100 on large clouds.

checkout lambda labs, they have very competitive pricing on GPU, they are almost half the price of CoreWeave at the last check.

It's funny that that is exactly how disruption works. Use tons of VC money to flood a market with a cheaper product, become dominant, then jack up prices hoping people forgot the alternatives.

It doesn't have to be jacking up prices for exploitation either, just sometimes the costs were always there dictating the price and, eventually, you have to foot the bill for those costs.

I've configured Cisco routers, F5s, LOMs, PDCs, NAS boxes, built servers from parts. And I'd never do it again.

That shit has literally no value, and it's quite difficult to configure, manage, and maintain. And no matter how good your process, you (or your team member) inevitably will forget to go into the BIOS and turn off power saving...or forget to turn off (or on) proxy arp. Or you'll reboot a box and it won't come back.

All costs can be negotiated with up-front commitments. We are small, but we pay .01/GB for egress bandwidth. And we got that price from akamai, fastly, and AWS by asking. I don't remember how much real colo places charge anymore, but it was a whole lot more than that.

Enterprise SSDs, redundant hardware, etc cost money. Does your startup qualify for a lease?

And really, why spend money on something that has no value to your business? Does having 5 data centers add any real value or competitive advantage? Does having that F5 expert on staff make your end-users happier?

If you answer yes, then go for it. But really, it's a total waste of time and money. You might as well make your own pencils and paper for the amount of value it delivers.

This may be an overkill.

But I would argue most projects don't even need a cloud. Just go to Server Hunter, find a server up to your specs, and save 90-95% of your cloud costs.

These days you can scale vertically or horizontally for a long time without moving to a cloud.

Agree. I had to calculate the cost of running and upgrading an existing HPC infrastructure that was approaching it's end of life vs cloud. In house HPC beat cloud cost in almost all scenarios.

There's an alternative path where cloud companies actually start to compete on price. Right now you will pay gigantic premiums on your AWS infrastructure just by virtue of being in their ecosystem.

Personally, I see Kubernetes having huge potential to drive down costs. The ecosystem is still young but it's still possible to run such a wide range of services and tools on top of it, and it can easily be run on-prem or using managed providers.

I personally am part of a team building a much cheaper k8s service as I think costs have grown out of proportion and k8s offers the first real good shot at challenging AWS on cloud & costs.

They have a project which addresses this concern as well: https://skyplane.org/en/latest/benchmark.html

And isn’t the biggest issue with running potentially large jobs in the cloud the cut off when it’s cheaper to use your own hardware. After a few months or dozens of runs of your large model in the cloud you may have reached the point where purchasing would have been cheaper.

Something that could look at your code, data and budget and say upto X runs use cloud A, for more than Y runs it would be cheaper to buy/lease these GPUs etc. would be interesting.

I think it’s common to train 100s of models on the same data for experiments. Then you would only need to copy data once to all the cloud storage and run experiments as you wish.

Also most cloud provider don’t charge for ingress so you could move the data from something like R2 to cloud as many times you want..

I'm one of the creators of SkyPilot. Thanks for the thoughtful questions and let me try to take a stab:

SkyPilot is not just for multi clouds. It's useful for all of these scenarios:

- using a single region of one cloud

- using multiple regions of one cloud

- using multiple clouds

Data transfer between zones/regions within a cloud is much cheaper than across clouds. We see many users falling in the "one cloud" category and they frequently read 10s of TBs of data across regions to do ML training.

Finally, saving money is one of several key problems we aim to solve, and there are quite a few ways to save other than lots-of-compute-on-small-data. Other reasons why you may want to use a system like SkyPilot include

(1) improving resource availability (big pain point for GPUs/TPUs)

(2) use one interface and know that your jobs can migrate across regions or clouds

More rationale in the intro blog post: https://medium.com/@zongheng_yang/skypilot-ml-and-data-scien...

do you use SAR?

One big reason people use multiple regions or clouds: higher resource availability.

Allocating scarce resources can be very hard on the cloud. These are things like high-end GPUs (both on-demand and spot; the latter being much harder to get) and also beefy CPU-based instances. We've seen 10s of hours of waiting times or longer.

The natural solution is to have the flexibility to allocate resources in multiple regions (and ultimately, clouds) to increase the total pool size.

Then lucky you: continue to use JSON since it's a subset of YAML