Hacker News Reader

Comment by discodave | Hacker News Reader

> The Grid runs more than two million jobs per day. At peak rates, 10 gigabytes of data may be transferred from its servers every second.

So that would be 80 Gigabits per second, or the networking capacity of 4 hosts with 20Gbit connections (but let's say you had 3x4=12 hosts to be safe).

On AWS you can get m4.16xlarge, p2.16large, x1.32xlarge or r4.16xlarge hosts with 20Gbit networking.

Those hosts can be had for 10-50k per year. So that works out to be 120-600k dollars per year.. which doesn't seem like that much in the scheme of things. 120k is less than one developer in the bay area!

Of course there would be other costs like networking and storage, but overall it seems like they could save $$$ in the cloud.

View on HN

That is just the interconnect. Grid jobs are usually very compute heavy: "[...] includes the majority of the 100,000 processing cores in the CE." That is the expensive part. We bought p2.16xlarge equivalent nodes for just about 1/20 of the "3 Yr All Upfront Reserved" cost of EC2.

Depending on the utilization (i.e. somewhat low utilization) you could save some money in the cloud if you compare it to the usual 5 years depreciation cycle.

But for scientific computing you usually still use your computing nodes longer than the 5 years of depreciation/support contract (often until they are close to fail).

Another reason why cloud is not very attractive is with how budgeting rules work in those old institutions. You propose a cluster and get the money, all calculated from the start. On-demand cloud pricing will be very hard to get approval for.

> [...] other costs like networking and storage [...] I am currently using the equivalent 20k dollar in yearly EBS costs on our scientific computing cluster. You can buy a lot of storage for that. And the internal IT department that exists anyway will manage it.

Cloud can be a very hard sell to scientist.

Edit (a point I forgot): Cloud only makes sense if you can use spot market capacity extensively, then it can become very competitive with local installations (again depending on required utilization).

I don't know the exact numbers, but it is regularly evaluated, and they found that it won't save money moving to the cloud (yet).

Apparently, grid jobs have different IO/CPU/Memory characteristics from typical cloud applications. My jobs tend to use a lot of CPU and bandwidth, but are mostly IO bound. A friend did a very, very CPU intensive analysis for his PhD, and they estimated that it would cost $30 million to run it on AWS. I'm not sure where they got that number from, so take it with a grain of salt, but even if they are an order of magnitude off, it is still prohibitive.

Another issue we are facing is RAM consumption. Many scientists are not trained programmers, so there are a lot of memory leaks. It didn't use to matter, an analysis program ran only a few hours and was single-threaded anyway. Now we are moving to using multi-threading, and we have been using multi-processing anyway... And as I understand it, in modern hardware the RAM/CPU ratio is getting lower and lower. If your job or thread needs 8 GB RAM, you can't run many of them on a 32 core CPU...

So yeah, I think the main issue is wierd resource usage patterns. Not that it would be impossible. Grid computing is basically just a weird parallelly evolved version of cloud computing, after all.

For the Alice experiment most jobs are IO limited. Although throughput should be improved by run 3 via backend changes (I hope so at least, currently doing my MSc. thesis on exactly that...).

Simulations are compute heavy, but analysis tasks vary, with the speed of a modern processor future analysis is still likely to be IO/bandwidth limited. The ratio of RAM to core on the test server I have access to is 128/20 so 6.4 GiB per core.

But for (future) analysis the plan is to use, or at least try to use, shared memory and grouping users tasks together based on the required data (They speak of an analysis train with wagons). So a large part of the ram requirements, namely the backing data, should be shared among the cores. I think ideally there will be a top-level scheduler for each node which tries to minimize the required bandwidth (but that is outside the scope of my work so who knows how it's implemented). With a few "best practices" it should be possible for most analysis to consume a 'reasonable' amount of memory at any given time in that case.

Reply on news.ycombinator.com