Show HN: Cloudrun – Numerical weather prediction in the cloud | Hacker News Reader

Show HN: Cloudrun – Numerical weather prediction in the cloud

cloudrun.co

41 pointsmilancurcic37 comments

Cofounder of Cloudrun here.

Cloudrun is a SaaS platform for the Weather Research and Forecasting (WRF) model. WRF is by far the most widely used numerical weather prediction model, but it currently requires access to supercomputers and domain expertise. Our goal is to make custom weather prediction possible for those without access to these resources.

This is our first MVP and we're working on making it more accessible for non-modelers. We greatly appreciate any feedback on how to make this service easier to use!

This is _literally_ a cloud service :)

Bad joke aside: I'm curious, what is the "business" need and market/end-users?

Congrats and good luck Milan and Josh.

Thank you! We encourage bad jokes at Cloudrun. Our next project thedarkweb.edu is a site for custom-made blackout curtains.

Our end-users are currently mostly students and those working in academia. For these gals and guys, it can be a PITA to use supercomputers to run weather models. Milan can speak more to this.

However, the potential user is anyone who needs custom weather prediction. Renewable energy, commodities trading, event planning (sports, concerts, etc.) to name a few.

We're hearing about new potential industries all the time! Ideas are welcome!

Not sure if you envision getting down to a consumer level in some way, but I race sailboats. There's plenty of people who use forecasting services to get a forecast tailored to a very small area for their race course or route (think 10 sq miles).

I'd presume that mountain bikers, kayakers, climbers, and really anyone else who is into sports that are dependent on weather may be interested in similar information.

Delivering the information to a community like that seems like it would be too niche for your service, but for the people that already offer services to those people, perhaps there'd be interest.

Thank you for the feedback! At this point we don't target end consumers, but rather small businesses that create these kind of products. However, we do get a lot of requests for specialized and intuitive graphics/visualization and may go further in that direction later this year.

Naive question from a semi-beginner about your market: isn't there already a lot of competition for hyperlocal forecast and good accessible visualizations (e.g., darksky)?

Most existing services that I am aware of provide nice visualization of existing publicly available datasets (GFS, NAM, HRRR, etc.). Cloudrun allows the user to create their own forecast product/dataset, so it is mainly a power tool and less a graphics/visualization service.

Thank you, mino!

At this point, we are targeting existing WRF users and newcomers into the field. These are either academic meteorologists without access to big hardware, or SMBs with focus on weather intelligence for agriculture, commodities and energy trading, and similar.

At a later stage, we will probably move toward operational weather prediction, and expand to ocean circulation (ship routing, search & rescue) and surface wave models (custom surf forecasts).

As Josh said, though, we've just started and learning every day about potential use cases.

credit card form is broken. chrome and FF.

Doh! Thanks for the head's-up!

Any thoughts of supporting MPAS?

It's possible, we had a few people ask for MPAS already. Beyond WRF, the priority will probably be on surface wave modeling, but ultimately prospective users will decide.

Do you have preference for MPAS Atmosphere, Ocean, or both?

Surface wave, as in Wave Watch III? If so how about CICE and or HYCOM too?

My preference would be for MPAS-A.

Thanks

At the moment, I can't say much about the wave model that we will offer, other than that it will support both global and any custom regions, and it will not be WW3. NOAA already does as good job as you can get with WW3.

Do you supply all cores (NMM-B & ARW)?

How about all the other configurations of WRF (HWRF, WRF-DA, WRF-CHEM, Global)?

What sizes can you scale to? What type of hardware are you running on, if you don't mind me asking?

Thanks.

For now, we only offer ARW in real data case mode (em_real). We will slowly enable other modes and WRF components if there is demand.

All runs go on Intel Xeon Platinum 8168. We currently restrict the execution to shared-memory mode while we fine-tune the scaling in distributed memory. If you try to submit a run too big (either for RAM or scalability), Cloudrun will not let you proceed.

Adding creditcard is broken. Btw - some comments below are asking about use-case and joking that physics grad students are happy. I'm looking for accurate hyperlocal mountain weather forecasts generated with the latest possible data. This is useful for avalanche prediction, planning expeditions, and trips in general. I live in British Columbia btw.

Thank you for reporting the CC issue. We are looking into it.

Also, great to know about your use case! We have been looking into applying this for snowfall prediction for ski resorts, but I wasn't thinking about mountain expeditions in general.

What is the typical forecast time window that is important for your use case? Is it < few days or out to a week?

Currently we use SpotWx [0] for 12hr. It gives you the choice of whatever numerical model you like. These are very important for mountain trips because rain in the summer will make things very slippery etc. In the winter (ski-touring), precipitation is not a big deal but avalanche hazard is -- so there are avalanche forecasts [1] and [2]. You'll note that they breakdown the forecast by elevation range, below treeline, treeline, and alpine.

The shortcomings of the existing systems is that it is cumbersome to get elevation properly into the model. SpotWx for example, gives you the forecast for the elevation of the square you clicked on, which is the average of the surrounding elevation. This may be irrelevant if you're walking a ridge vs a valley bottom.

SpotWx, however, does something very right, which is very good hourly breakdown. The avalanche forecasts are typically for a very large area, and the timing of precipitation isn't important, but heating due to temperature change or sun is very important.

[0] spotwx.com/products/grib_index.php?model=hrrr_wrfprsf&lat=49.25963&lon=-123.12047&tz=America/Vancouver

[1] http://www.avalanche.ca/map

[2] https://www.nwac.us/mountain-weather-forecast/current/

Nice! The terrain issue that you mention is the limitation of the grid resolution. For example, HRRR, the operational model used in the charts in your link, has 3 km horizontal spacing. Individual peaks and slopes are poorly resolved at that resolution.

We are already (internally) experimenting with 1 km and sub-1-km grid resolutions using HRRR as initial and boundary conditions. Creating custom forecast domains at fine resolutions is something that we are currently developing and should be ready in the next 2-3 months.

Thank you very much for your input, it's super helpful!

You've got my email address from the email I sent you regarding the malfunctioning CC form. I've got a bunch of super eager UA-testers from BC Mountaineering Club if you're interested.

Fantastic, thank you, I will follow up!

What's the advantage of this is over running, say, at Penguin scalably on Infiniband nodes if you need the compute? (I wonder how weather researchers work if they don't have access to HPC facilities, or why they don't take it seriously enough to learn how to submit jobs if they do -- though that seems to be a common syndrome these days.)

I don't know weather models; is there something about WRF that allows people without domain knowledge to get sensible results? That's not the normal experience with complex models -- sometimes even with people that are supposed to be domain experts.

These are great questions!

> What's the advantage of this is over running, say, at Penguin scalably on Infiniband nodes if you need the compute?

POD gives you the nodes and a secure shell and you take it from there. In contrast, we are building a SaaS, so the hardware and the configuration of the model (from compiling the MPI libraries to running the model) are abstracted away. The user provides input (files for now, parameters in the near future), and gets output as a result. We scale in parallel also, but don't use Penguin under the hood.

> I wonder how weather researchers work if they don't have access to HPC facilities

Many meteorologists only analyze existing model data, and not necessarily run the models.

> is there something about WRF that allows people without domain knowledge to get sensible results?

Yes, in the sense that the output data represents 3-dimensional future snapshots of the atmosphere. So you don't necessarily need to understand the internals of the model (and definitely not the labor intensive setup) to interpret the results. It really depends on what you're after. If you study atmospheric dynamics, you're likely already a meteorologist. However an energy trader knows how to interpret temperature, wind power, or insolation trends, but does not need to know about how cloud microphysics are implemented internally.

Is there a guide on how to create runs for someone from a non weather modelling background?

In particular for the input files, 'namelist.input' seems like a list of parameters I could google for/tweak, but I'm not sure where I'd go about sourcing data for the initial/boundary condition files 'wrfinput_d01' and 'wrfbdy_d01.

Unfortunately, at this early stage, Cloudrun still requires some level of WRF expertise, needed to create these input files. We are working on automating these steps so you don't have to, but it will take another 2 months or so.

In the meantime, you can email us at help@cloudrun.co, let us know your region of interest and time window, and we will happily create the input files for you.

We considered not launching the Beta before the input file creation was done, but we wanted to get feedback as early as possible. I really appreciate your interest and feedback!

If by any chance you decide to go creating these files yourself, the process is tedious but well documented here: http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_...

We do plan to start writing short tutorials on setting up simulations and geared toward non-experts.

Appreciate the link & response, happy to wait a few months.

I've been looking to find a way to get short term localised rainfall forecasts in Australia (for sporting events) and excited to see how this goes!

Shoot us an empty email over to help@cloudrun.co. We'll send you some free credit to get you started once this is ready!

A lost opportunity to call this CloudCloud!

Geez what a glaring oversight! We'd change it right now if not for the hundreds of Cloudrun beer koozies we had made :(

for those who are curious, a teeny bit of googling reveals this to be a hosted version of https://github.com/NCAR/WRFV3, which is a super duper open source numerical computing code base (and the license grant is roughly "this is public domain" but with more words).

i've always wanted to learn about weather simulation models so happy to stumble into this domain

I think I was able to run WRF or a similar model on my laptop in 2010 (A Macbook Pro with a Core 2 Duo). It took about a day to compile and run the sample dataset. I gave a presentation about it at my weather forecasting class. I almost got it running with our local sounding data but I ran out of lab time I needed for the class. It was fun though I was reading reddit while it was compiling and running. I was the only person to give a presentation about it in my weather forecasting class.

Kudos for sticking through with it for your class project! I have been working with and have contributed to WRF code since 2009, and it's sure not straightforward. Porting it to different HPC systems can be especially painful. This pain point was one of main motivators for building Cloudrun.

So, the class was actually structured so that you had to perform ~40 hours of lab work. By using your software I could read reddit and perform lab work by babysitting a compiler. So thank you for allowing me to use up all of that lab time :)

Stoked you're excited about learning weather modeling! WRF is indeed super duper, duper open source. Scope it out!

Cloudrun's purpose is to run WRF. Most runs are computationally expensive enough that they could take days to run on your home computer. By then, your weather predictions are less relevant.

Because of this, WRF is typically run on supercomputers, but even people with access to these tell us they can be difficult to work with.

The other goal of Cloudrun is to make weather modeling accessible to the layman. It can be a pretty complicated process to run WRF. Let us know how your journey goes!

If there's anything we can help you with, don't hesitate to contact us anytime: https://cloudrun.co/help/contact-us

Credit card form is back up. Sorry to anyone who tried to sign up and couldn't before!

Somewhere, I assume, some atmospheric physics grad students are very happy.

Reply on news.ycombinator.com