Hacker News Reader

Comment by JumpCrisscross | Hacker News Reader

> a few companies owning the data

The data are public [1]. Dark Sky et al process and repackage those data, along with their own analytics and UI.

[1] https://www.ncdc.noaa.gov/cdo-web/webservices/v2

Et al - maybe, but Dark Sky built their own near-future "hyperlocal" forecasting engine and they were the first ones to do that. They basically invented the whole genre.

Perhaps they expanded into repackaging NOAA data now, but that's not what Dark Sky is about.

Hyperlocal forecasting, who just ejected most of the world from their data collection pool.

IIRC they built the models with user data, but stopped collecting from phones because so many different sensors are used for rH and temp in phones plus being in pockets, battery temp, etc meant the data had too much variance. Even though I hate that they did this bc it's my favourite weather app, it actually would make sense to just use iphones if they were going to start collecting that data again since all the hardware is the same.

The barometer data that comes from phones (both Android and iOS) is extremely resilient and there is a lot of value to be derived from it. Ongoing research by Cliff Mass and team (at UW) shows that machine learning can help bias-correct and error-correct barometric pressure data on the fly with very good results. That data can also be assimilated into WRF to make forecasts with higher accuracy.

I work on a related app to do that barometer data on Android: https://play.google.com/store/apps/details?id=com.allclearwe... (US only for now).

I feel obligated to point to Cliff's blog, it's a great resource especially if you live in the northwest https://cliffmass.blogspot.com/

Thanks for the info and link, it looks like All Clear is positioned well to pick up dark sky android users. I'm going to give it a try.

Oh they were collecting data from phones? Sounds like a good excuse to delete the app before they spike it anyways.

http://jackadam.github.io/2011/how-dark-sky-works/

They used an ML/statistical approach to forecasting, rather than physical modeling. They had some secret sauce in doing fast ML on phones, before libraries like CoreML existed.

> they were the first ones to do that.

AccuWeather has been doing that for ages with MinuteCast. Dark Sky just looked more 21st century and was better marketed, probably

Careful there with unsubstantiated statements.

DarkSky - Nov 2011 - https://www.kickstarter.com/projects/jackadam/dark-sky-hyper...

MinuteCast - Mar 2014 - https://www.accuweather.com/en/press/24293762

That is, it's the exact opposite. MinuteCast is a rip of DarkSky.

As someone that's tried using the weather.gov API, let me just say it's atrociously bad. I'm sure the data is accurate and all, but the API is actively hostile towards developers.

Is that all of Dark Sky's value proposition? They take a beastly API and repackage it into something pleasant?

You've basically described all commercial weather apps. Dark Sky (and some others) do incorporate other local sources of info, but free govt data covers much/most of what they use.

I thought weather underground was crowd sourced from user weather stations.

It's designed that way. AccuWeather and other weather companies actively lobby to make sure that people continue using them, and not the free government sources of weather data.

See https://www.cnn.com/2017/10/14/politics/noaa-nominee-accuwea...

Thankfully, Barry Lee Myers's nomination was finally withdrawn in December 2019, but he won't be the last nominee to pull this sort of thing.

And it's US only.

The API does not seem that bad at all. Maybe it's been improved?

https://www.weather.gov/documentation/services-web-api

You can turn lat/long into a grid point:

https://api.weather.gov/points/39.7456,-97.0892

Then a grid point into a forecast:

https://api.weather.gov/gridpoints/TOP/31,80/forecast

If you have your own GIS you can presumably store the grid points yourself.

Oh, it's JSON!

How did you get JSON? When I was looking I could only find XML, which I wasn't willing to deal with.

Shame it's US only...

If you don't have precipitation data, then I would agree that it's fine. Their hourly forecast data is quite friendly.

But I wanted precipitation data so I had to use this endpoint: https://api.weather.gov/gridpoints/TOP/31,80

That endpoint groups hour-by-hour forecasts (see below) together to deduplicate data, but it makes it very painful to parse. For instance, "2020-03-31T15:00:00+00:00/PT4H" means that value is valid for 4 hours beginning at 2020-03-31T15:00:00+00:00. The durations varied from time interval to time interval and it made it super annoying to parse through.

{ "validTime": "2020-03-31T15:00:00+00:00/PT4H", "value": 0 }, { "validTime": "2020-03-31T19:00:00+00:00/PT1H", "value": 2 }, { "validTime": "2020-03-31T20:00:00+00:00/PT3H", "value": 4 },

Dark Sky claimed to do significant weather data collection themselves, via barometers in phones. I don't know if that data was ever used or available in their API, but definitely, the data Dark Sky had was all not public data.

I’m not in the US though...? :)

Reply on news.ycombinator.com