My team pulls a lot of weather data for utility engineering work we do, and the NOAA APIs have proven very hard to navigate. Some data only available via FTP, lots of different weather station networks with different metrics (e.g. lots of variety in the cloud/irradiance data).
Darksky provided a nice clean interface, but I guess we'll go back to cobbling things together directly from NOAA.
I think it will be pretty hard to do well in open source though. Server costs and keeping things working when NOAA (etc.) sources change is probably going to be expensive.
Develop the (Ideally extensible) logic to parse a given government funded weather source into a common format, and have “API consumers” run their own instance of services (or set them up an instance for $ and use the profits to help with project costs).
There are also a lot of open projects that distribute the work of handling tweaks to parsers and source lists when the upstreams drift.
Even globally I can't imagine the minutely data from reporting stations everywhere being "big data", maybe in the 10s of gigabytes a day (very rough guess, I haven't looked into this specifically)? I'd still be willing to bet API / web requests dwarf the processing and bandwidth requirements of the raw data for a public service like this.
That probably does put it pretty reasonably in the realm of self-hosting if you put a threshold on how much historical data you want to keep and geofence the region you care about.
I'd floated a Weather-Service-In-A-Box to my clients a few months ago, but got no takers. The idea was something you could set up on a small number of instances fairly quickly and leave running to harvest NOAA/ECMWF/etc data. Fairly simple front end to answer queries. Data tiles for the visuals. That sort of thing.
If people (with money) are interested now, hit me up.
Storage and processing is relatively easy since it's just a matter of throwing resources at it. However, it'll be damn expensive to store months or years of historic simulation data.
The difficult part is writing and maintaining the processing scripts. The different weather services store data in different formats, have different ways to download the data and sometimes change data structures or fall over.
Edit: removed a spurious sentence.
If you wouldn't mind: how do you go about creating a simulation around the empirical data? This seems to be very core to how Dark Sky is able to provide such accurate realtime data for geographies in between weather stations.
Depending on your needs, I may have data you can use. I only process forecast weather simulation data - not weather stations. I'm a solo dev so I can't offer high levels of service, etc... but feel free to contact me.