Hacker News Reader

No one in the foreseeable future is going to use cellphones as parallel processors. First of all, the communication latencies are way too large to coordinate anything usefully. Second of all, users are already upset about the lack of battery life available for their phones.

It's not exactly parallel processing, but the Dark Sky app already does its forecasting calculations on the iPhone: http://jackadam.github.io/2011/how-dark-sky-works/

> The reason we encode velocity data as an image is so we can pass it off to the GPU on the iPhone and iPad. Both the storm prediction and the smooth animations are calculated on the device itself, rather than the server, and all the magic happens directly on the GPU.

In case anyone starts wondering, in the article it says that they're assuming that atmospheric motion is linear, which they justify (reasonably) because they're only looking over short time scales. So, although it is a forecast, it's a pretty simple one ;-).

> Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor...

Sadly, until we have a breakthrough in battery technology, all of those will be parallel processors programmed to be asleep as often as they can.

Sure, that's true - but they can just as happily run their GPUs if they are plugged in and charging without any battery issues. Although for the moment you are right, they aren't that useful as a powerful distributed supercomputer. They ARE, however, really really good at collecting live weather data and sending that to the supercomputer to process. One step at a time.

Smart phones are good at collecting live barometric pressure data but not much else. Temperature sensors aren't very useful because they are poorly calibrated, you can't reliably tell whether someone is outdoors, and the reading can be thrown off by the heat generated by the device or the person holding it. UV sensors are seldom pointed in the right direction. And I haven't seen a phone with a wind speed or precipitation sensor yet.

I genuinely don't mean to sound passive aggressive when I ask, do you know anything about this subject or are you just guessing?

I don't know anything about it, but I can think of numerous ways to mitigate the issues you mention. E.g.:

* Does it actually matter that the absolute value of temperature sensors is inaccurate, or is it enough that they have good relative accuracy (do they?)

* Do you really need to tell if someone is indoors or outdoors given a large enough sample of users in a given area? That also seems like something you could detect with heuristics (work hours, jumps up/down in temperature).

There's a lot of ways to tease useful signals from noisy data. Is it really the case that smart phone temperature sensors in the aggregate are useless as an input into weather forecasting models?

I also know nothing about the limitations of the temperature sensors on smartphones (I don't even know of any phones that have temperature sensors).

What I do know is that whilst teasing signals out of noisy data is achievable when the data are unbiased, the possibility of biases makes it significantly harder, especially when you don't know what the biases might be. This can be a significant problem with all sorts of data that you might think were fairly good, including both in-situ and satellite measurements.

I'm not sure how useful the usual techniques for normalising sensors would be for moving sensors.

Fixed weather stations have corrective mappings associated with them. For example the station at the airport might be assumed to be 1 degree warmer than the generalised temperature for your city. It might be known to collect 5% more rain when the wind is northerly, and 8% less for southerly. These mappings are very important, because they let you shut-down a weather station, and build a new one in a different location, and compare the data from the two.

if the weather stations aren't in consistent locations, with discoverable local behaviors, then converting the raw data into generalised data about the area is going to be very difficult.

I'm not an expert but have played around with sensors on some Android devices. The absolute thermometer accuracy will vary by device. When the temperature rises by 1℃ over the course of an hour is that because it actually got warmer, or is because the user moved inside or put it in a warm jacket pocket or ran another app that caused the CPU to heat up? In theory I suppose if you had thousands of devices in a small area uploading simultaneous temperature readings you could perform a regression analysis and extract a little bit of signal from the noise. But in practice I just don't see how you could ever get enough device density in a small area to make that work. Remember that in urban areas the vast majority of smart phones spend most of their time inside climate-controlled vehicles and buildings, so any temperature readings from those are worthless for weather prediction. But they do still give useful barometric pressure data because the pressure is fairly consistent inside and outside for a given location and altitude.

I'm a graduate student in Environmental Fluid Dynamics and I deal with a lot of environmental sensors. The temperature sensors on phones are pretty accurate, but they're quite sensitive to small changes. This means that nradov's original statement is correct: Body heat and cell phone usage are absolutely going to affect the temperature sensor. For example, Sensirion makes a high quality temperature/relative humidity sensor and they recommend the PCB have cutouts around the sensor to offer proper ventilation and minimize heat transfer from surrounding components [1]. On the iPhone 6s, the pressure/temperature sensor is on an edge, but it's also right next to the NAND flash chip, so I'm not quite sure what the heating effects would be.

I can't answer much to your second question but I imagine it's easier in colder environments, especially if the phone is not in a pocket. In addition, here's a publication that claims to be able to get useful signals from noisy data, so I don't know if I would say the measurements are useless [2].

[1] Fig. 10 in https://www.sensirion.com/fileadmin/user_upload/customers/se...

[2] http://onlinelibrary.wiley.com/doi/10.1002/grl.50786/full

just curious, do you have references to smartphone sensors being utilized in numerical weather prediction?

i'd been browsing through some WMO reports and other associated NWP lit recently and i'd seen stuff on GPS-RO, but nothing on anyone assimilating "smart" devices.

Sure. I've been working on this problem for about 5 years now, starting with the collection of barometric pressure from Android devices. I'm currently working on this with iPhones at Sunshine [1], where we collect pressure data (along with other metrics), but pressure is the most valuable.

There are researchers who use this data - we have collected about 4 billion atmospheric pressure measurements that we have distributed for academic and government research. The primary researchers are Cliff Mass and his lab at the University of Washington. There are also groups in Canada and the US that are using the data. IBM is now also collecting and using smartphone pressure data through their mobile apps. [2]

Generally speaking, the current trend is to take the live data stream, run it through a quality-control algorithm and then use kalman filters in the WRF data assimilation package.

There are some papers published, but it is still early. I will find some links to papers if you'd like to read them. [3]

[1] https://thesunshine.co/

[2] http://www.nytimes.com/2015/10/29/technology/ibm-to-acquire-...

[3] Utility of Dense Pressure Observations for Improving Mesoscale Analyses and Forecasts: http://www.atmos.washington.edu/~hakim/papers/madaus_hakim_m...

ah, thanks, that's fascinating.

"It also learns every time you actively report to the community on sky conditions and hazards, translating this information into weather predictions."

does sunshine actively run a numerical weather prediction model like WRF?

it'd be great to assimilate all these extra sensors into the NWP centre's models, but i imagine things like cal/val and WMO agreements to share data might make things difficult for commercial companies?

Yes, Sunshine is running WRF!

And yes it would be great to have all these sensor readings available to NOAA, Environment Canada, ECMWF, and everywhere. I have made lots of progress in getting them to talk about it, but it's a long road before any government starts using this data in its own NWP models.

you're probably already aware of this, but there is some US legislation trying to open up commercial data to gov't entities:

https://www.congress.gov/bill/114th-congress/house-bill/1561...

there's some stuff being written this year as well for the side that i work (space). with all the upcoming sensor gaps, they're looking at alternative ways to cover.

This is really interesting stuff. How are you guys managing that scale of streaming data? Some kind of Kafka based stack?

If you'd like to submit your own weather observations from anywhere on the globe, the National Severe Storms Laboratory has created an app[0] to report them.

[0] http://mping.nssl.noaa.gov

I don't know much about it, but I know some people doing a startup all about these sorts of things

https://senseai.io

In other words. This is how the actual IoT revolution (if one can call it that) looks like.