IBM to Acquire the Weather Company
nytimes.com
nytimes.com
http://www.wunderground.com/history/airport/KSFO/2015/10/28/...
Source: http://www.cnbc.com/2015/10/28/ibm-plans-to-acquire-the-weat...
api: http://www.wunderground.com/weather/api/d/docs
National weather service, http://weather.gov is my goto weather forecaster, but at the time I wrote the code, getting hourly data based on lat/long for several thousand locations (some outside the US) was problematic. and XML..
For example, I've used it to crank out random temperature histograms in the past[2], and even plotted some long-term averages to decide on temperature and precipitation ranges when going on camping trips. It's really useful if you're willing to work with the data a tiny bit. You're getting csv files, so it's not terrible to work with in any way via Python or R, or whichever data-friendly language/tool you want to use.
[1] http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn_map_interface.ht...
[2] http://btus.us/decadal-histograms-of-monthly-yosemite-temper...
I realize that it's theoretically possible to get this working. But it's not easy. What I want is a way to enter a postal code and get a JSON response with the precipitation for the last week. Without having to convert to scientology, learn GeoDjango, license GIS software, read an 800 page manual on parsing a bytestream from a satellite, etc.
It wouldn't be terribly difficult to mechanize this and get all the data at once, but I thought you wanted to occasionally see how much it rained, not build a service out of this.
Does this page[1] not contain what you need?
I share the parent's sentiment; finding easy access to good weather data is hard.
Do you want your side project to be an experience in learning how to parse obscure formats and coordinating obsolete dependencies, or HTTP GET some JSON and immediately do something with the data?
They're literally giving you all of the data in a plaintext format with documentation, and you're still complaining?
Sure, having everything handed to you on a silver platter is nice, but why should they do your project for you?
You know what you get with a format that's not "modern"? A huge, mature set of tooling that works. For instance, without trying because just responding to this whinging is barely worth the wear on my keyboard, I'd be disappointed if OS X wouldn't cheerily open the file, and unarchive it into the original directory structure simply by typing "open weather_data.txt.gz" from a terminal window. In other words, one could script it. I'd show you how, but holding your hand leaves me with only one hand for typing.
> HTTP GET some JSON and immediately do something with the data
Well, that's the magic of JSON: any code I write just wondrously knows what to do it without me telling it how to parse it. How can poor ol' ASCII text possibly compete?
Why can't we ask for nice things, even if they are "free"?
How are you going to get a kid interested in programming historical weather analysis when he's got to figure out how to configure an entire stack just to query your dataset. If you really want my opinion, this should be an AI problem, not yet another tedious MVC application.
Furthermore, when I jump into projects, I am most definitely appreciative that there is a responsive system on the other end, no matter how much I'm getting paid. I'm not some noob and yes I have imported gigabytes of minimally-structured data into my own applications, but I don't want to have to do that every time. Especially just to tinker.
If you're not thinking about how your software development experience can be made quicker, more direct, and less painful, I pity you. Shit on JSON all you want, but I'll take a populated native object and a blinking cursor over FTP'd text files any day.
I'm not sure this particular hill is worth dying for. Remember, text was the universal transport long before JSON was (and probably will be long after).
wget -O - http://cdiac.ornl.gov/ftp/ushcn_daily/state01_AL.txt.gz | gunzip -c | in2csv -s schema.txt | csvjson
I once bought data from them for couple of hundred dollars.
IBM is in the business of providing data about a whole lot of things. I would expect the cost for access to this data to go up, not down -- the majority people who care about how much it rained in the last week only care because they can make more money by knowing that.
There is also a link to the python script I made to scrape it
https://www.ncdc.noaa.gov/data-access/land-based-station-dat...
$ curl -s "http://www.wunderground.com/history/airport/KMSP/2015/09/31/MonthlyHistory.html?format=1" | grep -v Precip | cut -d "," -f 20 The company also has a popular mobile app, which Mr. Kenny said
collected barometric pressure from its users’ phone sensors.
Millions of locations are turning in information 96 times a day.
permission for location services in general
Also if I'm spending most of my time indoors in buildings, how much of it would affect the readings as compared to when I'm out in the open?
Indoor vs. outdoor isn't really a problem most of the time - the pressure is basically the same inside and outside. A bigger problem is fast altitude changes, which make the absolute reading from the barometer very difficult to interpret safely.
One of the reasons that it took so long for TWC to finally do this is because of the concerns you just voiced: that having this data might actually make the models worse, if they don't take enough care to quality-control it first.
A lot of the research done by scientists (like Cliff Mass at the University of Washington) is how to assimilate this data in a reliable and trusted way. It is likely that The Weather Company does not know how to use the exact pressure readings from the barometer yet (nobody does). Instead, they will probably use the trends over time (say 6 hours), because those are more resilient to noise.
Well, four years later here we are! Finally. We are at the beginning of a revolution in how we understand our atmosphere. There are now billions of internet-connected barometers that can be used in weather models - this should provide a revolution in our ability to forecast short-term mesoscale events like thunderstorms and tornadoes, among other things [2].
[1]: http://gizmodo.com/5851288/why-the-barometer-is-androids-new...
[2]: http://cliffmass.blogspot.com/2015/10/apple-computer-could-r...
Some startups working on this are Sunshine, PressureNet, Weathermob, WeatherSignal:
The biggest deal is to collect the data. After data arrives, they can then look for any patterns they need. Filter anything that seems wrong.
In the work I've done on this problem we've never been too concerned about inside vs. outside. A much bigger problem is small altitude changes that mess with the exact readings by huge margins, but again, if we're focusing on trends instead of the immediate exact value, we can smooth out the noise relatively easily.
However, to actually answer your question, an app can make a very good guess about whether it is inside or outside. Key sensors and metric for this are: ambient light sensor, humidity sensor, GPS location / barometer. During the day it is normally brighter outside than inside, and during the night is normally brighter inside than outside. So that's clue #1. Clue #2 is to check GPS to see if the user is moving - if you are moving rapidly then you are probably not inside. And if the barometer is rapidly going up and down, but GPS is not changing, then the user is probably inside (elevator, stairs, etc) and that's clue #3.
Again, none of that is necessary to solve for this particular problem, but it's interesting.
Statisticians (et. al) are well versed at receiving a whole bunch of messy data and figuring out the value that they want to observe.
From a rough perspective, I think it would look something like this:
1) There will be a large concentration of people reporting pressures that are on the low side, this is the true outside pressure.
2) There will be clusters of people reporting high pressures, and that will be inside buildings.
3) To ignore funny circumstances where large amounts of people are reporting lower pressures than the true outside pressure (I'm not sure why this might be, but this is not my domain of expertise) you can compare it to the pressure that people are seeing around that area, and to official pressure readings.
However, it is now mandated by the FCC that all barometer-carrying smartphones will report your accurate altitude information to 911 emergency response if you make a 911 call from your smartphone. So the location tracking element of the barometer is a very real concern for a lot of people, as it can tell a tracking company what floor of a building you are on at any given moment.
However I think it's quite clear that weather apps, and The Weather Company, are much more interested in the local atmospheric trends than they are about your current floor.
Edit: FCC citation: https://www.fcc.gov/document/fcc-adopts-new-wireless-indoor-...
How can this possibly be true? Wouldn't the pressure around a couple of square meters simply equalize instantly? You're not saying "within a mile" you're saying "to within 1 meter", and I don't get it.
The barometer is sensitive to altitude changes much much smaller than a meter, probably something like 0.1 meters. So if you are a person, with a smartphone, and you are in a tall building, the barometer can assist with determining which floor you're on. Other sensors would need to be used for calibration (since the changes weather will affect the floor-level) and it doesn't help with the x,y coordinates, just the z. But other location systems are now very very good at x,y and they mostly lack the z.
So by adding the high-accuracy relative altitude capabilities of the barometer to the existing x,y systems, you can get extremely precise locations.
Is there a reason why we haven't seen a thermometer in a phone so far?
In fact, every phone does have a thermometer, but that is not usually accessible by developers and it does not measure the actual air temperature - it's used by the phone to make sure it doesn't get too hot from the battery or CPU. We might not see many app-level thermometers get added to phones because the data is extremely difficult to use. For example, most of the time it will measure the temperature in your pocket or in the room where the phone is - and even then, the reading will be radically affected by what the phone is doing (browsing the web is going to cause a different temperature reading compared to watching a video, compared to stand-by mode).
I do expect that we'll see more hygrometers and thermometers and other sensors make their way into phones as time goes on, but I think the primary reason for the slow take-off is because the data is really really hard to use. That's the thing with the barometer - it's very resilient to noise and even when the data has problems, they are fixable.
Wunderground generally feels clean and on-topic.
Weather.com is often filled with "Doomsday scenario with newest hurricane season" and "Check out this massive shark jumping" news stories.
More than that now, I ran into a guy in the Haight wearing a WUnderground fleece about 3 years ago, right after the buyout and told him it was my favorite weather website and that I hoped Weather.com wouldn't ruin it.
He then told me he was on the Weather.com team who led the buyout -- Oops. He still offered to send me a WUnderground hoodie though, so kudos to him for having a sense of humor.
First article I found re: the buyout in 2012;
http://www.nytimes.com/2012/07/04/us/as-weather-channel-buys...
Any extraneous stuff I notice but filter out automatically. I love the site and don't have any of the sort of issues you've described. I do seem to have a higher tolerance for the minor inconveniences of life when I compare my experience to those of the people who surround me, so maybe I'm just lucky. I think you can develop this attitude though, and it sounds like it would help ease your way through the world.
The only issue I described was extraneous noise on the website (most of which has nothing to do with weather). It's just less usable/aesthetically pleasing than the alternatives.
I'm not losing sleep over it or anything. I just use the alternative site instead.
They probably didn't want the competition anymore because I don't think the Weather Channel has learned a damn thing about how to run a clean website since acquiring Wunderground. On the plus side, it appears they have left Wunderground to do their own thing and not interfere.
let's not forget that x86 itself could have been a Motorola 68000 if this company had not decided in Intel's favour. That's the extent to which this company defined the computer era.
The notebook you're using right now will boot DOS, commissioned by IBM. And I'm not even starting on the fact that almost all programming language roots go back to languages for the IBM 360 mainframes.
I'll gloss over the extent to which the architecture of your modern computer is essentially a scaled down version of what was invented by IBM in its mainframe business. Disk? Check. Processor cache? Check. Asynchronous bus? Check. GPU (IBM 8514) check.
It's not unfair to say that this company invented modern computing's engineering, and even if it has (very sadly) lost its way, one can hope that some of the culture of well-financed innovation remains, and this Watson business might be one of those.
With the possible exception of Watson/supercomputing, IBM is a shadow of its old self.
IMO, keeping your company rich, powerful but small (in terms of the number of products and employees) might last longer. But this seems unrealistic for real business in the real world. Reason is, the undead like AT&T (my colleague always talks about his experience working there and comments the Lab in NJ is a ghost town) will continue to live because of the deep root of AT&T in our American life.
It's a paradox and sarcastic. You either prosper and everyone admires, or you remain undead and disgusted by many. And to both you have to be big...
-- EDIT --
To the commenter: Most of the empires and dynasty power degraded after 50-100 year. That's the life span of one or two emperors. Almost all of these empires continue to split until full conquer. During splitting, you may have a couple more strong emperors in place. This is like HP splitting, or IBM selling its desktop and laptop division to Lenovo. They are merging or cutting arms to survive.
Roman might be one that last longer before a big split than the rest. In the ancient Chinese history, most dynasties you have heard of would lose dominance and prosperity after about 100 years, mostly due to incompetent leadership. In the case of the Mongolians, they conquered half of the world in less than a decade, but they divided into several ruling regimes to avoid big civil war. The one that controlled ancient China (called Yuan dynasty) went defeated to the Han people in less than a hundred years. The last dynasty in China, the Ching dynasty, did enjoy its dominance for four different emperors, and then rapidly declined due to foreigner invasion, corruption and severe political conflict, and unaccountable spending from the previous administration (plus many social issues).
You make a massive error in your analysis, and that is to limit your perspective to Chinese dynastic political cycles. You ignore the idea of geographic empire building, where there is ample evidence of multi-century empires, and we are not talking only Roman. Also Byzantine, then Ottoman, British, French, and Spanish, not to mention Persian or Zulu. On China, while internal politics mean dynasties come and go, Chinese territorial integrity has been unchallenged for many centuries, testament to the fact that while the individuals, families, dynasties, may come and go, the underlying political structure is very resilient.
In ever single one of my cited cases, the nation-level structure is multi-century in longevity, and beats by an order of magnitude any commerce-driven corporate leadership structure we have ever seen.
So. My point is that corporations as a structure for human interaction/cooperation are deficient in that they are not able to deploy force. Only money. The latter is not long lived without the former.
Except if IBM had it's way, we'd be using PowerPCs, not x86 and ARM.
IBM existed for 50 years before any of the products you mention. It did financial services before that.
Edit: I slipped and said Intel instead of IBM!
IBM used to make pretty good laptops but sold the laptop part off (Lenovo)
IBM used to have a pretty decent server line or lines, though they sold a good part of the Intel stuff off (now Lenovo for the server line, but had sold off many other parts earlier, such as storage, and started using more vendors)
There were things Software Group did like Websphere, which I imagine still does good money, but why you'd actively want to seek it out over free alternatives I have no idea. They also have DB2 in database land. Software Group used to be cooler and have better resources than hardware groups, or so we thought. I have no idea how well they are doing.
They also got rid of microelectronics and who knows what else. In general, innovative parts of IBM get sold off, and they keep the services parts? Feels that way.
Their margins on services and people are not good, so they also have the reputation of restructuring/laying-off like crazy and seemingly at random, with lots of pressure on stock price.
I was there in 2001-ish when they scrapped pensions, though the thought of pensions actually existing is a novel concept today. It used to be a place where people thought they could work forever, and that is decidely no longer the case anywhere, it shows how the views of what IBM was changed rapidly over time.
They also increasingly pushed for a contractor-heavy work force for more ability to make cuts and reduce benefits.
IBM doesn't offer any kind of pension as a benefit? What does everyone do? Buy privately? I think that's even illegal in some countries like the UK.
Basically most companies in the US offer some form of 401(k) with a selected amount of supported funds to choose to invest in, often with some degree of employer matching. Sometimes there are vesting requirements on employer matching. Startups are sometimes likely to not offer any matching.
The difference is pretty much what it sounds like. If you had a pension, you knew exactly how much money per year you'd make in retirement (the benefits were agreed to ahead of time). With a 401(k), you know how much exactly you're contributing and how much is in 'your' account, but the value of your retirement accounts in the future is dependent solely on how much you contribute and your personal investing prowess.
As companies and public entities underfunded pensions over the past 40 years, pensions started going bankrupt or dramatically cutting contractually obligated benefits that were to be paid to retirees, so companies began switching to employee-owned plans.
Shifting the burden of investing in retirement to the employees rather than the companies has its benefits but I think we're going to see a lot more elderly poverty as the boomers retire since on average, people are terrible at investing.
I think you're right on a macro level, the tax burden is being shifted to the younger and less politically active, but based on actual legislative action, it seems defense and social issues are more important to the elderly than retirement benefits.
For IBM, any accrued pension remains in a managed fund until you quit/retire. Most employees had a 401k option at the same time as a pension - you pay 6% of your salary into it and the company matches half that, another 3%.
If you didn't make the pension cutoff, IBM now matches the 6% plus some more depending on what date you started. I think it comes out to something like 14% of base pay for most employees. In a 401k, the individual has much more control over investment direction, whereas with a pension you had none.
They made worksplace pensions opt-out recently as well due to the massive scare we have over so many people of working age not having saved anything for retirement.
I was automatically opted in to my workplace pension, into which I just pay 5% per month which my employer matches.
I'll likely increase that amount next month though.
However they are currently focusing on cloud, for that they have Softlayer and BlueMix.
In terms of other products MQ is still pretty big, so its CICS and as you said Websphere (which I believe has a free offering?). They still have the Power business and the Z server offering. They have SPSS, Tivoli, Lotus and Rational. Then they also have GBS (Business services) and GTS (Technical services).
Obviously they also have Watson. Some of the Watson services are available on BlueMix and they're pretty good. Currently I think Watson is branching out into all sorts of new areas. Health care is a big area of interest.
They also have ETS (Emerging Technology) which does client stuff, research and the like.
There is also a fairly big IOT department (But I don't know what happens there).
They also broke out an additional "Education" unit within the IoT business unit, ostensibly to compete with PTC Thingworx, and Intel+Arduino, Raspberry Pi etc. in the student/teacher/maker market. [2]
[1] http://www.businessinsider.com/how-ibms-3-billion-investment...
[2] https://www-03.ibm.com/press/us/en/pressrelease/47660.wss
Had them sitting in front of the filers for two days before we yanked them out. Then it took them another TWO YEARS to send us boxes to return the systems despite repeated requests.
Novel. Hmm, I was working in software industry for 15+ years never seen place haven pensions. Do you mean 401k plans instead?
There were a number of lawsuits over the conversion: https://www.google.com/search?q=ibm+pension+lawsuit&ie=utf-8... with a mix of results for employees and the company.
Do other companies in tech have pensions? I just never heard of that before. I guess maybe in the context of the US Post Office (how Republicans forced it to fund retirement for so many year ahead, in hopes to bankrupt it and show how government services are ineffective and broken)
About the only organizations that still have them feed at the public trough. And they're having problems with funding. Here's a recent story about a state transit agency. Its pension plan is over $800 million in the hole. And that's just one agency in one state. https://www.bostonglobe.com/business/2015/10/27/pension-liab...
Government pensions are worse in that few government entities ever set aside a truly funded pension fund, using the argument that as a continuing entity theyd always be able to top off the fund with additional tax revenues if the investment projections did not pan out. In practice no elected official will voluntarily vote to raise taxes to meet the obligations to top off oension funds, hence the slow unfolding train wreck with public pensions in Illinois.
A general failure of any defined benefit pension seems to be the dependency on the sponsoring organization as a continuing, growing, profitable entity. No accomodation for changing economics, diminished industries, or reduced workforce.
They sell services, main frames, cloud, software used by enterprise (storage, project management, ...) provide consultants, and even have a bank! that companies can get financing from (when they buy all that expensive other stuff from it).
IBM has 12 research laboratories worldwide, bundled into IBM Research. As of 2013 the company held the record for most patents generated by a business for 22 consecutive years.[16] Its employees have garnered five Nobel Prizes, six Turing Awards, ten National Medals of Technology, and five National Medals of Science.[17] Notable company inventions include the automated teller machine (ATM), the floppy disk, the hard disk drive, the magnetic stripe card, the relational database, the Universal Product Code (UPC), the financial swap, the Fortran programming language, SABRE airline reservation system, dynamic random-access memory (DRAM), copper wiring in semiconductors, the silicon-on-insulator (SOI) semiconductor manufacturing process, and Watson artificial intelligence.
Things.
A question: how has the accuracy of weather prediction improved overall?
Here is a recent New York Times article on that topic, with emphasis on hurricane prediction:
http://www.nytimes.com/2015/10/03/upshot/hurricane-joaquin-f...
"Over the last few decades, faster computers, superior models and new data have allowed all weather forecasting to improve, by a lot. But the United States hasn’t quite matched that effort. It didn’t invest in computing power and models that kept up with the potential for better forecasts."
Does anyone know if organisations like Home Depot, for example, roll their own solutions or rely on 3rd parties to help move things like generators into stores ahead of storms?
http://www.wsj.com/articles/SB100014240531119047166045765424...
Hope they enjoy the Blue Washing(TM)
Just send me a check.
> The company also has a popular mobile app, which Mr. Kenny said collected barometric pressure from its users’ phone sensors. Millions of locations are turning in information 96 times a day.
Fascinating...The new-weather startups like Sunshine and PressureNet are going to be watching this closely.
"While we can typically predict general weather patterns up to 3 days ahead, predictability for detailed local weather such as rainfall or fog formation is much less."
see more here http://research.metoffice.gov.uk/research/nwp/ensemble/conce...
IBM Board meetings notes: "We should buy the Cloud" "What?" "You know, the Cloud" "Why?" "We want to be the biggest"
buys weather channel