Hacker News Reader

Comment by cwal37 | Hacker News Reader

Hey buddy, I use this resource[1] all the time (and they're actually housed into the building next to me here at work). It's the long-term climate records from monitoring stations across the country, the United States Historical Climatology Network, and you can get daily data going back over a hundred years in some cases.

For example, I've used it to crank out random temperature histograms in the past[2], and even plotted some long-term averages to decide on temperature and precipitation ranges when going on camping trips. It's really useful if you're willing to work with the data a tiny bit. You're getting csv files, so it's not terrible to work with in any way via Python or R, or whichever data-friendly language/tool you want to use.

[1] http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn_map_interface.ht...

[2] http://btus.us/decadal-histograms-of-monthly-yosemite-temper...

View on HN

So apparently in order to use this I need to enter a NESDIS ID. When I Google that, the first result I get informs me that a NESDIS ID is "an 8-character identifier assigned by NESDIS that becomes the RAWS identification number for use with the GOES satellite." Clicking on 'glossary' in the breadcrumbs then fails to take me to an actual glossary.

I realize that it's theoretically possible to get this working. But it's not easy. What I want is a way to enter a postal code and get a JSON response with the precipitation for the last week. Without having to convert to scientology, learn GeoDjango, license GIS software, read an 800 page manual on parsing a bytestream from a satellite, etc.

Um, maybe to use it in an commercial setting? I've literally no idea what you're talking about as I've never come across that before. I've always just pulled down the station stuff I need by hand (since you can get all the station's data at once), or by some lightly mechanized webscraping. Your comment sounded significantly more chill than what it seems like you might need.

It wouldn't be terribly difficult to mechanize this and get all the data at once, but I thought you wanted to occasionally see how much it rained, not build a service out of this.

Does this page[1] not contain what you need?

[1] http://cdiac.ornl.gov/ftp/ushcn_daily/

*.txt.gz is not exactly a modern data format.

I share the parent's sentiment; finding easy access to good weather data is hard.

Do you want your side project to be an experience in learning how to parse obscure formats and coordinating obsolete dependencies, or HTTP GET some JSON and immediately do something with the data?

Not modern doesn't have to mean difficult to parse, it depends on your personal outlook. You can also download individually customized csv files if you want, which my original link led to. csv and txt files seem eminently reasonable to me when dealing with a huge historical cache of data that's being put out by a public institution for free.

You're shitting me, right?

They're literally giving you all of the data in a plaintext format with documentation, and you're still complaining?

Sure, having everything handed to you on a silver platter is nice, but why should they do your project for you?

Imagine what is going to happen when they discover the entire financial system is built on copying CSV files around by FTP.

Don't forget the emails back and forth and manual fixes and resubmissions from either end when there's a problem causing the CSV to not parse!

I could write a node server that converts all that fixed length ASCII data into JSON faster than you guys can complain.

What entitled nonsense. *.txt.gz is not a problem for anyone of even minimum competence.

> *.txt.gz is not exactly a modern data format.

You know what you get with a format that's not "modern"? A huge, mature set of tooling that works. For instance, without trying because just responding to this whinging is barely worth the wear on my keyboard, I'd be disappointed if OS X wouldn't cheerily open the file, and unarchive it into the original directory structure simply by typing "open weather_data.txt.gz" from a terminal window. In other words, one could script it. I'd show you how, but holding your hand leaves me with only one hand for typing.

> HTTP GET some JSON and immediately do something with the data

Well, that's the magic of JSON: any code I write just wondrously knows what to do it without me telling it how to parse it. How can poor ol' ASCII text possibly compete?

You don't even need the terminal, just double click the file.

Can't script a double-click in Finder. :-) (Well, there's AppleScript...). But, yeah, point taken.

Wow, you guys really missed the point.

Why can't we ask for nice things, even if they are "free"?

How are you going to get a kid interested in programming historical weather analysis when he's got to figure out how to configure an entire stack just to query your dataset. If you really want my opinion, this should be an AI problem, not yet another tedious MVC application.

Furthermore, when I jump into projects, I am most definitely appreciative that there is a responsive system on the other end, no matter how much I'm getting paid. I'm not some noob and yes I have imported gigabytes of minimally-structured data into my own applications, but I don't want to have to do that every time. Especially just to tinker.

If you're not thinking about how your software development experience can be made quicker, more direct, and less painful, I pity you. Shit on JSON all you want, but I'll take a populated native object and a blinking cursor over FTP'd text files any day.

"Shit on JSON all you want, but I'll take a populated native object and a blinking cursor over FTP'd text files any day."

I'm not sure this particular hill is worth dying for. Remember, text was the universal transport long before JSON was (and probably will be long after).

  wget -O - http://cdiac.ornl.gov/ftp/ushcn_daily/state01_AL.txt.gz | gunzip -c |  in2csv -s schema.txt | csvjson

Fixed that for you. And in less space than it took for you to complain.

Try this http://www.forecastwatch.com/

I once bought data from them for couple of hundred dollars.

Reply on news.ycombinator.com