Show HN: StatisticalAtlas.com: demographic charts and maps down to the city block | Hacker News Reader

Show HN: StatisticalAtlas.com: demographic charts and maps down to the city block

statisticalatlas.com

51 pointsjacobn19 comments

I'm one of the developers. This site is all about making public data about places more accessible. We have pages all the way from the national level all the way down to individual city blocks, and everything in between.

The data itself is mostly demographics: age, sex, race & ethnicity, income, employment, and education. We separate by entity and by topic and each page has sections for the stats within the entity, comparisons of child entities, and comparisons of the entity with peer entities.

Everything is extensively cross-linked, both in the maps, and in navigational lists.

The data came primarily from the Census Bureau's American Community Survey, and the 2010 Census.

This is really great. I was pleasantly surprised to see that there was information available about household income at the "tract" level. I had searched some months ago for a data set that exposed high resolution income/poverty statistics at this resolution for Chicago, but could never find much beyond neighborhoods, or as anonymized sets. But I wasn't until now aware of the firehouse that is http://factfinder.census.gov/.

I assume the lack of an API for automating calls to your wonderfully structured data is to comply with the TOS of the census.gov site. Or is this for some other reason?

Thanks. We may build an API at some point, but it is not to do with the TOS. The raw data is so big (and we use memory mapped files, not a DB) that it is a pain from an infrastructure perspective to have available on a web box.

Can you tell us a bit about the software that drives this both back- and frontend? It looks great!

The site was developed in Scala + Java, and the dev-version uses Play Framework, but the public site is statically hosted on S3 + CloudFront.

Each page takes O(10 seconds) to generate and the machine that does the generation needs ~20+ GB of RAM, so for ops reasons we pregenerated all the content.

(I'm one of the devs)

Is this data from the 2010 census, or is this updated to reflect 2015 estimates? (Is the number of 0–4 or 25–29 year olds listed the number that there were in 2010, etc.)

It would be nice if next to each chart (or at least somewhere on each page) there were a more explicit description of the precise data source.

Is there any way in this tool to explore past demographic data, e.g. from 2000 or 1980?

The about page talks a bit about the sources. Everything comes from the latest 5-year American Community Survey (ACS), which is part of the Census Bureau's data collection. Data from that survey is in some sense an average over the years of the survey. They use that wide a window to get the error bars down on the various metrics.

The exception is the city block level data. The ACS does not publish on the block level, so we fall back on the 2010 census. This is why the individual blocks have way less data.

No, we don't have past data.

Enjoyed looking at the stats for Boulder, CO, both as an interested resident and because of the range of useful displays. Cool site!

Umm... I can't seem to find Romania. It would have been nice to mention "U.S. only"

Fantastic resource. I could easly spend hours browsing.

How are you guys funding the development of this? Grants? Seems like something the govt should be providing.

Self funding. Yes, the government should provide something like this directly, but they are not very good at building web sites that are easy to use.

awesome work, thanks for this. Where do you get the city block and neighborhoods shapes from?

Neighborhood shapes are from Zillow (link in the site footer on pages with neighborhoods), the city block shapes come from the Census. They have shapefiles for all their geographical entities (state, county, tract, block group, block)

(I'm one of the devs)

Right! I missed that Zillow reference... very cool to see they provide that data for free. We are working on a project that will be using open street maps and will also have heavy loads of data, we have geo spacial data on Arc Shapefiles but we are not sure yet on how to work with that yet, do you have any tips on that?

We ended up using some random C library to parse the shapefiles into an easier to read text format of our own design, and then pulling them into packed binary files with an external index that we memory map in the code to maximize the speed. The system is all extremely custom and is highly optimized for write-once/read-many. Making even a tiny change to a single polygon means a full refresh that takes hours to complete.

Can't find Palo Alto...

ok, it's in the San Jose area.

http://statisticalatlas.com/place/California/Palo-Alto/Age-a...

Reply on news.ycombinator.com