City of Boulder Open Data | Hacker News Reader

City of Boulder Open Data

open-data.bouldercolorado.gov

114 pointsmooreds64 comments

Just be VERY careful when using any of these open data sets. Datasets will undoubtedly be filtered by some unwritten criteria. It's best to use these datasets as a sanity check for some things, with follow-up FOIA requests for the serious work. And even that will be filled with similar filtering. Source: spent the last ten years working with open data and FOIA in Chicago.

Also, here's a tweet-fitting one liner I wrote that downloads all Socrata CSV datasets across all gov instances:

  mkfifo f;echo a>f&xargs -E, -a<(cat f) -I{} bash -c "wget http://api.us.socrata.com/api/catalog/v1?scroll_id={} -O-|jq -r '.results[]|.resource.id+"\","\"+.metadata.domain+"\"/api/views/\""+.resource.id+"\"/rows.csv\""'|tee <(cut -d, -f2|xargs wget -P{} --content-disposition)|tail -1"|tee f

It's not just the filtering. It is also that many datasets are full of errors. I wrote a data analytics tool that I would test out on a bunch of open data sets like this one. One of my favorites was the Chicago crime data. The tool could quickly find (and fix) a bunch of errors. I tried contacting the people who published the crime data and even offered to show them how to use my free tool to clean up the data. They could not have cared less that their data set had large numbers of errors in it.

interesting! what sort of errors did you find and were there any sort of take aways your had other than lack of care by the data provider?

Each crime report would specify (among other things) what crime was reported and what city block it had occurred. Using my tool, it was easy to analyze specific crimes over time for specific areas. One of the problems was their database would have multiple spellings for the same location which made it much harder to do the analysis since you had to account for that.

This is a common problem, especially for "crime" datasets and very emblematic of the kind of data capturing processes for each reported "crime" (really, they're some of the worst datasets because of the extent of imperfect information and incentives to capture some "crimes" some vs others). Send it through a geocoder like geocodio to resolve most of that problem. It won't be perfect.

This strikes me as a Wikipedia Article type caveat. Sure the warning applies to certain categories of article/data, namely the ones where people are jockeying to have the authority of an encyclopaedia / data to reinforce their political stance, but on any sort of non political topic neutrality is a non issue. I fully anticipate that say, police and crime data by demographic shouldn't be trusted. Other data, like say pounds of recycled garbage collected per census tract, is probably fine.

I mean, sure? I'm very much saying this for police datasets, where for example, a "crime" database might not include homicide information.

And no, I don't think it's fair to say that the amount of pounds of recycled garbage collected per census tract would be remotely accurate. Too many private companies with private incentives. Just a taste, for example: https://www.wbez.org/stories/bga-investigation-reveals-major...

What FOIA request methodologies would you recommend to ensure you’re getting the most comprehensive data possible out of an agency or government body response?

One of the most productive things I've seen is to request for "data sufficient to show" rather than describing the data you're looking for. That way you have a bit more wiggle room in the space of interpretation. Another way is to just look up forms ("filetype:pdf site:yourmunicipality.gov" in google works wonders) that are online -- if filled out reports have text filled in by a computer, then that information was likely filled into a database, and the form you're looking at is a product of a multitude of SQL queries. In Illinois, Hites v. Waubonese compels an agency to run a SQL query if data is represented in a database, so being explicit about the existence of that law can be helpful. I'm sure similar can be said about other states. Also, be sure to explicitly request your data in an excel compatible format (some states legally require it).

Also, have a look at the Reporters Committee for the Freedom of Press's FOIA guides per-state. https://www.rcfp.org/open-government-guide/

Incredibly helpful, thanks for the reply. These tips will make their way into my requests immediately.

Anytime!

Just curious: for a municipality with limited resources and very minimal understanding of data governance, why should a city government follow in the footsteps of Boulder?

Do the missteps of Boulder make the case for other municipalities to not pursue open data in a similar way, given FOIA is sufficient?

These are surprisingly common for local governments these days - this site lists 598 of them: https://dataportals.org/

Here's another collection with more than 3,700: https://data.opendatasoft.com/explore/dataset/open-data-sour...

My local city government also recently published a detailed interactive online GIS map of the city, including all lot boundaries, buried utilities, etc. It's great to feel like the public can easily access what is meant to be public record, rather than relying on for-profit third parties to make it available to us.

I really like the ESRI API's (much faster than Socrata). This is a crazy url, but they do allow for different SQL operations (including spatial operations) right in the API. Here is top 10 crime street segments in Raleigh for example:

https://services.arcgis.com/v400IkDOw1ad7Yad/ArcGIS/rest/ser...

If you go to https://services.arcgis.com/v400IkDOw1ad7Yad/ArcGIS/rest/ser... you can putz around in the query engine to figure out the API strings (easier than reading the ESRI docs anyway!)

Looks pretty standard for a small city open data site.

Yes but it's Boulder, the promised land, according to here

Been there. It's very expensive and crowded but it's really nice being like 15 minutes from some of the nicest hiking trails in the country.

I live in Boulder. It's certainly expensive, although more reasonable than some of the other popular tech cities. I honestly don't think of it as crowded though. There's tourist spots to avoid, but plenty of open space for all.

I found it crowded. I guess it depends on where you're coming from.

On weekends, It takes 3 hours to make (what should be) a 1 hour drive at 5AM going to the mountain to ski, very difficult to find parking at most trailheads, camping spots are pretty much swamped on the peak to peak...etc. etc.

But I guess if you're coming from L.A. or somewhere that kind of traffic isn't too bad lol

And that's not all Boulder traffic, I'm sure there's a lot of Denver and surrounding city population crush, was just miserable on the weekends I found.

Oh yeah, definitely not disputing the I-70 traffic on weekends, it's bad!

>and crowded

Oh yes, with the 5 story limit of the height of the buildings - and a year over year growth of 1%.

Super crowded.

Hence super expensive.

I don't know why our city council and planning boards waste so much time on "affordable housing" when it's clearly not fixable.

The economics don't make sense to have a highly desirable place to live with a highly constrained housing supply and also make it affordable. The affordable housing plans here basically boil down to a lottery.

>The affordable housing plans here basically boil down to a lottery.

They don't "basically", Section 8 housing is literally a lottery.

https://boulderhousing.org/

Wouldn't 5 story limit's on buildings just make it feel more crowded by constraining the growth?

I don't follow. "It feels crowded because not everyone can live there?".

It feels crowded because instead of being able to increase density vertically the density has to be increased horizontally.

The CAGR is 0.1%, 0.6%, or 0.4%, over 10, 20, or 30 years respectively.

CAGR is misleading because it tracks population when what is more important is household growth. Household size has fallen from 2.63 in 1993 to 2.51 today, which doesn’t sound like a lot, but means you need 5% more houses to hold the same number of people.

Yes, and there is a general misunderstanding, mostly a willful misunderstanding among older, established homeowners, of the fact that you need to build more housing just to keep the population the same, especially when older people are squatting on empty family-sized homes.

The city and county GIS is exceptional. Backpacking and camping permits are egalitarian, but you need to plan months or years ahead.

The GIS has been really helpful when we've had close by fires too

Boulder is the worst best small town in America. You know what I mean.

I lived there for 6 years and halfheartedly agree. It's a very nice town in terms of general quality of life - definitely the best I've ever experienced, and I've lived all over America. But there's a pretentiousness in the air that can be utterly nauseating at times. The "Boulder bubble" is real.

I enjoy some of the pretentiousness, namely the physical fitness and cycling fanaticism. There are some really, really tremendous athletes there.

But Boulder lost me at the rock worshipping yogis that appropriate "brown people cultures" into their daily routines while chastising white people who celebrate Cinco de Mayo. They're fucking everywhere.

lol, well said. The trustafarians, as I liked to call them.

Also yeah, the athleticism there is off the charts. I'm a fairly lazy dude and I was in the best shape of my life, simply by biking around town.

It's pretty, and it would be nice to be that close to the mountains, but I can't stand the political climate in Boulder. Denver is fairly blue but still diverse enough that I feel comfortable here even though I don't personally share a lot of those views. Boulder, though... it's like people are trying to build the Berkeley of Colorado there. Which is fine, live and let live. But I would never want to live there.

I don't know, but based on this entire thread, I think this was one of the places that was popular for people to move from cities during the pandemic, when everyone was working remotely. Its fascinating to me that for decades people have been moving to the same cities from places like Boulder (and many such all over the country) for opportunities/careers/lives and now a few chose to move from the cities to these small towns. While I understand that people do not like change but how is this any different, its still people making a choice of moving to a different place in the country which they think is better for their lives? Isn't this NIMBY, its fine as long as everyone moves to NY, but not here.

Boulder is more accurately described as a neighborhood or region of Denver.

If that’s not obvious today it certainly will be when the staggering construction in Arvada, Broomfield, and so on, has filled in all of the areas that could plausibly be used to argue otherwise.

Part of the reason it's so expensive here, though, is the swath of city and county owned open space around the outskirts of town. Maybe some day it'll get developed, but I think it'll be a long time before that can happen.

It's not a small town, but rather a suburb of Denver.

It's 25 miles away from Denver.

Boulder probably has enough of a unique character that thinking of it as its own thing rather than just a commuter suburb is probably reasonable. But there are certainly places within 25+ miles of Boston, say, that are reasonably described as suburbs and probably exurbs out to about 50 miles. (It's how ESRI would generally describe them.)

It sort of depends. To use another Boston area example, no one would reasonably consider Portsmouth or Nashua in NH as suburbs even though people will go into Boston for a night's entertainment and probably fly out of the airport.

What's your point? Denver is a massive sprawling metro.

There's still a lot of open space between Boulder and the surrounding cities. I'd argue Denver ends in that direction at Broomfield.

From my limited perspective there isn't much space.

I'm a bit of a stranger to the area- right now I am in a Starbucks in (I think?) Boulder. But I came out to work as a tech with an audio company I contract for to do concerts up at some ski resort. And I am sleeping in my truck front of the Warehouse in Broomfield. But the guy who hires me has a house in Boulder where he keeps a lot of our equipment... and I am often not super clear about when I am just crossing a large green space and when I am crossing into different cities.

So you are probably right, but that's not how I've been experiencing this over the last summer's worth of gigs.

Of course, to me the drive from Boulder to FoCo seems like one continuous city.

Contextually, it might be the fact that I come out for work from Durango, and crossing the san luis valley is what "open space" looks like to me.

So maybe I'm just not sufficiently calibrated for the city yet.

Denver is in a different county altogether. There's a county in between Boulder and Denver counties, saying nothing of them being the same city.

So…? Arlington and Fairfax are different counties than DC. Heck, they’re in a different state and across a river too. But nobody would argue they aren’t suburbs of DC.

25 miles is well within commuting distance for many major metro areas.

Boulder may arguably be a suburb or not. But an arbitrary 25 mile limit isn’t what defines suburb or not.

Brooklyn and Manhattan are in different counties too.

Ask yourself questions like what airport do you fly to and where would you take your kids to the children’s museum if you lived there and you’ll realize what city Boulder is a part of.

>Brooklyn and Manhattan are in different counties too.

You're going to use the most densely populated city in the US for your example?

>Ask yourself questions like what airport do you fly to

Colorado has one international airport - are you saying ALL of Colorado is Denver? Are you saying Newark NJ is in the same city as Brooklyn? I don't know where you're heading, but it's just logical gymnastics.

Colorado has commercial service to half a dozen airports including Colorado Springs, Aspen, Vail/Eagle, and Grand Junction.

And Newark is definitely a part of greater NYC, absolutely nobody is confused by that at all.

I lived in Colorado for years. Boulder is definitely a suburb of Denver. In 20 years Fort Collins probably will be too at this rate.

My old boss when I worked in Boulder (commuting from Arvada) claimed he hadn't been to Denver proper in at least a decade, not counting DIA or the interstate to it. I believe him. Boulder is definitely its own thing.

Too commercial, too many rich brats, expensive, nice access to nature but too crowded. The western side of the Rockies are superior imo.

Nothing is more Boulder than moving to a college town and complaining about college kids.

What cities do you suggest west of the Rockies?

They're talking about the western slope, which would mean roughly Grand Junction to Durango in this context.

Yes exactly!

San Francisco and Los Angeles

This is pretty neat. Wonder if they're using some data catalog under the covers for the search screen or is it all custom built.

It looks like ArcGIS, which is what my city of Halifax, NS also uses: https://catalogue-hrm.opendata.arcgis.com/

It is ArcGIS. Here is their Emergency Operations Center overlay:

https://bouldercounty.maps.arcgis.com/apps/webappviewer/inde...

You get used to refreshing this page during down-slope, windy, red flag days.

Looks like that one is using https://hub.arcgis.com/

Another common platform for these is https://dev.socrata.com

I like SF's. You can plot recent crime and stuff. https://data.sfgov.org/browse?tags=crime+reports

is this associated with vehicle tracking by license plate, 24x7x365, by any chance? In San Diego County it is, same vendors most likely.

Reply on news.ycombinator.com