Hacker News Reader

Comment by dougmccune | Hacker News Reader

Almost all the weird spikes in the SF crime data are data issues, not real crime trends. If you're naive about it you'll get to the conclusion that one spot in SOMA along Bryant is the crime capital of SF, except that's just where the Hall of Justice is where tons of crimes get geolocated to. That doesn't mean they actually happened there. Same thing for many of the police stations and hospitals throughout the city. The top 5 locations in the data aren't "real". The fact that Monday, the 1st of every month, and Jan 1 are the maxes too doesn't strike me as legit.

The article mentions some of this, that "It could also be an effect of how this data is collected". That's my guess for every insight in this article.

View on HN

Crime data is always awful like this.

You can really only do useful analysis in a single jurisdiction, as there aren't any meaningful standards for data collection, and different jurisdictions vary dramatically in their reporting for lots of different reasons.

If someone gets shot in the middle of a large park/campus/highway/bridge/etc, what's the location? The answer varies!

This impacted a friend who is a bar owner -- some municipalities have a point system where your property gets a classification based on crimes and other events that happen nearby. He had issues during his license renewal because a number of crimes (assaults, vandalism) took place "within 500 ft" of his establishment. In reality, the scene of the crime was about 3/4 of a mile away in a park, but his business was about 400 ft from the nearest boundary of that park!

I did a followup to my analysis of this dataset by plotting each arrest location as a single point on a map. (http://minimaxir.com/2015/12/sf-arrest-maps/ ). The cluster locations of those are meaningful and not randomly geolocated. (All of Tenderloin and 16th St. Mission).

The major clusters in the TL and Mission are real, for sure. But as one example, the Hall of Justice at 850 Bryant still sticks out in a lot of these. You can see it in the version by crime type where plotting the points. It's the spot just south of the highway before the Bay Bridge. In your hex binned map it's one of the pink hexagons in every crime type map.

So yeah, the bad data effect goes away somewhat when you aggregate the data by larger areas. And it doesn't make the overall trend of TL and Mission crime rates invalid or less obvious. But it's still visible in all your maps.

Hey, I helped edit the blog post. Definitely agree that the different spiky behaviors are driven by data collection, but that's the beauty of real-world datasets! It would be interesting to learn if it was bulk first-of-month reporting, that crimes where the day wasn't specified get marked as the first, or some other phenomenon. The next step might be to interview police officers and ask :-)

Here's the top crimes broken down by the first of the month vs the average for the rest of the month - some crimes are only reported (or never reported) on the first of the month, and "FRAUD" for example is twice as likely to be the first of the month: https://sli.mg/lHjo83

Another thing you'll notice in that image is that sex offenses get treated differently than other crimes. My understanding is that in an effort to protect the privacy of the victims, those crimes never have accurate geolocations, and in your image it looks like they are always reported on the 1st of the month, so they probably also never have accurate dates/times.

There are a LOT of these types of idiosyncrasies in crime data. If you do end up interviewing police officers in SF ask them why they don't publish homicide data as part of their crime data (I genuinely don't know the answer).

Reply on news.ycombinator.com