Comment by ALittleLight | Hacker News Reader

Comment by ALittleLight | Hacker News Reader

Parent Full thread

I like writing web scrapers and this is an interesting project idea. If I understand right you are looking for volunteers to write scrapers that would take a police department, scrape the PD website, and download any PDFs or documents that gather data about the police department. Is that right? If so, I feel that's not super clearly communicated - I had to look at a couple example scrapers before arriving at this guess.

I do have a few questions too:

1. Will this scale? One problem with scrapers is that they break when people update their website. I'm imagining this problem multiplied by 18,000 and compounded by each scraper potentially being written by a different volunteer.

2. Where are the scrapers getting run?

3. How do the documents that the scrapers collect get transformed into usable data?

4. It seems to me like a scalable solution would be a standard to report data, a law to compel police departments to follow that standard, and then a system to collect that data and make it available. Do you work with police departments at all on data reporting?

1. I replied to the parent comment here; our answer to the scale problem is to recognize that people doing web scraping are as decentralized as the police. Our goal is to empower people who have questions about the police to answer them.

2. You can run them locally. We're not running the scrapers anywhere, or storing extractions anywhere.

3. This is a big, big question. Right now, the answer is dependent on the use case. Rather than trying to make the world's biggest database, we're going to respond to community needs and build this kind of thing as it comes up.

4. https://measuresforjustice.org/ is doing something like this! We're interested in creating incentives for police departments to make their data more accessible and transparent.

Not to be too rude/negative/mean - but it seems like a big concern to me that a "police data" project that's existed for three years, has a couple employees, has some funding, etc - doesn't really seem to have any police data. If I wanted to write a scraper to gather documents off of police websites and run it by myself and store my results locally - I could. What does your project add?

What I would expect to see is something like:

1. Here's what data we want from each police department each day. Here's what value you should use to indicate that data is not available.

2. Here's a list of police departments. Write a scraper. If it passes tests to show it's generating valid data, and code review, we will run the scraper in a daily basis storing the output in this database.

3. Here's how you can query our database.

It's ok, this feedback is how we understand what our work looks like to other people, and how we improve!

We've only had paid staff for about 6 weeks. We've run several experiments and started from scratch a few times over the years. We've been slowly inching the project forward in our spare time; I was the only volunteer for much of that time, and I can't even code!

1. Sounds great, we're building something like this.

2. Even making a list of police departments is a big challenge. We've made a good start but have work to do.

3. Yep, soon we'll have something to query.

Reply on news.ycombinator.com