Ask HN: Can we help the police sort through the photos of the Boston Marathon? | Hacker News Reader

Ask HN: Can we help the police sort through the photos of the Boston Marathon?

116 pointsjbaudanza80 comments

The police are going to need to sort through and catalog thousands of photos and videos. They may not be prepared or have the infrastructure to do this efficiently. Maybe we can build something to help.

Some ideas:

- A central place for people to upload photos and videos from the event

- A system to sort by time and location of the media

- Duplicate detection of photos and videos

- A forum to discuss the photos

- A way to crowdsource tedious tasks. For example: See if person X or item Y is in any of the photos of videos.

I'd love feedback on this idea. Would this be helpful? Does it already exist?

Edit: Getting lots of great feedback. If you're interested in helping, e-mail jon@jonb.org.

View on HN

One way I see one could crowdsource the problem without turning this into a witch-hunt would be to collect as many pictures as possible and try to recreate a 3D model of the scenes at various timestamps. Something like,

  * Go on project website.  There's a 3D mesh of the area.
  * Upload your picture:
   - Select a time frame you think fit your picture.
   - Navigate to the location you took your picture from.
   - Try to apply the picture on the mesh as well as possible.
  * Or, look at what people posted:
   - Try to correct other's time-positioning, or
   - Try to correct other's spatial-positioning
   - Fix the mesh for missing things.
  * Investigators can use that to look for clues on the scene.

Like OpenMap or Google Maps where people can send corrections. Of course, I don't assume that this is trivial to implement. Just, an idea of how such future problems could be approached.

Microsoft's photosynth[1] sounds like a good fit to automate much of what you're describing (without the time function).

[1]http://photosynth.net

If anyone is attempting to build something like this, I'll just note that automating this process (spatio-temporally registering photographs and reconstructing 3D models of cities as they change over time) was essentially the topic of my PhD thesis, but on historical time scales (1860s-2000s) rather than minute-to-minute.

Slides from my CVPR 2010 presentation here: http://www.cc.gatech.edu/~phlosoft/files/schindler10cvpr_sli...

Videos: http://www.cc.gatech.edu/~phlosoft/

The best way to get a system running quickly would be to use Noah Snavely's Bundler project for the 3D reconstruction: http://phototour.cs.washington.edu/bundler/ and, at first, trust most EXIF tags for time.

Super interesting, thanks for sharing!

Northrop Grumman makes a product which collects the images and automatically stitches them together to allow law enforcement to record where bad people come from before a bust. http://www.reuters.com/article/2012/08/30/us-northropgrumman...

If only this plane was in the air surveying all of Boston before the bombs went off <sarcasm?>

I understand you want to help, but i would have some serious reservations about building a proof of concept crowdsourced surveillance analysis platform and demonstrating whether it's viable. There are an incredible array of surveillance tools involved in the fight against terrorism. I'm not sure encouraging the general public to actively participate in a domestic intelligence platform would have a net positive impact on society & our basic rights.

But given that there are no technical limitations for this being developed, it's safe to assume that it will be developed at some point, if it has not been already.

Wouldn't you prefer that something like this exists in the public domain?

Just because a given surveillance/warfighting/CNO technology COULD be developed doesn't mean that one should immediately go do it.

The only practical way of making something like this work would be to collect all the photos taken ~2 hours before the bombs went off near the finish line. You could do this by searching Facebook, Twitter, and Instagram for geotags and keywords/hashtags then sorting by date and time. Dump all the photos into a database and do your best to sort by time (duplicates shouldn't be much of a problem). I don't think there would be too many photos, probably less than 5,000. Then see if you can find what the bombs looked like before they went off, then go back and find when they first placed.

If anyone is interested in doing something like this, I can help, so let me know.

Thanks gusgordan! Can you e-mail me at jon@jonb.org?

Perhaps this could be combined this with the information below and we could search pictures taken prior to the blast for black duffel bags that might contain 6 quart pressure cookers.

I don't really like the idea of putting a lot of photos of traumatized people on internet, photoshit happens, the last thing that those sons, women, family could merit is to be lynched on internet.

...but

If you know or suspect that the bags were black, a reasonable solution that maybe could have success without to blow up all the remains of privacy could be:

1-to create a filter that makes a copy of the originals AND change all colors of the bunch of photos in a named directory to i.e. soft blue... EXCEPT a small range of black tones.

2- Thus, you could check or search then in the bunch with another script what photos have those suspicious black tones. And you could choose to move those photos to a different dir for a preliminary analysis, maybe saving a lot of time.

2- or you could search for photos taken from same place in different times (very frequent in a competition) that have black tones NOT changing of place in a specific time. People moves, claps, node, etc, bags with bombs don't.

3-You could refine the script to permit to enter any desired tone also and mask all the other colors. The goal is to find quickly a bag, but keep the privacy of innocent people and victims that do not deserve this.

I thing that should not be much challenging technically.

They shouldn't be too traumatized in the hours before the blasts ... and that's when we'd be looking for the black duffel bags.

They are traumatized now, that's for sure :-(

To be exposed on internet again and again, or to put pictures of happy friends and members of the family that now are in an hospital will add a lot of unnecessary suffering and probably will enlarge the "satisfaction" of the authors, the "success" of the operation, and I'm writing this with a very angry face.

We have to avoid this.

If you have a photo of the marathon, any photo, not only of the finish line, send this photo to the police.

To care and show respect and love for the victims, dead and alive, should be a priority, in the same level as to put the authors in jail. (only my opinion).

A similar technique was used to identify suspects in the 2011 Stanley Cup Riot in Vancouver, Canada.[0] With people using the footage and pictures of the riot and submitting identities to the police. I'd be interested in seeing similar techniques used else where. It seems to have worked extremely well in the case of this specific riot.

[0] https://riot2011.vpd.ca/

This thread reminds me of the "hunt for things on Mars" online tools https://www.zooniverse.org/project/planet_four.

You could basically tag all the specified features in images (in this case it would be faces, bags) and add special comments if you thought you saw something special (a person you recognised, a bag make, an apparent clue).

That would crowdsource the facial/bag recognition and identification aspects without giving scope for trial-by-webapp.

I love the idea, but I think it impractical. It's too difficult stop it degenerating into "this guy looks a bit shifty", "this guy is standing weird", "this woman isn't laughing", etc.

Just look here[1] and you'll see it's already happening.

[1]: http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_m...

Or worse, this guy is $ethnic_minority or $people_I_dont_like

Anyone thinking about enabling further crowdsourced investigative work would do well to read that thread [1] with an eye to how quickly it got out of hand.

In my opinion...

Efforts to help with this should focus on helping people to submit any data and information with relevant context they have in a consistent format to the relevant authorities, nothing more.

1: http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_m...

[After reading Reddit thread.] My goodness. Flying while brown is bad enough. Now do I have to worry about being tackled by vigilante do-gooders every time I step out of the house with a backpack? Or, Cthulhu forbid, put my bag down on the ground!

If you go to the police or FBI and offer help I am pretty sure they will say no. Except in very controlled situations they are unlikely to share any of their evidence. It is just not their culture.

However I wonder what you could achieve without their cooperation. Could you "crowdsolve" a crime? Say a website where people post what they know and any evidence they have.

We need to be careful not to start a witch hunt though, vigilantism can be dangerous.

Yes I think this would be a place for people to voluntarily upload their photos. I don't think the FBI can hand over any evidence.

The FBI will indeed crowdsource the evidence: https://news.ycombinator.com/item?id=5560609

They don't understand the term crowdsource. They think this means that the crowd will be their source. This is explicitly spelled out in the article.

What do you think "crowdsource" means?

In social science experiments, it's not unusual to have undergrads or grad students classify the raw data for certain characteristics (a process generally referred to as coding). If we produce (the crowd) a large amount of data, do you really think they'll reject classification information? If someone walked into their office and said "Look at this picture I took of a guy putting something in a trashcan", I suspect that person would get pretty immediate attention.

These agencies often appear to be slow because they've got political bureaucracies as well as defined processes (chain of custody in evidence is pretty important), but it's a mistake to conflate their slower speed with a lack of intelligence.

They're crowdsourcing the evidence, not crowdsourcing the processing of the evidence.

I'd be wary of people being able to submit arbitrary "evidence." The two risks that leap immediately to mind are the perpetrators offering things that direct the investigation elsewhere, and people with an agenda against $group submitting things that make it look like $group was responsible.

It is just not their culture.

Not to mention that they don't want to hand out material that might be used to target wholly innocent parties. While law enforcement personnel sometimes engage in willful miscarriages of justice, this is nonetheless the exception rather than the rule.

Boston here. Couple of us had the same idea and are meeting tonight in the CIC (101 Main in Kendall Square) to pull together an app. Designers and PR ppl especially welcome. For now, please contact me at @jaredchung on twitter and I'll rope you in.

Neartime is a very interesting piece of software that might be in the direction this is going, could help in building a timeline of publicly available (fb, Instagram, flickr, twitter, etc.) photos based on lat/long+time. I believe it was only built for flickr.

http://blog.logicalrealism.org/2007/09/23/neartime-find-flic...

This is something I was wondering about as well. Would evidence gathered in this way be admissible in court?

I worry that once the general public gets ahold of it, it would simply become a "middle-eastern-looking person finder".

Yes that would be unfortunate. I would like this to be a "crowdsource" tool, not a "mobsource" tool. I'm open to ideas on how to prevent that. Maybe moderation?

Maybe rather than focusing on finding the culprit, the crowd could simply try to identify and account for the movements of everyone present during some time window.

This, or give people objectives rather than just stabbing in the dark.

The AP[1] is reporting the bombs were in black duffel bags containing 6ltr pressure cookers. Make people look for them instead of "anything suspicious".

[1]: http://bostonglobe.com/metro/2013/04/16/some-areas-downtown-...

I'm no lawyer but I don't we why it wouldn't be admissible unless it somehow violated the perpetrators rights. Even then since we are not the police the situation might be different.

Anyone know for sure? There must be rules of evidence that cover this situation.

I think there may be issues with authentication. Usually, you would have a witness authenticate a photo ("yes, i took this with my camera at this day and time, etc."). If it's a random photo on a website, it may be more difficult to prove its accuracy.

Seems like there would be two stages: (1) solve the crime and (2) gather the evidence. Seems like as long as the evidence is not tampered with it would still be usable in court. For example uploading a picture shouldn't destroy its value as evidence.

Even if it's not admissible, it may provide helpful leads.

Google and Facebook both have facial recognition technology and the infrastructure to handle this amount of data easily, one of them should step up. It would probably be good PR.

I find the idea that the Federal authorities don't already have the technology and infrastructure necessary themselves somewhat laughable / unbelievable. Am I way off base here?

They probably have the facial recognition tech, but they probably dont have the Web stuff setup so people can just drag and drop their pics to a service that automatically sorts through them and stores them intelligently.

Heck, Google/FB probably wouldn't even have to ask people to upload photos they could just ask them if they can use the photos they've already uploaded. Heck, they probably wouldn't even need to ask. Heck, they are probably doing this for the FBI already and not telling anyone :-/

EDIT: Apparently you can email your photos to the FBI if anyone reading this has anything they think is useful: Boston@ic.fbi.gov

Yes you are. Have you been to a police station lately, like as a witness or a victim, and have had your information processed?

Here's a very telling example that happened a few weeks ago in NYC, with arguably the most advanced police force in the nation (NYPD). A woman was violently beaten and robbed in a subway station. The mugger was wearing a fraternity jacket with his nickname that, if you entered it into Facebook, you would've found his public facebook page full of photos of him wearing that jacket.

The police took a month to release this brutal video. It took commenters on Gawker, of all places, to doxx this guy on Facebook, who ended up getting charged with the crime.

http://abclocal.go.com/wabc/story?section=news/investigators...

Do not assume anything about the technological capability of law enforcement.

You are conflating level of interest with technological capabilities.

No, I'm not. Violent muggings are not a backlog type of crime for the NYPD because they do not want a serial robber running around. You're right that high-profile crimes will bring in new technology and capabilities, but if the infrastructure isn't there (i.e. the tech skills of the officers in the field and in the office), then it's not a given that the processing of mass digital information is going to be smooth.

Actually, the PR can be quite bad because the immediate goodwill that comes from whatever help they can provide to the cops will eventually, over time, be superseded by the growing suspicion that these data giants are in league with police authorities, which is generally an impression that data companies do not want to embrace.

News organizations have dealt with this conflict for a long time. In any given reportage of a crime scene, news orgs have hours of footage and rolls of film...however, news orgs will, in my experience, almost never just hand police this information, even under threat of subpoena. Why? Because the news orgs are meant to be completely independent of the state.

It's not just an ideological stance, but has practical implications: at my college newspaper, the police demanded to see the photos we took of a riot that we didn't publish, and we declined. The photos we took were possible because people don't see us as collaborators with police. However, if the press is suspected as an arm of the police, then a press photographer is going to get punched in the face at the next riot.

The general feedback so far seems to be "Good idea, but could result in mob justice."

How can we prevent this? Some ideas so far:

- Moderation

- Give people specific tasks: "Find this duffle bag" (thanks shocks)

What else?

What about just a way to forward on important images to authorities?

I like that. If you flag a photo as suspicious, that flag doesn't necessarily have to be publicly visible.

Boston@ic.fbi.gov

http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_m...

I was thinking that same thing this morning. I've seen people on reddit find where people live just from a photo out of their front window. I've seen people track down stuffed animals that were made 20 years ago. The human hivemind and it's capabilities has to be put to a valid use occasionally.

Anyhow... What specific goals would you be trying to reach? Identify every individual who was there? Try to find photos from before the incident of people with suspicious objects?

I was down at the finish line (on the opposite side of the street) a couple of hours before the bombing watching a family member cross the finish line. My worry is that the crowd was so tightly packed that it might be tough to get any footage of what was happening at ground level. If I had simply looked down to see my feet, I wouldn't have been able to see them because we were so crammed in. If an object was on the ground, or if someone carried a backpack in at ground level, it seems like it could be missed by even the thousands of people snapping photos at all angles.

That being said, I'm wondering if you couldn't use some sort of facial recognition in conjunction with timestamps to track/detect unusual movements (like someone who quickly moves into the area of the detonation, then turns around and quickly moves out.)

Some of us tech entrepreneurs in Boston built this site yesterday to aid in the collection of photos and videos from the Boston Marathon. http://evidenceupload.org

It can be used from a mobile phone to bulk upload photos/videos directly from the phone and preserving all metadata (timestamp, lat/lon coordinates).

We used Filepicker.io for the upload widget, Amazon S3 for the file storage, Heroku for hosting and Ruby on Rails.

I suspect the software already exists and here is why: Following the 2011 Vancouver Stanley Cup riot [1] there was tens of thousands of photos and videos posted to social media. Vancouver Police (VPD) where initially overwhelmed with tips and digital media, so VPD setup a website [3] to aggregate the data and identify suspects. ICBC (Vancouver BC's provincial entity that issues drivers licenses) offered facial recognition software [4] and their database (although I think this idea was scrapped due to privacy issues). An organization called LEVA helped investigators analyze 1,600 hours of video evidence [5]. The VPD riot website [3] puts the hours of video evidence at 5,000 [6]. VPD used a team of 40 LEVA forensic video analysts and technicians from across the United States, Canada, and the U.K. was assemble for an intense two-week period to review the evidence. [5]

So, I think these people will step up and help if the government has not already developed this capability. Wired states that "Data for the Boston Marathon Investigation Will Be Crowdsourced" [7]. This process is likely already underway.

p.s. since I mentioned the vancouver stanley cup riot, I should also mention the vigilante justice that followed [8]. Obviously these two instances are much different but I think it is worth mentioning. People were "naming names" on-line, people were fired [10], people were expelled [9], people were targeted with death threats [11]. IMHO this software can be very dangerous when operated by the public.

[1] http://en.wikipedia.org/wiki/2011_Vancouver_Stanley_Cup_riot

[2] http://m.theglobeandmail.com/news/british-columbia/riot-inve...

[3] https://riot2011.vpd.ca/

[4] http://www.straight.com/news/icbc-offers-facial-recognition-...

[5] http://pipelinecomm.wordpress.com/2011/09/14/leva-activates-...

[6] https://riot2011.vpd.ca/faq

[7] http://www.wired.com/dangerroom/2013/04/boston-crowdsourced/

[8] http://www.vancouversun.com/technology/Online+vigilantes+slo...

[9] http://www.cbc.ca/news/canada/british-columbia/story/2012/09...

[10] http://www.cbc.ca/news/canada/british-columbia/story/2011/06...

[11] http://www.vancouversun.com/technology/rioting+teen+nathan+k...

This is a great idea! I haven't seen anything like it and also I think you kill of the forum to discuss photos and just attach comments to photos directly (sort of like instagram). Wonder if there is anything legally against chipping in like this?

We have a technology that goes much further than the suggestions above. For this use case, would need a little customization. If you want to get involved, email me - barrie@cliplabs.com - jon@jonb, perhaps combine our efforts?

Despite negativity of this idea. I think you should totally go for it! How do we know how useful it will be until it's done? There's no point speculating.

What harm can come out of this?

You could simply have all photo's uploaded and allow users to click and select things they feel might be suspicious with comments added. If items are highlighted by many users, it would show that it might be suspicious and it's worth the police looking at! That way they could order the most important and possible remove ones that they have personally looked over and think there are no problems.

This reminds me of this project: http://www.snapshotserengeti.org/

I see a web app that lets the police point to a spot on Google Maps, and the app provides a list of files stored that would show that spot in video or film. And on the maps screen, show the angle of those pictures. Then allow the police to highlight the file, and show the picture source point, and the V of the picture window.

If that makes sense.

Thoughts?

Amazon Mechanical Turk has been used for this type of thing (specifically attempting to find Steve Fossett).

Great idea Jonathan. I think a really simple and quick-to-build tool similar to what you suggest would be really helpful in situations like this. Right now people are eager to help out but there isn't a great place to see all of the photos and the best leads/analysis.

there is a reddit thread already attempting to do this, but it's meeting some resistance.

source: http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_m...

From reading the comment I think this evidence enough that it will be very difficult to stop this from being a "mobsource" application.

Reminds me of the search for Jim Gray. Story here: http://www.lastwordonnothing.com/2011/03/17/searching-for-ji...

Photosynth still around?

Great Idea! Not sure how you deal with time though... maybe one synth every 5 minutes?

http://photosynth.net/

Still a problem as you can't guarantee the timestamps on images gathered.

You don't. You can't, really.

You just let synth collect things into geo-locations to speed up the sorting process. You still have to manually sort all images that overlay a 'point of interest'. But the real boon is that you can quickly, intuitively and effectively do that sort.

And if you see a 'person of interest' you can 'walk' down a possible path and sort the images at intermediate points to see if you can establish a route or potentially see them from other angles.

If you want to volunteer to help build this, please leave a comment or send me an e-mail. jon@jonb.org

You can't because where do u think the bad guys would go right away. Not only that but they know where and what to look for. In other words you're giving them all the info you have on them. Not the smartest thing to do ;)

it would be great if the news organizations that are capitalizing on advertisements on boston bombing articles, would donate those proceeds to victims.

That's crazy talk you socialist. </sarcasm>

That sets a pretty crazy precedent for news organizations doesn't it?

I don't know if it's fair to say news organizations are using the bombing to gain financially. They are doing their jobs -- which is, providing a service to gather and disseminate news. In order to do that service, reporters and editors need to be paid.

You could look at another way: If the service didn't exist, many people would likely not know the news, or the extent of it -- and so might not be compelled to donate or otherwise help out.

I'm not implying that they are using the bombing for financial gain, however it is clear that horrific(and great, positive) events correlate with a certain spike in revenue in the news industry. What I am proposing is charity. Suppose event X creates Y dollars in ad revenue from it's initial spike in traffic. This would be a donated percentage of that money. Let's not forget that reputation is a key factor for creating loyal users.

If one had to choose between news service A vs news service B, knowing that choosing A has the implication of assisting those in need, people are going to choose A.

Call it pre-emptive humanitarian news campaigning.

Except that doing so might open up news organizations to claims of bias in their reporting on aid organizations and on how aid money is spent.

A week's on-scene coverage cost more to mount than the usual news from the studio. Beyond that, live coverage 'bumped' many commercial avails. This was an expensive week, not a profit center.

Reporting the news is their entire job.

Reporting on news like this may be the only way for journalism to stay in business ... I'll let you draw your own conclusions.

Reply on news.ycombinator.com