Options for Enhancing Video of a Burglary? | Hacker News Reader

Options for Enhancing Video of a Burglary?

62 pointsjes79 comments

My home was burglarized last week. My neighbor's security camera captured two 30s video clips of the individual, one at my front door prior to forcing it, and one with him leaving with a cloth grocery bag containing property he was taking.

The video shows his face, but it's not as clear as I would like. I don't know how to quantify its quality except to say its "not bad" but "not great either."

I would welcome any suggestions for extracting and improving still images from the two clips. I'd prefer to outsource this to someone that has experience in this kind of work.

I don't have a lot of confidence that the local police are going to invest a lot of time and effort in this, only because they are overloaded.

Any recommendations for providers or other actions to take would be appreciated.

View on HN

I work in the video surveillance space. The short version is that you are unlikely to be able to do anything worthwhile with those videos.

If you want to contact me privately I can take a look at them and maybe offer some specific suggestions.

Thank you.

I would welcome any suggestions you might have. If you'll contact me at john.sambrook@protonmail.com I'll send you the two clips. I don't see how to find your email and I'm reluctant to ask you to disclose it publicly.

Might want to obfuscate that address a little – perhaps replace the @foo.bar with "on ProtonMail's dotcom domain" or something, if it's worth it to stay off unsophisticated scraper lists.

Is this worth the trouble? I'm seriously asking. Seems like a ripe arena for a cat&mouse game similar to adblockers. The script kiddies writing email scraping tools surely have regex searching for "[at|@] * dotcom" type of concepts.

Maybe I'm too cynical, but I pretty much assume that any email address is "compromised" (as in no longer spam free not pwnd) as soon as it is created. It may not be that bad, but I honestly believe the first time the email address is used as an ID then it is.

>Maybe I'm too cynical, but I pretty much assume that any email address is "compromised" (as in no longer spam free not pwnd) as soon as it is created.

It's not binary, as in, either "compromised" or not. There are lots of different spammers using different methods. Posting your email address on a public website will dramatically increase the amount of spam you get.

It’s often about being the second easiest option, not the hardest.

Over several years' use of Protonmail and multiple instances of my address being available unobfuscated, spam to my inbox has remained astonishingly rare.

I post my protonmail publicly and I haven't gotten spam for a year or so.

Spam doesn't come from web scraping anymore. It is all hacked databases, contact scraping from mobile apps, and harvested victim inboxes.

If you put an address on an "About Us" page or something you might get some business related spam now that WHOIS data mining is dead, but regular 'ol users you want to target haven't posted their email addresses in HTML for 10+ years now.

I don't know how good protonmail's spam filtering is, but I've had my unobfuscated gmail address in my HN profile for years and I've never seen any spam (unless you count the occasional recruiter email).

Sometimes, if you have multiple shots where the subject hasn't moved much, you can combine them to get accurate subpixel resolution (i.e., resolution higher than the camera has). However, this works best to subjects that are "fixed" like license plates.

Here's a "practical" way to do it:

https://www.dpreview.com/articles/0727694641/here-s-how-to-p...

And here's an example of a research paper about it:

https://arxiv.org/pdf/1801.04590.pdf

The local police will spend no time and effort on it. And any video you produce/enhance will not be able to be used as evidence by them.

Thank you for this.

I don't know much about this topic, but it seems reasonable to think that multiple frames ought to be useful in estimating subpixel values.

I agree with your take on the suggestion that that police won't spend time on it, and that any altered images wouldn't be admissible as evidence. They did spend considerable time taking fingerprints in my home. And even the original video is damning. I am hopeful that better quality images can be used to get the name and possible location of the suspect. I'm offering a reward of $1,000 to the first person that gives Kirkland PD information that leads to them interviewing a credible suspect.

I doubt that any of our property will be recovered. I'd like to see this individual off the streets. Perhaps someday, he will get treatment that would help him.

Even if the photos can't be used as evidence, they may be useful in identifying a suspect enough that a warrant could be executed to search his apartment or whatever. For commercial crime cases in my city, the police almost never count on the courts to convict people of theft; their primary focus is on getting stuff back to its rightful owners if possible.

This is what I was thinking as well. Superresolution techniques aren't going to surface information that was not already in the image. However, combining multiple sequential images of the same subject can have this effect.

The problem is that I'm sure the subject moves considerably from one image to the next. You will have to isolate his face in each frame and transform the face grabs so that they line up perfectly before the images can be combined by averaging.

It's worth a shot!

Yeah, it would be a pretty good weekend project.

Detect faces in each video frame, skew/stretch so that the area of each detection is the same, apply this super resolution technique in OpenCV.

> And any video you produce/enhance will not be able to be used as evidence by them.

It may not be good evidence in itself, but if the person is recognized, they could be investigated and more evidence found. It's not unlikely they'd be in possession stolen property, that their clothes match the video, that they left cell phone records in the vicinity and so on.

A company called MotionDSP makes a tool called "Forensic" that costs $250/month: https://www.youtube.com/watch?v=u_B5PUYaIIA

Wow. That seems like a pretty compelling tool. I may license for a month and see what it can do.

Thank you for taking the time to help me with this.

I was able to save a group photo with this technique once: https://www.gimp.org/tutorials/Smart_Sharpening/

It would be interesting to save a frame to a modern iPhone, edit it in the photos app AND then apply this technique to the output of the iPhone edited pic. I have been able to get a lot more detail from a photo in a dark setting on the photos editing app on iPhone.

You might want to consider reaching out to ClearView AI. Ethics aside, they probably can help you identify the burglars. With regards to the comments suggesting super-resolution, just remember that the Second Law of Thermodynamics prevents recovery of information that does not exist. Neural networks and ML can give you a good guess at what the face may look like, but you won't necessarily get more information out of it and if done incorrectly, might even get a false positive. What neural networks are useful for here is to extract the feature vectors from the face and perform facial recognition on them.

The key breakthrough here (for the typically crappy footage) will probably involve

a) using many frames to ID the perp

and perhaps

b) working at a native H.264 I/P frame level with the neural networks, rather than at a decoded framebuffer level.

Why would the encoded frame be better vs the decided? Deblocking? Other filters? I would think temporal filtering might help? Not an expert here.

This is a totally random shot, but I wonder if Pixelmator Pro's new ML Super Resolution could be of help here:

https://www.pixelmator.com/pro/machine-learning/

You could grab individual frames of the video and run them through this and see what the results look like?

(no connection to Pixelmator - just played with the software and it looked pretty darn cool)

Thank you for this. I'll take a look at Pixelmator Pro and their "ML Super Resolution." It seems credible to me that some kind of ML approach ought to be useful here.

There is a reason Google's initial DNN project was called "deep dream"- It's because it essentially uses "dreaming" to generate new content based on a source database of unrelated images.

It is making up new image details (highly convincing and realistic new image details) and hence this tech is completely inappropriate for handling criminal evidence.

It's definitely not appropriate for suggesting that a certain person is in the video. Instead, what it can do is help (or poison) the imagination of a viewer, by inserting what a person matching the few details that are in the image/video would look like with average other details added.

Maybe your person has a darker pixel on the one side of their face you see: might be a mole, a face tattoo, or a five-o'clock shadow, or maybe they've just got a bit of mud on their face from falling in the dark. DNN will 'helpfully' provide that large facial moles and tattoos are less common than facial hair in their database, and most of their models wipe mud off their face before having their picture taken, so it will give your suspect a bit of realistic salt-and-pepper growth. Or maybe it has a bunch of interesting pictures of a guy with some facial hair and face tattoos that it can fit to the video. Suddenly you find you've been burglarized by Post Malone.

It's only useful if you're trying to get an idea of what an average situation looks like because you can't imagine what you're seeing in the noise.

By your logic, criminal sketches would be deemed inappropriate for criminal investigations as well. They’re both imperfect, but useful at the same time

You might want to try out some stacking based resolution improvement methods, like the ones used in astronomy. They take data from multiple "sources" and combine them to produce a high resolution picture.

For example, check out https://www.astronomie.be/registax/

Beyond the video...Depending on what was stolen, if it is likely to be sold, then monitor craigslist, offer up and FB garage/yard sale groups in your area.

Second that. Actually got 4 tires/rims back that way (found them on ebay)

OpenCV has a bunch of image processing algorithms that I searched for, but don't see mentioned CLAHE Contrast Limited Histogram Equalization _ I have used to good effect to get better contrast Neural Network DeNoising https://papers.nips.cc/paper/4686-image-denoising-and-inpain...

https://arxiv.org/pdf/1807.08176.pdf

I'm sure github has some repos on this

You can enhance the overall image quality using: https://github.com/alexjc/neural-enhance on every frame. https://letsenhance.io/

For faces: https://www.geoffreylitt.com/2017/06/04/enhance-upscaling-im...

I would strongly recommend against neural enhancing for identifying faces. There's a good chance the face will get "enhanced" into somebody in the training set that kinda looks similar to the blurry photo.

With a video player/editor, locate the I-frames (or simply the best looking ones, playing frame by frame), extract them, then use _any_ photo editor to sharpen, adjust contrast/luminosity, enlarge, etc, and print them. It is not rocket science and will give you something to show to the police/whoever, in about an hour of work.

Before investing real money on this, be aware of the legal value of these videos: do we really see the burglar forcing the front door? Taking your property? Is your neighbour entitled to film that area? The answer strongly depends on your local laws.

If you have the time stamp you could also reference other areas where he could have walked and get more footage. But the faster you go and get these cottage the better as some are not kept after a few days

I've always assumed those scenes in TV shows where they pull up blurry photos / video and "enhance" them 10-fold was just hollywood.

Can this actually be done to any degree of usefulness?

Correct, if the information simply isn't present no amount of trickery will conjure it up.

Or the degree to which entropy gets reduced still isn't sufficient to åfford admissible amounts of certainty on its own.

Well, these days there is plenty of trickery that will conjure up plausible details to put in place of the information that isn't there. It won't be true information though.

Sometimes enriching possibilities is good enough to break out of an investigative minima but other restrictions prevent you from recapitulating that evidence in your line of thinking. That's why parallel construction is helpful.

While you are correct for a still image, there is more information in a movie. That could potentially be exploited similar to the way a long exposure can cancel noise. This normally requires a static subject, but some AI technique may be able to normalize the position of the objects so that they are amenable to be averaged.

Not as much as you see on TV, of course, particularly the "zoom in" that you see there, but there are specific enhancers that sometimes do what appear like "miracles".

If you want to test one, for "motion blur" (let's say the licence of a car driving away), get this (last freeware version):

https://github.com/Y-Vladimir/SmartDeblur/downloads

And test it on the images here (actual test images from a reknowned Commercial vendor of a suite that is also for videos):

https://articles.forensicfocus.com/2014/10/08/can-you-get-th...

Here are links to the two videos:

https://www.dropbox.com/s/jkyxlkh0aa5d24m/IMG_5402.mp4?dl=0

https://www.dropbox.com/s/oxgwpe6bn39p0re/IMG_5404.mp4?dl=0

First video shows he touched a few things. Perhaps fingerprints off those? Those clothes would mark that guy depending on locale. Perhaps other cameras saw him as well?

Also, now you know your camera isn’t doing the best, add other camera(s) where it would have helped. Eg to capture face-on angle as you approach door. Consider upping the resolution of the cameras you add.

On a more general note, rethink about how this guy found your place as a suitable target. Mitigate what you can. He knew he had plenty of time. And he seemed quite comfortable being there.

One observation is that he had a different way of walking, particularly the way his foot landed. Combine this with the fact that he was comfortable being there means that he is from around your place

Topaz Labs have some amazing photography enhancing tools that can help. They are claiming they use AI for the enhance. Topaz Studio have this feature ClearAI for me even on standard settings is doing an amazing job to clear the photos. For batch processing you have to use it as a Photoshop plugin , the frames of interest from the videos have to be extracted as separate jpeg photos.

Using AI to enhance CCTV footage for the purpose of identifying criminals seems somewhat unfair to the individuals whose photos were used to train those AI models.

This is not a practical recommendation by any means... but fast ai has a lecture on image restoration with deep learning.

https://www.fast.ai/

https://youtu.be/9spwoDYwW_I?t=2913

Oh god this is a terrible idea when talking about identifying a suspect. You’re basically making an educated guess to construct evidence that could then be used to falsely accuse someone. Please don’t do this.

Yeh that is a good point. It would likely be a composite of training data rather than the suspect.

It may or may not work, but try applying the below project to a few frames to see what you get. Make sure to preserve the original footage on the original media, though.

https://github.com/alexjc/neural-enhance

You can't get information from a photograph that isn't in that photograph in the first place. Any software technique that purports to fill in those blanks, like a neural network, is drawing a face in based on what it knows about faces in general. The face you get may look realistically like a face, but it won't look like the face of the person who robbed you, except perhaps by pure coincidence.

I think the idea is to merge the multiple bad images available from the videos to make one good image. They're not asking to improve the quality of the image using just one image.

Video frames can be composited to improve clarity the same way it is done in Astrophotography.

And your assertion that "any software technique...won't look like the face" is incorrect. Again using multiple frames, it's entirely possible to infer what clearer underlying detail would result in the less-sharp pixel values in the video, given the training data establishing the relationship. It's a specific technique for reconstruction, not just drawing in a realistic but synthetic replacement like Photoshop's content-aware fill.

https://www.ghacks.net/2019/10/23/videocleaner-is-an-open-so...

The tools provided by Topaz Labs are incredible for enhancement and have free trials.

Not going to out OP's location by linking, but the actual video was posted in Ring (so publicly available). OP, if you want experts to help, maybe consider just posting the link.

I wonder if the “stacking” software often used in amateur astrophotography might help.

Or possibly using the Super Scale feature in Blackmagic Design’s DaVinci Resolve?

You can try with ffmpeg to upscale video and improve quality. On GitHub there are some tools that do a pretty decent work upscaling up to 4K

A better camera next time sadly. I prefer indoor cameras, better light conditions help make better videos.

Wow small world! I saw you post this on Ring. Hello from the greater Seattle area :)

Even if you can see his face clearly, how will you find him? It's just a face.

If police put out the clearest image possible, you still have to rely on those that know him to be willing to identify him. Friends of criminals aren't usually the type to turn each other in.

I can try and take a shot at it using my Pic.Hance model

wow small world - I saw you post this on Ring. Hello from the greater Seattle area :)

You’re assuming that getting a clear face view would result in:

1. Identifying the person

2. The authorities finding this person

3. The arrest of this person

4. A conviction of this person

and possibly also

5. You getting your things back.

In my cynical experience, every single one of these five assumptions is extremely unlikely to actually happen. Note that each one of them depends on all the previous ones.

That's not what OP asked and probably not the answer he/she really wants to read. This being hacker news I had hoped the top comment would be something that actually might help someone who wants to create better stills from video.

I see this on a lot of forums: people not answering the question but feeling that they just have to give their piece of advice that OP is having the wrong problem.

Stop it. It isn't helpful. Especially to others who look for answeers to the same question (but perhaps for entirely different reasons).

There are already tons of answers here in the comments about solving the problem as stated. I’m just pointing out that solving that exact problem might not be what they actually wanted. Realizing this before you go down a long, arduous, and possibly expensive, path to solve the problem, is valuable.

See: https://en.wikipedia.org/wiki/XY_problem

But this isn't just a Q&A forum. It's also a discussion forum, and OP's post contributed to the discussion. It's good info, and can save the poster a lot of time, headache, and expectation.

The OP didn’t ask for legal or procedural advice. The question was a rather straight forward technical one.

However, it came with context, which was a mistake because it drew focus away from the actual question.

It's mostly true, but your cynical experience is probably also anecdotal, unless you are in law enforcement dealing with these cases daily. It's also defeatist not to "stick up" for yourself when you've been done wrong, which consequently only serves to embolden the wrong-doer.

In fact I watched these two videos on Ring, since by insane coincidence this happened in a neighborhood less than 3 miles away. Someone there had already recognized seeing the person sitting outside a nearby Starbucks.

Facial recognition, image search, and the internet has come a long way. It shouldn't be long or far-fetched that local law enforcement can start leveraging these tools to pursue more of these cases. For all the fear-mongering, and true ethical issues, of facial-recognition and surveillance, this is a situation where it can actually pay dividends.

(and no they'll probably never get their stuff back, but that is what insurance is for. Hopefully it was nothing sentimental or otherwise irreplaceable and truly just "stuff")

I agree that everyone should feel some obligation to do their civic duty to report and pursue criminals. I’m just saying, do it because it’s the right thing to do, but don’t actually expect any of it to have any practical effect for you, personally. It might happen, as in your anecdote, but I’m saying that it’s very unlikely. Coincidentally, do you know if that person in your story was, after being spotted, subsequently identified, apprehended, arrested, and convicted?

Posting to HN about wanting to digitally enhance surveillance video indicates, at least to me, someone expecting some practical result (which is unlikely), not someone simply doing what can reasonably be done for the sake of society.

Put contact info in your profile or here so I can contact you. I do research in exactly this area (super resolution for target detection/identification).

OP replied to another comment with john.sambrook@protonmail.com

I'd welcome hearing from you. Email address is john.sambrook@protonmail.com.

Where does enhancement stop and tampering with evidence (18 U.S.C. § 1519) start? With sufficiently powerful machine learning the video can be deepfaked to have Elvis robbing you.

https://www.law.cornell.edu/uscode/text/18/1519

Is it still tampering if you provide the original and the “enhanced” one?

It will probably not be considered tampering, but at the same time, it will probably not be accepted as a proof officially.

Reply on news.ycombinator.com