Eulerian Video Magnification for Revealing Subtle Changes in the World | Hacker News Reader

Eulerian Video Magnification for Revealing Subtle Changes in the World

people.csail.mit.edu

566 pointsclockwork_189129 comments

I like how this kind of turns the "main idea" of computer vision on its head. Namely, instead of trying to emulate and catch up to human abilities, it augments them the way only a computer can; it plays to computers' strengths instead of their weaknesses.

Also, who would have thought you can determine someone's pulse just by looking at the color change of the face due to blood rushing in and out? If asked beforehand, I wouldn't have even said it was theoretically (physically) possible.

Yes, it's hard to believe, but it's really possible to sense the small color changes due to blood flow in the face using cameras. My earlier work described how you just need a simple webcam :)(http://web.mit.edu/newsoffice/2010/pulse-camera-1004.html). As a result, I'm working on Cardiio (http://cardiio.com) to bring this technology to both the web and mobile very soon. If you're interested in computer vision based biosensing, please sign up!

That’s awesome!

Hmm, when I tried to sign up I got a message that I was already on the waiting list.

EDIT: It looks like the page is making a request to cardiio.com, specifically, and if you’re at www.cardiio.com (the one you linked to) the response gets blocked by Same Origin Policy, and the page treats it the same as a you’re-already-on-the-list response. You might want to avoid hardcoding the hostname in the JS, and/or force everyone over to the www or non-www domain.

thanks for catching that!

Do you think it would be possible to see shifts in photosynthetic activity in plants due to low/high nutrients/humidity? Especially when IR-filter removed?

Yes. My father does research in this area. Specifically, inferring the health of vegetation (among other attributes of land cover) from statistical analysis of satellite imagery.

From what I recall, you can measure the level of photosynthetic activity in an area given the reflectivity of the land (and, I would think, some basic model of what type of vegetation is present in the area)

One example from his research: the fall of the Soviet Union had a statistically significant impact on weather patterns in Kazakhstan (the breadbasket of the USSR) because the land usage changed so much with agricultural policies before and after the fall.

Quick link to the research overview page: http://globalmonitoring.sdstate.edu/research.php

To measure the pulse of a person, you need to consider an interval of a few seconds, so it's safe to assume that the only thing that the position and strength of the light are almost constant. The time that takes the plants to reduce the amount of nutrients or water is much bigger, so I think that it would be much more difficult to compensate the changes in the ambient light.

Congratulations for, what seems to me, opening up the door [1] to a new world of CV applications! And of course for having the courage to commercialize it while your academic career is taking off.

[1] famous illustration from the enlightenment: http://www.lehrer.uni-karlsruhe.de/~za874/homepage/aufklaeru...

Just curious have you tried on people with darker skin colour/tan. I am trying to imagine, what adjustments you'll need to make to detect on them?

There was a sample in the video, right after the white guy. The effect was more subtle, but it was there.

that's right. what you're looking for is the transient changes (AC) in color due to increase blood perfusion whereas a person's baseline skin color can be considered the DC component.

That sounds awesome.

But from a scientific point of view, you don't really need spatial resolution for the heartbeat example, though, do you? If you just spent the same amount of money on getting high temporal and spectral resolution, surely one could do very cool things with 'just one pixel', so to speak. Especially if you had say 16 different wavelengths, instead of just 3.

If the face of a "patient" was tied to his bed, and light wasn't changing, and there was no movement around (i.e. in ideal conditions), then yeah.

But in the real world scenarios you want to take a look at more than one pixel, and use the rest of the picture for noise reduction & calibration.

That is for heartbeat. But yes, with better spectral resolution you might get some other cool data.

For increased temporal resolution you can just the rolling shutter with off-the-shelf hardware I guess.

What do you think of the many apps already on the market, possibly derived from your earlier work?

https://play.google.com/store/apps/details?id=com.macropinch...

I also found it very surprising that cameras could pick these things up, there's actually an iPhone app from Philips that claims to be able to be able to read heart-rate, I assumed it was snake-oil at first :

http://itunes.apple.com/us/app/vital-signs-camera-philips/id...

I was skeptical too, but when I tried this app a few months ago, I compared it to my Garmin heart rate monitor at rest and after exercise - they were usually within 1bpm of each other. I was very impressed.

There is so much faint and indirect information in the world it's dumbfounding. We're crossing a threshold where computation can make this visible. It's really exciting.

Pulse oximeters work by looking at the fingernails, and they're the usual way to measure a pulse. There's a lot of blood flow in the face - try bumping into something sharp - so this isn't too surprising.

A Pulse oximeter doesn't measure the spontaneous signal.

There's a red/infrared led on one side of the finger, a detector on the other side, and it measures the amount of light absorbed by the blood.

There's a whole business to be made off that five second clip where they amplify the motion of the sleeping baby's chest.

Anybody who's ever been a parent can remember going up to check on the baby at night, standing over the crib for a full minute thinking "move! Move! MOVE!" at the lifeless body of their baby, running all the horror stories they've heard about cot death over and over in their head until finally the kid's chest moves enough to breathe a sigh of relief and go back to bed.

The first company that adds this feature to a video baby monitor will make a mint.

That business already exists, though the existing sensors typically come as a pressure-sensitive mat (placed under the mattress) instead of a camera.

A business preying off the fears of new parents, marvelous.

There's a bit difference between building a business that preys on the fears of a new parent and a business that acknowledges that new parents already have fears and tries to reduce them. This I would say falls firmly into the later category.

It's the difference between a tool which helps you make sure that your baby is ok and one which tries to persuade you that if you don't use it you're creating unconscionable risks to your baby which this product could save you from. The former is a tool which is good, the later is one which preys on parents.

Totally disagree. The two devices are one and the same.

Checking on your baby in the middle of the night because you woke with a sudden fear - normal parent anxiety, makes a funny story later.

Providing a device to allow constant electronic monitoring - validating an irrational fear, reinforces anxiety, feeds into an obsessive loop, invents issues for new parents just when they need to be calm and rational.

We clearly disagree on a fairly basic level.

I'm very much of the opinion that anything which causes peace of mind is good thing and teaches the parents that their fears aren't grounded in reality. I'd say that most the fears that new parents have are rational in the overarching sense (in that it's entirely rational for a parent to worry about about their baby's health whenever the baby seems to deviating from the expected), but new parents lack the knowledge to determine if any one particular event is something that constitues an actual emergency, or is perfectly normal. Babies commonly have minor health problems (such as colic or jaundice) when they're very young which don't require medical attention, but sometimes they do require attention and it can be hard to tell which is which since babies only have very limited ways to express that they're suffering. A tool which provides more information to the parents (like, say, being able to see that the baby's heart rate or respiration has dramatically changed and so they might not be crying as a result of colic, but because there's something else wrong) and let's them make a more informed decision is clearly of benefit and is by no means 'validating an irrational fear'.

Peace of mind is most often effected by turning the mind from obsessive loops. Selling expensive gadgets to nervous parents is way too tempting a profit center to be responsibly marketed. How about by prescription only? That would be a reasonable compromise. Ideally someone objective could be put into picture, to prevent abuse.

I wasn't aware we had good indicators for SID. http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0002533/

Lists some. Right after saying we don't know what causes it. The things listed are pretty generic, like smoking around a mother expecting, etc. But nothing a hundred percent. It's also the biggest cause of death in children under one year of age.

Not sure where you are getting that it's something not to worry about. I do wonder how many children you've had though.

Edit: Oh the point's made above were to beg the how would one get prescribed something that so far seems pretty random.

Oh, yeah, good point, attack me instead of the argument when you run out of things to say.

I have 3 kids. I trusted in fate to take care of things I had no control over, and was lucky they lived to be healthy confident adults.

I know of super-careful parents that raised precious children frightened of the world. That loses something too - like the purpose of living.

I know engineers want to solve everything with an engineering solution. That way leads madness. Next we'll be sticking monitors in the womb, because, you know, sometimes the foetus doesn't develop right and maybe we can do something about that too.

I'm sorry no. The expression of "preying on parents fears/anxiety as marketing a device" is not literal. It is about drumming up those fears at the time of the sale, not at the time fo the use. In short, if I make no claims about making the motivator of your fears go away, but I do make claims about helping you manage the fears themselves, that isn't preying on your fears and anxiety. I believe the technical term for that is "helping".

The obsessive loop is, "maybe my baby isn't breathing <-> check on baby <-> it's breathing <-> go back <-> maybe my baby isn't breathing". If you can go right from, "maybe my baby isn't breathing <-> of course not, this idiot box would tell me if he/she weren't" (and sure it's fair that that is far from certain), that's huge progress and it can deflate obsessive compulsive behaviour, anxiety and fear. It's the equivalent of giving a kid a stuff toy so they aren't scared in bed at night.

Haven't been a new parent, have you?

$5 for a suitable app is awfully cheap assurance. $50 for a dedicated device, ditto. We're not talking "taking advantage" here, we're talking something every parent in history wanted.

Does it provide actionable data? If not, we're just discussing the dollar amount by which panicky, obsessive parents should be screwed.

IMO a more likely outcome than preventing SIDS is a device malfunction leading to parental heart trouble and general distress.

Perhaps an analogue would be a device that illuminates a green light when there is not a (dreadful disease|nuclear strike|zombie horde) incoming...

Does it provide actionable data?

What part of "the child is, in fact, not breathing" isn't obviously actionable?

Seems you've never stared at a child in dim light, waiting for some sign of life, knowing everything is probably ok and you don't want to disturb the snowflake to find out, but know that there are nontrivial odds of spontaneous expiration and the consequence of further inaction may be dire. Cheap insurance, peace of mind, good profits in mitigating a fair concern.

I'm asking how this "not breathing indicator" could be used to improve outcomes. What does one do if it signals a problem--are parents going to be trained to perform CPR on their infants (and would that even help if done properly)?

Pandering to parents' fears about something they can do nothing to solve does not seem to me like a productive thing.

"I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." - Babbage

Call 911. Do CPR (yes, some of us are trained). Get in the car and race the kid into the ER. Clear the airway. Solve the problem.

What, you're going to SIT THERE WHILE YOUR CHILD DIES???

I don't think it's very unhealthy for a parent to be obsessed with the health of their child.

To be honest, a soothing green light that indicates a bunch of dangers are not to worry about sounds pretty good to me too.

I think a device malfunction would at worst lead to a baby being forced awake by a worried parent to check everything is ok...

The worst case would be a soothing green light when things aren't ok.

Wow, reading someone's heart rate from a video.

Imagine that plugged into Google Glass. Overlay someone's heart rate next to their face while you're talking to them!

Excuse me while I book a trip to Vegas.

I was thinking about the implications of using this technique to analyze e.g. political speeches and try to catch people lying on the act. Your application (winning on card games) seems very interesting too.

It seems like this technology can be used on historical videos. It'd be interesting to see the heart rates from past speeches where we know they lied, to see if there is a good correlation.

A lot of the sample videos have very high frame rate, so it might not work on historical video.

That is an awesome idea.

I imagine the drug atenolol would become popular among politicians.

That, or the radio address will once again become immensely popular.

Until realize that even though the filters in this post have been applied to videos. It will be just as easy to apply the filters to audio, surely again resulting in a revealed heartbeat :)

Coincidentally so was I! With a person's microgestures amplified, you'd at least notice inherent changes in a person's physical behavior. Now, we need to remember that lying isn't as simple as "he's looking down and to the right" because that can just be a personality trait of the person.

With public figures, however, if you collect enough video of various activities, you could determine a baseline physical demeanor. Then, if the figure in question radically steps out of that baseline, you can send off red flags.

Ideally, this could be used to analyze facial dynamics to help determine focus, expression (for example, the SOEmote video that was recently posted), mood (though we now get back into personality traits), or, as the video showed, subtle changes in the physical body.

I was under the impression that micro-expressions (I assume you're referring to them) were very short in duration rather than in magnitude. So this would be the wrong kind of amplification.

micro-expressions .. were very short in duration rather than in magnitude. So this would be the wrong kind of amplification

This is exactly the kind of amplification you need. Something like the movement caused by mirror of an SLR camera (at the end of their demo) is very very short in duration, but they amplify the magnitude AND slow the video to allow it to be visible.

No, my point is that micro-expressions don't need to be amplified, because they are perfectly large in magnitude, just very short. So all you need is a freeze-frame.

Some amplification is still needed in my opinion, but you are correct. Amplifying the video of a high speed camera should aid in the some of the expression detection. Instead of something like eye movement, think something more subtle, like a person's brow being furrowed.

> analyze e.g. political speeches and try to catch people lying on the act.

Politicians in front of cameras are too heavily covered in face powder for their real color to shine through.

And if that doesn't work, special makeup will be invented that does do the job.

Of course, use of those techniques will themselves come to indicate guilt...

Kind of like paranoid internet users setting up TOR implies their guilt as well.

I wouldn't make assumptions. There's a valid point in wanting to defend your thoughts and intentions.

What about using this on a crowd of people? You could get an idea of what impact your particular statements are having on them.

It could be interesting to look over old films of riots to see how they may have been instigated...

The technology to do that face to face might already exist: http://news.ycombinator.com/item?id=2727960.

The technology is probably not mature enough for video analysis, though.

> analyze e.g. political speeches and try to catch people lying on the act

Does it work on cold-blooded animals too?

I see a bigger problem: to calibrate the data you would first need to find a politician who is not lying during a speech. Good luck with that ;-)

Something tells me Casinos aren't going to let you wear Google glass... :)

Just like you aren't allowed to keep your phone on the table.

Imagine when everyone is an amateur polygraphist ... some conversations are going to become unnecessarily fraught.

The use I thought of was figuring out when your date was trying to perk up the courage to kiss you.

And personally, I don't think any of the "modern-day lies" that people tell would cause any kind of increase in heart rate.

and then it turns out your date was extremely uncomfortable and stressed out, trying to think of ways to get away.

it is not as simple as you wish. and that is good!

How about amplifying micro-motor motions in the eyes or something like that?

If people find this compelling, they may also like this earlier work from the same group:

http://people.csail.mit.edu/celiu/motionmag/motionmag.html

This other algorithm amplifies only motion, and the algorithm is less elegantly simple, however its results are very interesting and fun to watch as well. Make sure to watch the video at the bottom of the page

http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2...

I watched that second video http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2... - that must be incredibly useful in engineering analyses.

Amazing for many reasons, but what I really find impressive is that messing with video encoding requires like a PHD in everything because of how complex things like H.264 are. At least that's my perspective after trying to grok the H.264 spec.

There are all kinds of optimizations and algorithms for network latency and video sizes, etc. I don't consider myself a layman but I certainly do not have the knowledge and skill to do anything at all with video.

Do you know what a Inverse sub-macroblock partition scanning process is? I don't, not even after reading the part about it in the spec.

This really doesn't have much to do with what makes computer vision difficult (or rather, interesting). I doubt very many computer vision researchers are conversent with the H.264 spec.

Typically toolkits like OpenCV take care of codecs for you (which make them quite annoyingly difficult to compile on different platforms).

(I speak as someone who had done commercial computer vision algorithm development.)

Unfortunately when they do (My dissertation is on motion analysis using H.264 compression artifacts) things get pretty complicated very quickly

What precisely is your dissertation about? Inferring motion from compression artifacts? Exploiting the fact that many codecs have already done motion estimation so that you don't have to do image flow again?

Yes, exactly that. Then filtering them because they don't actually show optical flow, they just show a lot of noise

No need to mess with the encoding format - just use an existing utility (or library) to grab the raw frame data. e.g. ffmpeg

a macroblock is a chunk of an image which is identified to have some constancy between frames. a sub-macroblock would be some subdivision of this for finer motion details. sub-Macroblock partitian scanning would be applying an algorithm to identify the sub-macroblocks in a set of images. Add the word "Inverse" though and now I have no idea what the hell that means.

This technology is badly needed today for skin cancer screenings. We could save tens of thousands of lives a year with better detection of cancers and pre-cancerous growths. It is crazy that most screenings in 2012 are still 3 minutes a year with a harried doctor.

From my experience with CV, one of the biggest unsolved problems and rarely publicized aspect of the field is how much variable tuning is often required to get good results. I'm curious how sensitive these demos are to different environmental conditions and the robustness of the algorithm.

i wonder if this could be used for lie detection

i wonder if this could be used for lie detection

I was tempted to downvote, but this is a legitimate question, and I guess you haven't had opportunity to learn about how bogus "lie detection" in general is. I happened to attend university where some of the earliest rigorous researchers on lie detection were active,

http://www.fas.org/blog/secrecy/2006/09/david_lykken_and_the...

http://images.salon.com/april97/news/news970410.html

http://antipolygraph.org/articles/article-018.shtml

http://www.psychologicalscience.org/observer/getArticle.cfm?...

so people in my town have known for a long time that purported lie detection tests don't work, and indeed our state has long banned the use of polygraphs or other purported lie detection devices for any courtroom purpose. Law enforcement investigators in our state use actual interviews of witnesses and suspects, and by state law all police interrogations must be videotaped. That helps immensely in reducing mistaken conclusions from improperly conducted interrogations.

I'm actually thinking if it might be a good tool to detect distress in a subject. Not lying exactly, but might provide insight into a subject's emotional state non-invasively during questioning or interrogation. People might become distressed for many reasons that may not just be simple lying.

Perhaps even a Voight-Kampff-like test a la Blade Runner/Do Androids Dream of Electric Sheep?

It never made sense to me that an manufactured android would be easiest to detect using a psychology test instead of say, X-raying them.

Androids in that case were fully human, just manufactured. The principle differences were improved physiology and durable mental processes, which could have occurred naturally.

In "Do Androids Dream of Electric Sheep", the androids are practically indistinguishable from humans in their physiology.

Its quite true that polygraphs and similar means of telling whether a person is distressed are almost useless. However, there have recently been some advances in using fMRI to do direct lie detection by looking at what parts of a person's brain are being used at a given time. This doesn't seem to be reliable enough for courtroom use yet, but its getting there.

I said it above, but to carry on with what tokenadult said; Lie detection is a little more complicated than detecting facial expressions. If you have a defensive, introverted person looking downward, they might be lying or that could be how he responds to whatever interrogation they are under. Conversely, a mentally distressed individual (or someone who has done a heavy amount of NLP) could lie straight to your face because they personally believe they are telling the truth.

This type of technology will no doubt be used to attempt lie detection, but it won't work. The only thing we can do is use the context clues of the face to infer what we think is going on.

Is there an easier-to-follow explanation of the math that makes this work? The video went too fast for me to quite understand it.

According to the paper, it is really quite a simple idea.

Imagine a filter that one might use in an audio context that boosts the amplitude of frequencies falling in a certain range. But now imagine applying that filter to a timeseries that actually comes from the intensity of a single pixel over time.

This has the effect of emphasizing motions that occur over timescales that correspond to the selected band of frequencies.

The reason this is true actually comes back to the Fourier transform -- you can imagine the motion of part of a complex image to be composed out of the linear combination of many simpler 'sinusoidal motions', each of which is being amplified by the per-pixel filter.

What about motion amplification? For example, the baby breathing in the bed... Aren't multiple pixels involved then?

Definitely.

There are two streams of information being analyzed. First is color, observed as one position changing properties from one frame of video to the next. Second is position, which is a change in location of things within the field of view from one frame of video to the next.

Both color and position changes can happen at the same time, but it's easier to see how this information is used if you start off by treating them separately.

For color changes, imagine video of a stationary person's face where the center pixel of every picture is the center of the face. The color at that center pixel changes from frame to frame. At frame 1 the red value (of an RGB triplet) is maybe 200 (out of 255), and by frame 5 the red value is 203. This is a temporal red shift of just 3 levels over 5 frames.

For position changes, imagine video of a beach ball in a still pool of water, where the center of the ball changes from frame to frame. At frame 1, the position is (in x,y) (512,512), and at frame 5, the position is (512, 515). This is a spatial shift of just 3 pixels over 5 frames.

In both of these cases, the observed change is probably too smal for humans to notice in casual viewing, so an exaggeration of some kind is desired. To make the change noticeable, we'll take the change of signal (3 values over 5 frames) and "amplify" it. These modified ("amplified") values will be shown on top of the original source video, resulting in an exaggerated change. If a color shifted by 3 red levels, we can show that by varying the color by 30 red levels. If a position changed by 3 pixels, we can show it changing by 12 pixels. We can choose the level of amplification best suited for observing the changes we've isolated.

Now the really cool aspect of these data streams is that their resulting "amplifications" don't need to be applied to the same data stream that they were observed in. It's possible to take a position change, and represent it as a color change. It's possible to take a color change, and stretch it out over a longer time (over more video frames). Any combination of properties you like can be used to represent your observed set of changes. Some of the most interesting representations will be a mixture of different data streams, represented as another mixture of exaggerations to the original video.

Hm... I get the color one, and the position one in theory... However, if you have a real-life video, how do you determine where the "center of the ball" is? How do you even define what an object is, what are it's edges, and in which direction did it move?

These guys are going to make millions on baby monitors alone.

Maybe, does it work well in a darkened room?

Baby monitors should probably be IR based; then this would appear to be a great addition allowing a pulse to be determined from the video feed.

Current baby monitors use a pad underneath to detect movement including breathing.

Personally, we used the "you can hear them screaming half-a-mile away" method of baby monitoring.

This is pretty darn amazing! I'm so going to try to implement this set of algorithms later this week! :)

If someone wants to do a kickstarter to try and use this technology to build a glucose meter, I will be the first contributor

:( unfortunately might not be entirely applicable to this problem... unless some measure of blood density and hence probably glucose mix... but at least at this layer of filtering and reasoning, it appears to be about minute amounts of motion... is there something about motion that could denote glucose? trembling maybe? maybe a score could be derived using something that borrows from this technique, but you'd have to put more into it than it seems this technique specifically shows.

I was thinking more skin color or possibly changes in the eye (if I remember right eyes get swollen when blood sugar is high) Honestly, I don't know if there are enough outwardly visible changes for this to even be possible. But there's a lot of money out there waiting on the person who can figure out a way to use a cell phone do something like this.

Good idea, but apparently much money has been spent on it already. Excellent article on the physics of all things non-invasive (but especially glucose): http://www.mendosa.com/noninvasive_glucose.pdf

Wow, I didn't realize this was something of a white whale. Still sounds like a fun nerd project though so I might spend some time on it

Hey thanks, looks like an interesting read

This type of technology would be a nice novelty item to implement into a futuristic zombie story. For example, there could be a large battle between humans and zombies, and the humans are equipped with pulse-magnifying glasses, allowing them to identify friend vs. foe.

Instant Heart Rate for Android uses something similar to check the pulse using the phone's camera.

https://play.google.com/store/apps/details?id=si.modula.andr...

Even more similar: http://www.whatsmyheartrate.com/

Finally, you can "enchance" videos, just like they do in the movies: http://www.youtube.com/watch?v=Vxq9yj2pVWk

That was fairly ridiculous, but where on earth did that clip of rotating the video come from?

The team's initial work on topic:

http://people.csail.mit.edu/celiu/motionmag/motionmag.html

The Is He/She flirting with me App is just around the corner. I wonder what new privacy issues will crop up with this kind of technology.

If anybody could help clarify a concept (a specific filtering method), from video:

"we filter the sequence temporally using a "???" ...and boost the temporal signal to generate a video..." (at 2min. 45sec in video)

What type of filter is being described (missing from quotes; "???") ? Bonus upvotes for providing a URL to definition (detailed description of the missing term).

He talks about a "Temporal Bandpass Filter". Not sure about the "temporal" part, but in DSP a bandpass filter is a filter (think Audio-EQ) that passes frequencies within a specific range and rejects the rest. See https://en.wikipedia.org/wiki/Bandpass_filter

Thanks much noisebeuter!

It would seem to me from perusing the spec that they are simply taking a video, doing a 1D DCT on the time axis and amplifying a "frame" in the result corresponding to the desired period. Hopefully someone can correct me if there's more to it.

I'd like to get at one of their sample videos before filtering because it would be pretty easy to toss together a workalike based on the above.

I have a tangential question: What's the state of technology that would allow one to see through the eyes of another? This would be useful for piano tutorials, etc. I think it could be done with a separate eye-tracking device and a camera located between the eyes.

From 8 years ago:

http://www.cs.columbia.edu/CAVE/projects/world_eye/

http://www.cs.columbia.edu/CAVE/projects/eyes_relight/

Uh, what? If you play piano, you're looking at the score, not the keys.

Not necessarily; I usually don't use a score when I play. It would also be insightful to see the 1st person perspective of a jazz musician playing from a lead sheet, to see how his eyes jump between page and piano.

Wouldn't a keyboard with keys that light up be better and easier?

You don't get to play with proprioception that way :-)

The simplicity and elegance of this algorithm is really breathtaking. I'm mostly thinking out loud here, but I think you could use a related process to do inter-frame interpolation for smooth time-scaling of video.

You can! http://ompldr.org/vYWh5eA/test2.mp4

This was made by resampling in time with a DCT. Problems with this: 1. practically, it needs the entire movie in memory. 2. gibbs phenomenon in time is really weird. These could both be addressed by using short overlapping windows, and then the process becomes just like typical sinc/lanczos interpolation in time.

Well, yes, you can interpolate in the frequency domain like that, which works pretty well for resampling 2D data as well. However, it does not transition shapes from location to location; instead, it looks like one frame is fading into another. What's fascinating about this multiscale algorithm is that it appears to actually exaggerate the motions rather than merely increasing the contrast of the changes.

Excellent. Integrate this into the Tricorder projct: http://tricorderproject.org/

Does anyone have a link to the paper?

It's on the page: http://people.csail.mit.edu/mrub/papers/vidmag.pdf

I just downloaded an app[1] that measures heartbeat rate by taking video of a fingertip.

I was stunned by the brilliance of the application, how ubiquitous video sensors can replace other senses.

There's something else to discuss here: We have a recently published paper from MIT, including mathematical details of the process; and at the same time a stack of closed-source commercial applications already in the market with slick interfaces. Are they using the same technique? Is one app-maker trawling preprints for app ideas? Are many app-makers copying one pioneering app-maker?

[1] https://play.google.com/store/apps/details?id=com.macropinch... but there are many similar apps.

This is a similar idea to the temporal arterty thermometer, which uses an infrared camera: https://en.wikipedia.org/wiki/Medical_thermometer#Temporal_a...

The mathematical model for the sensor is probably different, though.

I'm not seeing what the big deal is here. Extraction of differential information from video frames has been standard in video editing and compositing software for years. Grab a demo copy of After effects, folks.

This is kind of the stock-negative comment that I don't like about HN. It would be redeemed if it had a link, but instead its just a look-how-cool-I-am-that-I'm-not-impressed-by-this actually-pretty-awesome-technology-demo, dammit.

The commenter you are replying to (and downvoting) has done (IIRC) professional digital video work, and written many detailed and helpful comments on HN about digital media. The comment you're replying to isn't snarky. Assume good faith. Maybe he's just mistaken? Or maybe he's right, and the thread just doesn't like that?

Here are some best-of's to peruse before concluding that 'anigbrowl is part of the problem with HN:

* http://news.ycombinator.com/item?id=1875412 (MIDI)

* http://news.ycombinator.com/item?id=995096 (3D film)

* http://news.ycombinator.com/item?id=4011770 (Google Moog doodle)

* http://news.ycombinator.com/item?id=1263289 (DV on DSLRs)

A couple suggestions:

* If a downvoted comment with no replies upsets you so much that you feel a need to pile on with a pointed negative comment, go to the bottom of the page, type the commenter's name in, click 'comments', then sort by 'score', and read the top 10 or so first. You may find the urge to pile on dissipates quickly.

* While it certainly isn't going to cause drama here ('anigbrowl being one of the more levelheaded contributors to HN, way more so than me), I think you'll find that comments like yours tend to read as personalizing disputes, and lead to embarrassing spats you'd be better off not attaching yourself to.

Fair enough. Please accept my apology, anigbrowl. You are correct, tptacek, to suggest I should assume good faith.

Looking back, I think what riled me was the word 'folks'. As in, "nothing to see here, folks", although he didn't actually say that.

blush

I referenced a particular piece of software, which I thought was well-known enough not to need a link. I am not trying to look cool, but rather point to existing cool technology, and stand by my comment.

quote from http://people.csail.mit.edu/celiu/motionmag/motionmag.html

"Naively, this sounds like one might (a) compute the translation from one pixel to the next in each frame, and (b) re-render the video with small motions amplified. Unfortunately, such an approach would lead to artificial transitions between amplified and unamplified pixels within a single structure. Most of the steps of motion magnification relate to reliably estimating motions, and to clustering pixels whose motions should be magnified as a group."

The math behind the grouping (recognizing the relevant area as an object, magnifying the motion of the area properly, and filling in the empty space intelligently) is the cool "computer vision" part.

I'll bite. I've used After Effects at the grab-a-demo-copy level for motion tracking and stabilization and rotoscoping. I don't remember anything that would have allowed me to detect and amplify color changes in someone's face that way, or amplify movement of a given period like the shots of someone's pulse or of a baby breathing -- especially not with a general-purpose temporal amplification filter instead of really specific code. I'd love to know how to do that stuff with After Effects, though. Can you be a little more specific?

My guess is you might end up with something along the lines of, say, take a specific clip of a guy's face, mask the face, average the color, detect the subtle changes with a script, and use the output of the script to control a color filter. Likewise if I had an existing clip of a particular sleeping baby, I could target the chest, motion track, and feed the tracking data into a filter. But that's a laborious way to fake particular outputs of what this software does in realtime on arbitrary clips of things moving or changing -- it's not nearly equivalent.

Reply on news.ycombinator.com