Eulerian Video Magnification
people.csail.mit.edu
people.csail.mit.edu
I was really excited about it, because, as a cardiologist, I immediately thought we could use this to identify irregular heart beats in the patient waiting room, for example. This was pre Apple watch etc. I was thinking people could briefly sit in the chair in front of the camera if they are willing to be recorded, as a quick screening whilst they wait for the doc.
Unfortunately, I found it incredibly difficult to reproduce their results, even using similar data. I remember videoing my foot as I have a relatively prominent posterior tibial pulse, which in good light I can actually make out visually (but not in the video I took, as intended). The Eulerian magnification didn't seem to do anything.
Has anyone else had better luck? I did wonder if maybe I had to use specific video capture equipment, certain frame rates or filters, but if so that wasn't clear from the documentation?
It's been 8 years since this paper came out, and yet I still haven't seen it being used in the 'real world', so I am slightly suspicious that I am not alone in my experiences.
Even in little videos on Instagram it could be a cool and eye-catching effect.
Their techniques are at least patent-pending if not fully patented by now, which is going to limit adoption.
Also, tuning parameters is hell.
I don't think he is willing to share the code, but my guess is that a python implementation should work just fine, with very nice results :) (I saw the results applied in his research on videos with impressive results). It took him something like several months to have a code without major bugs.
[1] Hahn, S., Absil, J., Debeir, O., & Metens, T. (2019). Assessment of cardiac‐driven liver movements with filtered harmonic phase image representation, optical flow quantification, and motion amplification. Magnetic resonance in medicine, 81(4), 2788-2798.
Sharing code to the public in research depends on several factors, it can get open sourced when the thesis is done, it may depends on who is paying the research (eg: some industrial PhD are not public), long term plan exist to publish the code but there are only 24h a day and a whole lot of work to do, the author/uni may want to make a business after the thesis, and a lot of other parametres. It gets even worse when the code is used in medical applications as a validation process with all the relevant authorities has to be done. This can take a lot of time and freeze the code in the waiting for a certification. Also sharing code between researchers is very usual, it doesn't mean the code has to go public. In this case I think it is part because the original matlab code is patented, research is ongoing and as it is research code it is not necessarily public ready.
It doesn't mean he will never release it.
If you desperately need it, the matlab code is available, and has the same functionality, you are free to adapt it or use it directly (for research purposes).
I must say I don't like that the original code is patented, but I see hundreds of reasons why the authors patented it even if I don't particularly agree :)
More notes and code here: http://nhergert.ozeo.org/doku.php?id=projects:remoteppg
The guy who used it seems to imply his version had a bit of 'special sauce'
It did indeed require a lot of additional tweaks and benchmarking. Honestly don't think any part of our original algorithm or the tricks in the paper made it into the final version. Just doing a Fourier type thing works rarely (~20%), but the principle does work; you can get it working ~90% of the time. That said, I'd be surprised irregular heartbeat were doable in a short recording period. But I'd love to be surprised!
[0] Hands Free Heart Rate Monitor (free, private) https://apps.apple.com/app/id1457939511
One important thing we learned is that the filter design is a very important part of the whole thing. We used scipy.signal[1] for filter design. One thing that was very noticeable was that even small IIR filters performed significantly better than FIR.
I thought we put our code online a while ago but looks like we didn't. If you're still interested I might have a look if I can put my stuff online later today. Not that it's all that fancy anyways. The C++ part is pure processing and the python interface just puts a gui around it for filter design and calling the application.
[1] https://docs.scipy.org/doc/scipy-0.15.1/reference/tutorial/s...
The pipeline started with face detection, then found the mean colour value of a rectangle of forehead. It then did some DSP and returned the pulse frequency.
It worked, but only when the person held extremely still very close to the camera, with auto exposure turned off.
Disappointed with how badly mine worked, I took a look at some open source webcam PPG projects to find out how they were working. It turns out they were pretty similar, except most of them were aggressively filtering the pulse signal to be in the correct range, without really checking there was already a signal there. Of course by taking white noise signal and filtering it to a certain frequency range, you'll get a signal in that frequency range.
I now believe that to make video PPG work, you need some of (1) a very good camera, i.e. not just a low end webcam (2) quite a long recording, i.e. 10s-1m (3) a controlled environment
Eulerian sampling is like dotting the countryside with weather stations. Your measurement data shows the weather at each of those fixed points over time and from that you can understand what was happening.
Lagrangian sampling is like sending up weather balloons. Your measurement data shows the weather around the balloons as they float along with the wind. That will give you all the same information, but from a different perspective.
The Eulerian perspective for weather makes it easier to understand what's happening at a specific location, but harder to understand what's happening to a specific particle of air. The Lagrangian perspective is the opposite.
There is an explanation of this in the paper, but IMO, it's a little more difficult to follow than the classic weather analogy. Just wanted to demystify the name a bit.
Is this it?
I worked on it a bit at some point, it does work pretty well assuming you don't move that much and the light isn't too bad. As you can imagine, it's more difficult to estimate on non-white faces, particularly very dark skin tones. All the algorithms (at least until 2016) differ in the way of combining the RGB signals into a pulse signal by making different assumptions.
If you are interested more, Philips in the Netherlands are particularly active in this research domain, one of the main application being non-contact pulse estimation in hospitals for example.
Motion magnification" means you pick up and amplify small motions so they're easier to see. It's like a microscope for motion.
Example videos reproducing these results:
Temposcope
The whole field of remote photoplethysmography (rPPG) is quite interesting. I have implemented a couple algorithms, but they are quite "brittle" if you will, and only work well with certain videos. Many recent papers have introduced deep learning techniques. Hopefully these techniques are more robust to movement, skin color, lighting, and other natural conditions (unlike the usual computer vision/signal processing algorithms previously used).
https://github.com/antimatter15/evm
Demo: https://raw.githack.com/antimatter15/evm/master/color2.html
and Steve Mould's layman's video https://youtu.be/rEoc0YoALt0
I've tried it on a couple videos with fairly interesting results
Sorry if that seem a very simplistic approach, but curious what other techniques have been tried outside of Eulerian/Lagrangian magnification?