I am just worried that they are picking up part of information from camera mic. Maybe camera mic output is not totally independent from the video sensor, and is encoded in the final video.
The subtitle of the article is "Algorithm recovers speech from the vibrations of a potato-chip bag filmed through soundproof glass."
The camera used in most of these experiments (a Phantom high speed camera) doesn't even have a microphone - so that would be quite impossible.
Thanks. However, there is still a possibility that mechanical vibration is coming to the video sensor through some path (floor+camera tripod), and affecting the video. After all mechanical vibration is affecting the potato chips. Why is it so impossible to affect video sensor? Especially in the sensitive high speed camera? Let the downvotes begin :)
Well, the chip bag weighs a few grams at most. The camera and gear weigh something like 50kg. The higher the video frame rate, the greater the damping you need to prevent unwanted vibration. Notice the tests were performed with a speaker playing back inside the room rather than a person, in order to minimize mechanical vibrations. The sensors are very firmly mounted inside the camera, trust me.
I fail to see how that would make this any less impressive.
If my evil :) speculation is true, the camera sensor is just picking up mechanical vibrations from environment (coming through air, tripod etc.) and encoding them into video. There is no proof that the bag of chips is vibrating (they even say that the vibrations are not visible on footage). They are extracting something from video, but that something may just be spuriuos pickups by equipment, and not related image/video of bag of chips. Thus it would be just a silly way to measure spurious mechanical vibrations.
They put the camera behind soundproof glass for one of their tests.
I know, but I still wonder about level of soundproof/mechanical vibration and the level of mic and camera sensor isolation VS level of optical vibration. In addition sound can travel many different paths since it is mechanical vibration, and traveling through sound proof glass is not the only path. Anw cool project.
I'm sure it's such a tiny amount that it isn't really relevant. Easy enough to test. Remove the bag of chips and just film the floor. See how much signal you can derive from that. Not exactly perfect science but gives a quick indication of "signal" that's reaching the system via other means.
Actually, what I think would be even better would be to record the bag of chips through the window, but play the sound outside where the camera is.