Hacker News Reader

In your example saying that the ear doesn’t work as an FFT, I think you’re confusing physical processes (the ear) with perceptual processes (psychoacoustics and interaction between the brain and ear). The cochlea itself is a physical FT, based on resonances in the thickness at various points. That gets passed to the brain as basically sine-frequency data. The autocorrelation part comes in when the brain processes this and actually reinforces it by feeding back signals it thinks are peaking back into the inner ear.

(Edit) also this is a really fascinating and enlightening way of viewing it - didn’t mean to imply otherwise with this quibble :)

IIRC, that's probably not quite true. It's been about a decade since I studied hearing, but my understanding is that the idea of the cochlea being a physical FT (e.g. the place theory of hearing) doesn't explain some phenomena, which also probably aren't psychoacoustic.

https://en.m.wikipedia.org/wiki/Temporal_theory_(hearing)

I'll read more about it, but the thickness/stiffness of the cochlea resonates at frequencies with sensitivity that matches our ability to distinguish frequencies - it has been removed from the inner ear and resonance tested outside of the context of other processes. Interesting that it doesn't explain some phenomena, but are the temporal theory and place theory mutually exclusive?

(Edit) After reading a bit more, it seems to make more sense that it's a combination of both, as the resonance on the cochlea is likely not 100% accurate, and conversely the impulses from peaks would tend to be around the areas of resonance, so rather than being mutually exclusive it makes sense these two effects work in parallel.

  Modern research suggests that the perception of pitch depends on both the places and patterns of neuron firings. Place theory may be dominant for higher frequencies.[4] However, it is also suggested that place theory may be dominant for low, resolved frequency harmonics, and that temporal theory may be dominant for high, unresolved frequency harmonics.[5]

> I've been trying to find a "proper" connection between audible sound and visible shape, a connection that would not only preserve all the information, but would also properly visualize the "symmetry" in sound, so that messy sound would turn into messy images and harmonic sound would turn into visually appealing images.

It is very exciting to come across others who are also interested in this topic. I am also very interested in the shape of sound but I have spent less time on empirical observations and more on imagining an abstract logic of numbers which can be visualized and heard. Real sound visualizations are also interesting to me but I decided to focus on abstract ideals because I thought it would be appropriate for a video game.

Hope you don't mind me sending some emails.

Your emails are welcome! As for a physical sound visualization, one exists already: see "Numerical simulation of Faraday waves" by L. Tuckerman. As usual, the dry technical paper glosses over the visual aspect of Faraday waves. Basically, if you create such a standing wave in a cup of tea (e.g. by shaking the table), you'd see its shape via reflections, similar to how we can see the surface of ocean waves via reflections of sun. However, if you could suspend the water surface, put a LED ring above (one of those used for professional photos) and take an extremely high quality photo to see all the reflections and refractions created by that LED ring, yo'd see a picture of remarkable complexity. It looks a lot like a 3D hologram. I've been trying to simulate this effect on GPU.

Surprisingly, I stumbled across a result that seemed similar yet weirdly different: https://twitter.com/theshawwn/status/1176070857468329984?s=2...

I take the FFT of the phase component, which is very similar to ACF; it’s the FFT of an FFT, but preserves phase. It even takes abs(), which might be mostly equivalent to your squaring operation.

Weird. I am really not trying to claim that I discovered ACF — quite the opposite. My result was shockingly different to what you found, even though the operations are so close to identical.

I think phase is extremely important in visualization. You can see why here: https://twitter.com/theshawwn/status/1176070853819342848?s=2...

You’ve come up with one of the most gorgeous visualizations I’ve ever seen for signals in general!

One way to incorporate phase: turn the angle into an x,y coordinate using atan2, then shade red and blue based on x and y. E.g. x of 1.0 is “full red”, x of -1.0 is no red; ditto for y, but with blue.

The other trick I used was to un-interleave the lines. Basically I noticed that every other line has a strong correlation; therefore drop every even numbered line to remove the aliasing artifacts. Then suddenly you get nice and smooth phase interpolations.

Interesting. Figuring out the phase problem is one of my biggest TODO items.

How did you compute FFT of the phase? The thing is, phase is discontinuous or multivalued function if we represent phase as a real number. We could also represent phase as a complex number of unit magnitude: exp(i phi). It would be continuous, but complex-valued.

And phase is indeed important for hearing:

https://auditoryneuroscience.com/vocalizations-speech/speech...

I didn't quite get the trick with uninterleaving the lines.

You're in luck -- I managed to dig up my WIP notes from a year ago.

https://imgur.com/xLcvLIm

As you can see, the raw phase waveform is very "wavy", as might be expected. It oscillates rapidly, making it hard to see the patterns. But if you go to the tweets I linked above, you'll see the phase is much smoother in those images. How did I do it?

The key is to focus on every other line. Notice that if you simply pay attention to every odd row, it will be smooth.

I think I simply did "row 0, row 2, row 4, ... row n" followed by "row 1, row 3, row 5, ... row n + 1"

As for the fft of the fft trick for phase, I'm rsync'ing all of my old code and demo images to here:

https://battle.shawwn.com/sdb/voicecloning/

You may be interested in the png images, in particular the ones with "phase" in the names. You can probably ignore all the code except repl2.py.

Those images were generated via unknown methods -- sadly my repl sessions weren't saved. But, I happened to write down in repl2.py how the tweet images were generated:

  cv2.imwrite(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase-fft-abs.png"), np.abs(np.fft.fft2(-1+2*1/255*cv2.imread(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase.png")))))

  cv2.imwrite(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase-fft-abs2.png"), -1+2.0*np.abs(np.fft.fft2(-1+2*1/255*cv2.imread(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase.png")))))

So, input: https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro...

Then, using the code above, the result: https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro...

I've verified that it still works. I think you can wget those images and copy-paste that code into a python repl.

So the only remaining question is, how was mel-phase-spectrogram-phase.png generated? Unfortunately that seems to be lost with the sands of time. But, as a hint, I think it was simply a matter of turning the phase component into x,y using atan2, then turning it into blue and red.

Also, completely unrelated, but I once made a super high resolution mel spectrogram that looked way cool and I can't resist showing it off: https://battle.shawwn.com/sdb/voicecloning/ultra-mel.png

I did all this when making 'Dr Kleiner sings "I Am the Very Model of a Modern Major General"' around a year ago.

https://www.youtube.com/watch?v=koU3L7WBz_s&ab_channel=Shawn...

Kinda funny that all of this visualization work was just to make memes, but the quest to meme turns out to be surprisingly motivating.

https://battle.shawwn.com/sdb/voicecloning/demo_output_101.w...

https://battle.shawwn.com/sdb/voicecloning/demo_output_75.wa...

Anyway, I think there's a lot left to discover in terms of audio visualization! I would definitely encourage you to play around with the phase component. The results can be pretty striking, as you can see from the "Result" image above (https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro...).

Sorry for the scattered explanation -- it's 4am here, but I wanted to give you some kind of writeup, even if it's rather disjointed. If you have more questions, be sure to ask! I can give better details tomorrow.

Thanks for the notes! Another idea I've been thinking about is to capture the phase by splitting ACF into multiple complementary parts at the spectral density step, i.e. when we draw the spectrogram, we use |r exp(i phi)|^2 of the FFT output and that |...|^2 drops the phi. However, we could split this |...|^2 into a sum of a few terms that would add up to the same spectral density, but would separately capture the phase. In the simplest case, |x + iy|^2 = x^2 + y^2, can be interpreted as two half-spectrograms: one for x^2 and another for y^2. If colored the same, we'd get the original spectrogram, but if colored differently, we'd know values of x^2 and y^2 and that's enough to know the phase up to a quadrant.

> those "Mandala" images

Chladni Plates? A few links from past research:

https://www.comsol.com/blogs/how-do-chladni-plates-make-it-p...

Video example - https://www.youtube.com/watch?v=dPTnGEEoFf4

https://www.youtube.com/watch?v=CR_XL192wXw

interactive version - https://www.dynamicmath.xyz/calculus/chladni-patterns/

https://www.youtube.com/watch?v=tFAcYruShow

Yeah, that's where my journey started. I've explored those quite a bit, even wrote a GLSL solver (wave-glsl.github.io/web) to visualize height maps of the u_tt = u_xx + u_yy equation (there is a shallow water equation solver there as well). Problems were that:

(1) Making a stable and fast solver is very difficult. A simple solver for the canonical wave equation is fast, but unstable, so the solution has to be periodically adjusted to avoid NaNs. A stable solver, even for the simplest equation, would be 20x slower. For complex cases, we'd have to involve the Floquet theory, but that would bring an already slow solver to a halt.

(2) A wave diff equation can barely visualize a select frequency, not even a simple mix of frequencies or let alone music. The thing is such wave equations and their boundary shapes have a few select "resonance frequencies" that produce semi-stable patterns. Even a tiny step from a stable frequency, e.g. 6.1 Hz vs 6 Hz, and the solution turns into a mix of unstable patterns morphing one into another, which is cool, but not visually appealing. Mixing multiple frequencies together often produces an unstable mess and even if a pattern forms, you're never sure if it's the pattern for that frequency or just a transient shape, and if it's transient, you can't know if it's due to numeric errors or due to the nature of the equation.

(3) Limited resolution. The rule of thumb is that on a 1000x1000 px screen, the densest Chladni pattern would make 500 full wave repetitions, one pixel per positive and negative sides of the wave. This means we can render only the 500 different frequencies, with 500 Hz slowly turning into a mess due to rounding errors (2 pixels per wave period isn't really enough). Increasing the internal solver buffer to say 4096px brings fps down to 3-4.

However, despite all this, Chladni patterns are hiding something very remarkable, that seems to be glossed over in technical papers. If you imagine that a Chaldni pattern is a water or glass surface, with reflective and refractive properties, and look at the reflection of a simple symmetric object, e.g. a ring, you'd see something resembling a 3d hologram: all these inter-reflections will produce a "virtual image" or remarkable complexity. This picture can be taken by a hi-res camera, but visualizing it with GLSL is again very difficult: the raytracer needs to be outrageously precise.

I checked the live demo with some music. I was expecting something different and maybe you agree that it would be a better visualization.

Right now, the visual experience is like watching movement through a high-speed tunnel. I was expecting the "mandala" you mentioned in the sense that the end result is the accumulated visualization of all waves.

The sound representation would not disappear out of the borders. The first sound recorded would be stored as a narrow outside ring right next to the circle limit. The next sound would be stored as another narrow ring right before the first one. And it would continuously being accumulated sound after sound. The final result would be like a tree cut. It would have a final image representation the whole song, not just sequential snapshots of the sounds included in the song as it is now.

Anyway, congrats for the project! It is awesome and inspiring!

If you're on desktop, try moving your mouse vertically. "Up" seems to zoom in, and "Down" zooms out. Fully zoomed-out, I think it's closer to what you expected to see.

That's it! Thanks for the tip

I'm guessing that triple correlation could also be rendered into pretty pictures, did you try that?

Heh, you're reading my mind. Try the URL below, but don't increase fps/n params, as that will eat all your GPU cycles very quickly:

soundshader.github.io/?s=acf3&n=512&fps=1&acf.decay=0

It effectively computes the bispectrum as B(p, q) = F(p)F(q)F^(p+q) and runs the inverse 2D FFT to restore the triple autocorrelation. The results are interesting, but not impressive and very GPU intensive (NxNxlog(N) per frame is slow). In any case, I strongly believe that bispectrum is hiding something interesting and I just haven't figured how to see it.

Cool! Maybe it's because it contains phase information which is not that relevant to hearing. FWIW I remember long time ago using it for image processing and the trick was to look at a sections of it, i.e. TC(p1, p2) where one of p1 or p2 was fixed.

Probably not related at all, but this reminded me of the Fourier-Mellin transformation, of which a nice overview can be found here[1], used for image registration.

Images are Fourier transformed, and the result is transformed to log-polar coordinates. This turns rotation and scaling in the source image into translations in the resulting log-polar data.

Anyway, fun stuff, thanks for the share!

[1]: https://sthoduka.github.io/imreg_fmt/ (follow link to the pipeline description)

I like the vaguely mathematical connotation of the word "ornament" here, I have never encountered that as a technical term before. It makes me realize that ornament is a good description for many of my favorite mathematical concepts/structures.

Your comment reminds me of the search for the mandelbulb fractal. It seems a fitting comparison, a bulb being a sort of ornament.

Anyway, interesting work.

An ornament is something that's almost invariant under some transforms. For example, these ACF images are almost invariant under some rotations. Sound waves are almost invariant under temporal shifts. I'd argue that what makes an ornament look good is the ease of recognizing those transforms and what makes sound sound sound is the ease of recognizing those temporal shifts.

This is gorgeous! I'm going to have to spend more time with this. Very interesting approach

Are you familiar with the "Circle of fifths"? That's the first thing that came to mind when I saw your circle patterns, and it directly relates to "harmony".

ACF composes all harmonics together and that's what likely makes the images visually appealing. However ACF doesn't give special treatment to harmonics that are exactly N octaves apart, e.g. A4 and A7 notes.

Would ACF be better if did give special treatment to those harmonics? I understand they are an arbitrary distinction, but humans do seem to like them.

It would. The 12 notes are usually mapped to 12 colors, and ideally the sound image would reflect that. One "brute force" way to do that is to split the spectrum into 12 parts, draw 12 ACF images and then mix them. A less brute force approach is to tweak ACF to recognize that F-2F-4F-etc frequencies are specially related, even more specially than just F-2F-3F-4F-etc.