Diffusion Is Spectral Autoregression
sander.ai
sander.ai
The lower frequencies (roughly below 4KHz) are created by the vocal chords opening and closing at the fundamental frequency, and harmonics of this fundamental frequency (e.g. 100Hz + 2/3/400Hz etc harmonics), with this frequency spectrum then being modulated by the resonances of the vocal tract which change during pronunciation. What we perceive as speech is primarily the changes to these resonances (aka formants) due to articulation/pronunciation.
The higher frequencies present in speech mostly comes from "white noise" created by the turbulence of forcing air out through closed teeth/etc (e.g. "S" sound), and our perception of these "fricative" speech sounds is based on onset/offset of energy in these higher 4-8KHz frequencies. Frequencies above 8KHz are not very perceptually relevant, and may be filtered out (e.g. not present in analog telephone speech).
Joseph Fourier's solution to the heat-equation (linear diffusion) was in fact the origin of the FT. The high-freq coefficients decay (as -t^2 IIRC) in there; the reverse is also known to be "unstable" (numerically, and is singular from the equillibrium).
More over, the reformulation doesn't immediately reveal some computational speedup, or a better alternative formulation (which is usually a measure of how valuable it is epistemically).
(Edit: note that Heat-equation is more akin to the Fokker-Planck eqn, not actual Diffusion as an SDE as is used in Diffusion models).
Connections between fields drive new ideas. And this has especially been the case for recent AI progress. With the speed at which the field is moving, ideas that are obvious to some still have a significant chance of not being tried yet.
Just as the connection between the Kalman filter and RNN models or the significant similarities between back-propagation and the whole field of control theory. If it's truly not surprising, then that's just another reason to try it out if nobody else has.
Does everything always need to be immediately "useful"?
Images have a large near-DC component (solid colors) and useful time-domain properties, while human hearing starts at ~20 Hz and the frequencies needed to understand speech range from 300-4 kHz (spitballing based on the bandwidth of analog phones).
What would happen if you built a diffusion model using pink noise to corrupt all coefficients simultaneously? Alternatively what if you used something other than noise (like a direct blur) for the model to reverse?
I'm not sure this changes if you look at a cepstral representation (as suggested in the article). In this case, the DC component represents the (white) noise level in the raw audio space (i.e., the spectrum averaged over all frequencies), so it doesn't have strong semantics either (other than... "how noisy is the waveform?").
As such it seems the statement is that stable diffusion is like an autoregressive model which predicts the next set of higher-order FT coefficients from the lower-order ones.
Seems like this is something one could do with a "regular" autoregressive model, has this been tried? Seems obvious so I assume so, but curious how it compares.
Had just finished watching the Physics of Language Models[1] talk, where they show how GPT2 models could learn non-trivial context-free grammars, as well as effectively do dynamic programming to an extent, so though it would be interesting to see how they performed in the spectral fine-graining task.
Man, reading on mobile phone just ain't the same. Somehow managed to not catch then end of that section. The first reference, "Generating Images with Sparse Representations", is very close to what I had in mind.
Both ADHD people and neurotypicals have deeply structured thoughts. "Serializing" those thoughts without planning ahead leads to the "stream of consciousness" writing style, which includes things like run-on sentences and deeply nested parentheses. This style is considered poor form, because it is hard to follow. To serialize and communicate thoughts in a way that avoids this style, it is necessary to plan ahead and rely on working memory to hold several sub-goals simultaneously, instead of simply scanning back through the text to see which parentheses have not been closed yet.
It could also be simply that ADHD people have "branchier" thoughts, hopping around a constellation of related concepts that they feel compelled to communicate despite being tangential to the main point; parentheses are the main lexical construct used to convey such asides.
That said, knowing when to use dashes—longer than hyphens—can help mix things up.
When you explain it serially you are forced to choose a spanning tree, and people usually stop listening when the spanning tree has touched all the relevant concepts, then they persuade themselves they got the full picture but miss some connections, that make the problem more complex and nuanced.
When graphs have more than one loop, loopy belief propagation doesn't work anymore and you need an another algorithm to update your belief without introducing bias.
I want to be able to simultaneously encode [[Computer Science]] and [[Computer]] [[Science]].
And [[Project1/Computer Science]] to at least provide a connection to [[Project2/Computer Science]].
Communication is kind of the game of transmitting the information in such a way that your interlocutor internal representation of things ends up mapping to yours. Low dimensional embeddings are often very useful, but sometimes graph are not planar. Symmetry is usually useful, and a symmetric higher dimensional embedding is often better, because the symmetry constrain it more making it easier to be sure it was transmitted correctly.
When people ends up with different concept maps, in one of which some concepts are located near each other and in the other the same concepts are located far apart, interesting things usually happen when they communicate, ranging from culture enlightenment to culture war.
Some of these mapping are sometimes constrained to 3d, by things like memory palaces, (method of loci), but this is somewhat arbitrary, and staying more abstract and working in higher dimension until you "feel" everything fall into the right place intuitively is often preferable, (aka the Feynman method).
I have a basic natural language processing system implemented in Neo4J (what I tried to use before Logseq). But to take notes I like plain text more than a database. Less dependencies.
The problem with embeddings, is I don't know how I would wire that into my workflow yet. Plain text notes have links, I would need a separate interface or mode to browse and analyze the connections.
He can produce whole paragraphs of this semi-regular language and it even has distinct structure and non-standard interactions like in the above sentence.
"One guy write in style"
in (most of the times <this> (or deeper)) style
Also: while I'm not the guy GP references, I am a guy that does that too - or rather did, at some point in the past, until I realized there's like 5 people in my life who could understand this without an issue, even less who'd indulge me or enjoy communicating this way. So over time, I got back to writing like a normal person[0]; I guess conformity is just less mentally taxing.
--
[0] - Mostly - I still use semicolons and single-depth parentheses a lot, and on HN, also footnotes.
Spoken and written languages are presented in a sequential medium. They still represent hierarchical trees in their structure though.
(Notable semi-exception to the linearity are the sign languages, which are are kinematic three-dimensional languages involving two hands, an entire upper body and facial expressions. While I don't speak it, I've read a bit about it, and apparently the most common error for non-deaf people who learn it is to make so-called "split verb" errors. That is to say: to sign in a linear fashion like one would with a spoken language, instead of making use of all the parallel communication options available)
The Deaf community has suffered a lot of discrimination throughout history, and two of the biggest issues are non-deaf people deciding on their behalf what is best for them, and forcing them to use vocal languages (which makes as much sense as forcing a blind person to communicate via colorful paintings) while denying them access to sign language. Ask any Deaf person about Milan 1880 and why Alexander Graham Bell is so controversial in their community. A major driver in this has always been that people who don't know sign languages tend to think of them as funny interpretive miming.
With that in mind comparing sign languages to "haha Italian gesticulation funny" jokes without being aware of the differences can become a form of infantilization.
[0] https://www.deafhistory.eu/index.php/component/zoo/item/1880
Careful with your musings, or you might start thinking semiotically!
Diachrony and synchrony
Also, you should probably enforce some kind of frequency cutoff later when you're generating the high frequencies so that you don't destroy low frequency details later in the process.
I'm not really sure how current video generating models work, but maybe we could get some insight into them by looking at how current audio models work?
I think we are looking at an auto regression of auto regressions of sorts, where each PSD + phase is used to output the next, right? Probably with different sized windows of persistence as "tokens". But I'm a way out of my depth here!
In images, scrambling phase yields a completely different image. A single edge will have the same spectral content as pink/brown~ish noise, but they look completely unlike one another.
So when generating audio I think the next chunk needs to be continuous in phase to the last chunk, where in images a small discontinuity in phase would just result in a noisy patch in the image. That's why I think it should be somewhat like video models, where sudden, small phase changes from one frame to the next give that "AI graininess" that is so common in the current models
I have an example audio clip in there where the phase information has been replaced with random noise, so you can perceive the effect. It certainly does matter perceptually, but it is tricky to model, and small "vocoder" models do a decent job of filling it in post-hoc.
Maybe the original author benanne could give his insight.
That said, the gap between perceptual modalities (image, video, sound) and language is quite large in this regard, and probably also partially explains why we currently use different modelling paradigms for them.
But another issue not mentioned in the article is that in images we can zoom in/out arbitrarily. So the width of a pixel can change – it might be 1mm in one image, or 1cm in another, or 1m or 1km. Whereas in audio, the “width of a pixel” (the time between two audio samples) is a fixed amount of time – usually 1/44.1kHz, but even if it’s at a different sample rate, we would convert all images to have the same sample rate before training an NN. The equivalent of this for images would be rescaling all images so that a picture of a cat is say 100x100 pixels, while a picture of a tiger is 300x300.
Which, come to think of it, would be potentially an interesting thing to do.
Hmm, how does depth affect this? The further away something is in a picture, the more width the pixel represents since it's angular right?
I was talking nonsense here - confusing the visual spectrum of light from red to blue with the visual spectrum of images, as in "how quickly the image changes as you move across the image". The article illustrates the latter concept well.
Huh. Does this mean that pink noise would be a better prior for diffusion models than Gaussian noise, as your denoiser doesn’t need to learn to adjust the overall distribution? Or is this shift in practice not a hard thing to learn in the scale of a training run?
> basically an approximate version of the Fourier transform!
You should take a step back and ask “am I actually muddying the water right now?”