Sharpened Cosine Distance as an Alternative for Convolutions | Hacker News Reader

Sharpened Cosine Distance as an Alternative for Convolutions

rpisoni.dev

72 pointsgk117 comments

Cosine features tend to measure if two things point in the same direction. Convolutional features tend to measure how much of a template was mixed into a signal. As such, one can expect cosine features to produce very similar results to a convolutional feature followed by a Sigmoid or Tanh. Remember what the first style transfer papers used for feature extraction? Conv+Tanh.

After that, cosine features have been used with great success for generating the attention masks in transformer architectures. If I remember correctly, the original Transformer paper also suggested to normalize the features and to pre-scale the result with sqrt(1/N).

So the new part in this formula is mostly the pow. The other parts (cosine distance & dividing by the norm) are already pretty much battle-tested in speech recognition. And the sign is just needed to restore what pow with an even exponent loses.

In short, this suggests that applying a gamma-like transform to features increases contrast.

The Sharpened Cosine Distance animation uses a different input. How would it respond to the reverse pattern in the input or the square pulse?

Indeed. If the problem with convolution is that the rectangular pulse gives a stronger signal than the ramped pulses (which match the convolution kernel), then please show how sharpened cosine distance would behave differently in this problem case.

I can't understand why they used completely different input signals as a comparison.

Is convolving using some sort of smooth derivative of each useful? Or, is that too sensitive to noise?

This twitter thread (https://twitter.com/_brohrer_/status/1232063619657093120) delivers.

Interesting. However, a matched filter is the optimum detector wrt maximizing SNR of the output signal (the features).

https://www.sciencedirect.com/topics/engineering/matched-fil...

https://en.m.wikipedia.org/wiki/Matched_filter

I wonder if this technique has been tested in “noisy” signal environments? Or is this just useful in infinite SNR applications?

optimal among linear functions, and the noise is gaussian...

of course noise is not gaussian in many interesting applications and nothing is stopping a nonlinear function from getting better results than a matched filter, we just have no theory to discover what such a function would be

so yes, like you, i'd need to see results on actual data before spending time to learn more about it...

In my opinion, this looks promising:

https://github.com/stevenwalton/SCS-CCT

4% improvement over Compact Transformers, which is considered SOTA

EDIT: As meragrin_ pointed out, I read the graph the wrong way. It's actually 4% worse. Sorry.

Where do you see improvement? According to the graph and text, there is a 4% decrease in performance. It is slightly worse.

I tend to think of a matched filter as an "optimum denoiser" (in the mean squared error sense with white gaussian noise), but it's not an optimum detector in the general case, or an optimum "de-interferer".

The case shown here should be more resilient to changes in signal amplitude and non-matching interference.

Of course, because the kernel is actually learned, it can adapt to interference and try to be a "matched filter to what we want, plus anti-matched filter to interference" (i.e. give near-0 response to common interference signals). But SCS does seem like a more natural way to achieve a similarity metric and feature detection. (although this non-flexibility might work against it)

This thread seems to summarize the state of this method well: https://twitter.com/_clashluke/status/1497092162193793028

I definitely buy that there is a use case for this kernel. Whenever I see this kind of optimization, I feel like it's at least flirting with classical feature engineering. Not that there's necessarily anything wrong with that. You could probably say the same thing about Relu if you wanted. I just think the "it responds better to certain features" argument, which is how I'm understanding this, can quickly throw you back into some classical computer vision work where you're trying to hand optimize instead of relying on gradient descent to find your features

> I feel like it's at least flirting with classical feature engineering

I, for one, would welcome more of a bridge between intentional human engineering and machine learning. Right now we have inscrutable billion-node language processing models that cause endless arguments about whether they actually know English or not, but don't actually integrate well with anything humans are actually doing. I'd love to see AI get mixed more fluently with manual solutions. It's too either-or right now; the best solutions will come from a mix of smart engineers working seamlessly with focused AI.

I remember the original twitter thread on the topic but haven't heard much about it since then. It would be interesting to see a nice arxiv submissions with some initial experiments across common tasks and architectures. Would also be interested in hardware efficient versions for fixed point data.

Here is some background and more thoughts/references on this on this topic: https://github.com/brohrer/sharpened-cosine-similarity

>it does not give the highest activation for a signal that matches the kernel

This has the feeling of being an important property, but it's not clear to me whether it truly is. Do I actually care whether the output of my feature is extractor is "how well does the signal match the kernel" as opposed to some other function of the signal and kernel?

Not being familiar with cosine distance, I found an NIST link: https://www.itl.nist.gov/div898/software/dataplot/refman2/au....

It seems like the main difference between convolution and cosine distance is the normalization. For example, if I associate vector A with the windowed data set and B with the cosine difference "kernel", then cosine difference will have a larger magnitude than a convolution for small A.

Cool idea. I plan to give it a whirl some day.

> A feature is detected independent of its sign or scale as long as the scale is significantly larger than p.

Pretty sure that is supposed to end in "... than q." instead. p is the sharpness, q is the noise scale (more or less).

Reply on news.ycombinator.com