Comment by sixo | Hacker News Reader

Comment by sixo | Hacker News Reader

Parent Full thread

A few questions occur to me while reading this, which I am far from qualified to answer:

- How much of this structure survives if you work on "fuzzy" real numbers? Can you make it work? Where I don't necessarily mean "fuzzy" in the specific technical sense, but in any sense in which a number is defined only up to a margin of error/length scale, which in my mind is similar to "finitism", or "automatic differentiation" in ML, or a "UV cutoff" in physics. I imagine the exact definition will determine how much vectorial structure survives. The obvious answer is that it works like a regular Fourier transform but with a low-pass filter applied, but I imagine this might not be the only answer.

- Then if this is possible, can you carry it across the analogy in the other direction? What would be the equivalent of "fuzzy vectors"?

- If it isn't possible, what similar construction on the fuzzy numbers would get you to the obvious endpoint of a "fourier analysis with a low pass filter pre-applied?"

- The argument arrives at fourier analysis by considering an orthonormal diagonalization of the Laplacian. In linear algebra, SVD applies more generally than diagonalizations—is there an "SVD" for functions?

Convolution with dirac delta will give you an exact sample of f(0), and in principle a whole signal could be constructed as a combination of delayed delta signals - but we can't realize an exact delta signal in most spaces, only approximations.

As a result we get finite resolution and truncation of the spectrum. So "Fourier analysis with pre-applied lowpass filter" would be analysis of sampled signals, the filter determined by the sampling kernel (delta approximator) and properties of the DFT.

But so long as the sampling kernel is good (that is the actual terminology), we can form f exactly as the limit of these fuzzy interpolations.

The term "resolution of the identity" is associated with the fact that delta doesn't exist in most function spaces and instead has to be approximated. A good sampling kernel "resolves" the missing (convolutional) identity. I like thinking of the term also in the sense that these operators behave like the identity if it were only good up to some resolution.

1. Numerical methods for solving differential and integral equations are algorithms for solving algebraic equations (vector solutions) that arise from discretizing infinite-dimensional operator equations (function solutions). When we talk about whether these methods work, we usually do so in terms of their consistency and stability. There is a multistage things that happens here: we start by talking about the well-posedness of the original equation (e.g. the PDE), then the convergence of the mathematical discretization, and then examine what happens when we try to program this thing on a computer. Usually what happens is these algorithms will get implemented "on top" of numerical linear algebra, where algorithms like Gaussian elimination, and different iterative solvers, have been studied very carefully from the perspective of floating point rounding errors etc. This kind of subsumes your concern about "fuzzy" real numbers. Remember that in double precision, if the number "1.0" represents "1 meter", then machin epsilon is atomic scale. So, frequently, you can kind of assume the whole process "just works"...

2/3. I'm not really sure what you mean by these questions... But if you want to do "fourier analysis with a filter preapplied", you'd probably just work with within some space of bandlimited functions. If you only care around N Fourier modes, any time you do an operation which exceeds that number of modes, you need to chop the result back to down to size.

4. In this context, it's really the SVD of an operator you're interested in. In that regard, you can consider trying to extend the various definitions of the SVD to your operator, provided that you carefully think about all spaces involved. I assume at least one "operator SVD" exists and has been studied extensively... For instance, I can imagine trying to extend the variational definition of the SVD... and the algorithms for computing the SVD probably make good sense in a function space, too...

Your can replace the real numbers with the rational numbers and maintain all of the vector structure.

If you wanted something more quantized, you can pick some length unit, d, and replace the real numbers with {... -2d, -d, 0, d, 2d,... }. This forms a structure known as a "ring" with the standard notion of addition, subtraction, and multiplication (but no notion of division. Using this instead of R does lose the vector structure, but is still an example of a slightly more general notion of a "module". Many of the linear algebra results for vector spaces apply to modules as well.

> If it isn't possible, what similar construction on the fuzzy numbers would get you to the obvious endpoint of a "fourier analysis with a low pass filter pre-applied?"

If that is where you want to end up, you could pretty much start there. If you take all real value functions and apply a courier analysis with a low pass filter to each of them, the resulting set still forms a vector space. Although I don't see any particular way of arriving at this vector space by manipulating functions pre Fourier transform.

I'm not sure I quite understand what you're aiming at with these questions, but there are certainly techniques in ML based on thinking of functions as vectors. The first one that comes to mind is Anyboost [1], which views boosting as doing "gradient descent" in function space - where each "gradient step" is not a typical vector (as you'd see e.g in neural nets) but a function, corresponding in practice to a base classifier. Another that is very popular are gaussian processes - one way to think about them is as modeling functions as samples from an infinite-dimensional gaussian (at least some of them).

[1] https://proceedings.neurips.cc/paper_files/paper/1999/file/9...

I’d guess that it would be factored as “nonlinearity”, which might be characterized as some form of harmonic distortion, analogous to clipping nonlinearity of finite-ranged systems?

Perhaps some conjugate relation could be established between finite-range in one domain and finite-resolution in another, in terms of the effect such nonlinearities have on the spectral response.

A fuzzy vector is a Gaussian? Thinking of what it would be in 1, 2, 3 and n dimensions.

Reply on news.ycombinator.com