Hacker News Reader

Comment by noduerme | Hacker News Reader

This is a fascinating take as far as I can follow it, which unfortunately is not that far. But does any of this formal logic help with deriving a function that describes a vector? Because it seems like the greatest inefficiencies and bottlenecks in big data analysis e.g. training networks still boils down to how to find functions that approximate output comparable to expected vectors, whether that's done by symbolic regression or layers of transformations. It would be "magic" if you could operate only on the vectors-as-functions without needing to distill or compress the relationships between their inputs and outputs somehow.

View on HN

Sure. You can Fourier transform a vector, drop some number of the least contributing terms (frequencies) and store only the remaining coefficients. Which is essentially the basic idea behind MP3 and JPEG compression. You're trading space for time, of course, as now to get an approximation of the original vector you have to apply reverse Fourier transform first.

You may be thinking of a vector as a concrete collection of values, like a vector in R^3: [x y z]. This piece is about abstract vector spaces, their properties (vector addition, scalar multiplication, etc.) and specifically how functions meet the definition, giving you vector spaces of functions (function spaces).

So the idea is that if you two functions, f and g, and a scalar b, then you can do stuff like:

f + g = g + f

b(f + g) = bf + bg

The existence of (-f) so you have:

f + (-f) = 0

Where 0 is the zero function (which also must exist for function spaces).

Isn't this just describing a tautology?

I was reading here earlier today about the naming of the constant for light as c, and I had a question which I was too embarrassed to ask. It is this: In e=mc^2, what are the units, and if the units aren't defined and it's just a relationship, then why specify c^2? What's the point of squaring a constant, since it's just another constant?

Not that I understand a damn thing about that equation. But the idea that two functions can be thought of operating additively on a vector - or space - seems... trivial.

This is where I fear I am too stupid to understand the value of this.

But the idea that two functions can be thought of operating additively on a vector - or space - seems... trivial.

No, the functions are not operating on a vector, the functions themselves are vectors, which means you can do things like define a linear transformation L and then you have things like this:

L(af + bg) = aL(f) + bL(g)

You may also define a norm on a function space (such as an L^P space [1]), ||f||_p, which maps functions to non-negative real numbers and obeys the properties of norms that we expect, such as the triangle inequality:

||f + g||_p <= ||f||_p + ||g||_p

[1] https://en.wikipedia.org/wiki/Lp_space

Thanks. I'm trying to understand this. So take a function that returns a random shuffled deck of cards with 52! possible real number vector outputs. What is the generalized insight from treating this function itself as a vector space... is there a shortcut to monte carlo-ing a million random shuffles if you can "divide" that function by one which produces a straight flush?

> So take a function that returns a random shuffled deck of cards with 52! possible real number vector outputs.

that is not a function in the (mathematical) sense that the article is talking about. a function is a mapping from a set of inputs to a set of outputs, and the same input will always map to the same output. (in programming terms it's what you would call a "pure function")

then remapping them by some arithmetic is tautological, isn't it?

not quite; in the most general sense the arguments to and result of a function (the 'domain' and 'range' in mathematical jargon) need not be numbers but any set of mathematical objects. the heavy work went into proving properties of these generalised functions that were universally true, and showing that they were isomorphic to structures built up in other branches of mathematics.

as a side note, one very important technique/idea in mathematics (in general, not just in the area of functional analysis) is describing something in terms of a set of properties that is both as general as possible and as minimal as possible. for instance numbers can be added, subtracted, multiplied and divided, with "obvious" real-world interpretations. mathematicians then asked themselves what properties exactly the numbers had to possess in order for those operations to be defined, and then they proceeded to find other classes of mathematical objects that also had those properties, and suddenly we were able to "add" and "multiply" things that had no obvious physical interpretation for those operations. but since their structure was mapped to the structure of the numbers, those operations could be mechanically defined over them, and you had all sorts of mathematical tools at your disposal.

here a similar thing was done with functions. there had been a lot of work put into studying the operations you could do on "vector spaces", a mathematical structure that generalised the notion of a vector as a collection of numbers. then mathematicians noticed that if you took the minimal collection of properties something needed to have in order to be a vector space, functions satisfied all those properties. and voila - everything that you could prove about vector spaces (and again, it was a whole lot) was suddenly applicable to functions as well.

(why some of this seems a bit tautological is that it also follows the properties of the real numbers, and even non-mathematicians have had a lot of intuition built up about how numbers behave. but it is by no means guaranteed that every mathematical construct will have these same properties.)

Assuming this is in good faith, the units are:

- c: speed of light in meters per second

- m: mass in kilograms

- energy: joules

yes it was in total good faith. soooo... the metric system just happens to perfectly conform to meters per second squared times kilograms equaling joules? That seems... mind-blowing since it was invented before kilograms and joules would have been interchangeable.. ??.

If you express it in a different unit system all E, m, and c will take different numerical values, but the relationship will still be true.

So there’s nothing special about the metric system. When we want to discuss this kind of relationship without reference to human convention we talk about a quantity’s dimension (not geometric like 3D). A Meter and a foot both have dimensions of length. c has units of length/time. 1 kilogram and 1 gram both have units of mass. And so on.

https://en.wikipedia.org/wiki/Dimensional_analysis

Yep. This was going to be the next phase: there isn't anything special about the SI system, but it is consistent. The poster didn't seem prepared for "or any other velocity units you want".

For the original poster, you could equally use c in miles per hour, mass in "pounds", and get whatever that produces for energy (I'm sure there's a name for it).

This is also why Imperial units produce such amusing things as Foot-Pounds (for torque). The math all works out in the end, but you get some amusing numbers along the way.

Kilograms and joules have been interchangable (well, they've had a defined relationship) for a long time though, you can calculate work in a completely Newtonian context and see that the energy requires is linearly related to the mass and quadratically to velocity (like with a spring).

That the direct conversion from mass to energy follows the same shape isn't really surprising, it sort of has to.

That said, the joule was only explicitly defined as kgm/s^2 in 1946 (or 1935), after Einstein and nuclear physics.

The kinetic energy of a body, in purely Newtonian mechanics, is Ec = (m × v²) / 2. Since energy is measured in J, mass in kg, and speed in m/s, it follows nicely that J = kg × m² / s². Or, we can also take the potential energy of a body at some height h above the Earth: Ep = m × h × g, where g is the gravitational acceleration, ~9.8m/s², and we get the same units.

It's really not in any way surprising - this is basically the definition of the Joule. The link between units of kinetic energy and units of thermal energy is actually more surprising.

The surprising thing about E=mc² is that it gives a definition of energy for a completely stationary body outside any external field.

One way to look at it is actually that this is simply the kinetic energy of the body, and that all "stationary" bodies are in fact moving with speed c on the t coordinate in 4D space-time ("a body which is not moving in space at all is moving with speed c towards the future"). [Note that of course speeds are all relative to some reference frame.]

It's not tautological — to pick one of the properties, you can conceive of spaces of functions where there is no -f to a function f. For example, consider the space of all positive-valued real functions.

Consider otherwise functions which take colours and output letters or the alphabet. Letters of the alphabet can't be added or subtracted, so there's no vector space structure on that.

On the flip side, vector spaces can be defined not just on real numbers, but complex numbers as well, or even other sets — specifically, any "field", that is, any set with addition, subtraction, multiplication, and division defined on it in a self-consistent way. There are even finite fields; vector spaces over three are relevant in cryptography.

If these rules seem silly and abstract to you (shouldn't a function just be a function?!), well, that's mathematics! By elucidating the very specific conditions under which results hold, and abstracting away all the irrelevant details, you end up with results of incredible generality.

You should look into the pigeonhole principle.

Looked it up. How does that apply to this, or am I lacking the imagination to see it?

If the data set is large enough then there is no way to represent it as a functional relationship between finite dimensional vector spaces. In fact, this problem already is visible in existing large neural networks because they can only work with data that conforms to the dimensional constraints of the input space. It's why image transformers trained on NxM images don't work on any other grid size.

ah. I mean, the dataset can be infinitely large and still be covered perfectly by a function, if it was generated from a function. That's why absurd functions can be found that overfit for some subset of basic real world results. (Ask anyone who built a funky physics system for a videogame in the 90s). I think what's more interesting is the question of what that essential function is for a stop sign or a pedestrian, as opposed to the function for finding it in a 512 grid or something.

I suspect the idea is that there can be many functions that fit a particular vector, because there are more functions than vectors, but I nay be wrong.

Reply on news.ycombinator.com