Fitting an elephant with four non-zero parameters
arxiv.org
arxiv.org
That said, the wisdom of the quip has been widely lost in many fields. In many fields data is "modeled" with huge regression models with dozens of parameters or even neural networks with billions of parameters.
> In 1953, Enrico Fermi criticized Dyson’s model by quoting Johnny von Neumann: “With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.”[1]. This quote is intended to tell Dyson that while his model may appear complex and precise, merely increasing the number of parameters to fit the data does not necessarily imply that the model has real physical significance.
What can't be fit by declaring the amount of dark matter that must be present fits the data? It's unfalsifiable, just because we haven't found it, doesn't mean it doesn't exist. Even worse than string/M-theory which at least has math.
Does the latter even get funding?
This is partially because there are two ways to detect dark-matter. The first is gravitational lensing. The second is the rotatinal speed of galaxies. There are some galaxies that need less Dark Matter to explain their rotational speed. We can then cross check whether those galaxies cause less gravitational lensing.
Besides that, the gravitational lensing of galaxies being stronger than the bright matter in the galaxies can justify is hard to explain without dark matter.
Besides the idea 'not all mass can be seen optically' is not that surprising. The many theories on what that mass might be are all speculation, but they are treated as such.
Both of these are pretty much ruled out though: you can't plausibly add enough brown dwarfs, and if it's black holes then you should see more lensing events towards nearby stars given how many you'd need.
But they're both concrete predictions which are falsifiable (or boundable such that they can't be the dominant contributors).
Not even anything that extreme. What's ruled out is interaction via electromagnetism (or if you want to get really nit-picky, electromagnetic interaction with a strength above some extremely low threshold).
It is actually a satisfying theory with regard to the Occam razor. We don't have to change our laws of physics to explain the abnormal rotations of galaxy, we just need "stuff" that we can't see and yet interact gravitationally. When we have stuff like neutrinos, it is not that far fetched. In fact, though unlikely given our current understanding of physics, dark matter could be neutrinos.
If, as it turn out, the invisible stuff we call dark matter doesn't follow the laws of physics as we know them, then the dark matter theory is falsified and we need a new one (or at least some tweaks). And it may actually be the case as a recent paper claims that gravitational lensing doesn't match the predictions of the dark matter theory.
The main competitor to dark matter is modified gravity, which calls for no new stuff, but changes the equations for gravity. For the Occam razor, adding some random term to an equation is not really better than adding some invisible but well characterized stuff, especially when we consider that the equation in question is extremely well tested. It is, of course, also falsifiable.
The problem right now is not that these theories are unfalsifiable, it is that they are already pretty much falsified in their current form (dark matter less than modified gravity), and some rework is needed.
Even this is granting too much: "seeing it" and "seeing its effects" are the same thing. No one has ever "directly seen", in the sense that internet DM skepticism demands, anything other than a photon.
The problem with dark matter is that there does not exist any second relationship from which to verify its existence, like in the case of normal matter, which takes part in a variety of interactions that lead to measurable effects, which can be compared.
The amount and the location of dark matter is computed from the gravitational forces that explain the observed movements of the bodies, but there are no additional relationships with any other data, which could corroborate the computed distribution of dark matter. That is what some people mean by "seeing".
Radical departure may well be needed, for other reasons too.
This is exactly it! Dark matter is strictly defined by its effects. The only 'theory' part is a belief that it's caused by yet to be found particle that's distributed to fit observations. Take all the gravitational anomalies that we can't explain with ordinary matter, then arbitrarily distribute an imaginary 'particle' that solves them: that's DM.
The problem is that the language used to talk about DM is wrong. It's not that DM doesn't interact with EM, or the presence of DM is causing the galaxies to rotate faster than by observed mass. These are all putting the cart before the horse. What we have is unexplained gravitational effects being attributed to a hypothetical particle. If we discovered a new unexplained gravitational property, we would merely add that to the list of DM's attributes rather than say "oh then it can't be DM".
All physical entities are defined by their effects! Suppose we found axions and they had the right mass to be dark matter. Would that mean we now "really knew" what dark matter was, in your sense? No, it would just push the defining effects further back - because all an axion is is a quantum of the (strong CP-violation term promoted to a field).
Just like the electromagnetic field is the one that acts on charged particles in such and such a way, and a particle is charged if the electromagnetic field acts on it in that way. There's no deeper essence, no intuitive "substance" with some sort of intrinsic nature. All physical properties are relational.
You're being too kind. It's worse. Especially when (in my understanding anyway) that added term doesn't even explain all the things dark matter does.
This obviously can match almost anything and it has extremely low predictive power (many future observations may differ from predictions, which can be accounted by some dark matter whose distribution was previously unknown), so it is a much worse explanation than a modified theory of gravity that would have only a finite number of additional parameters.
The point is that even with current observational data there's no reasonable distribution of dark matter that correctly explains all evidence that we have.
Your intuition that "if I have an infinite number of degrees of freedom anything at all can be fit" is leading you astray here.
At minimum this is a ~200 billion parameter model, and more if you’re looking at smaller structures.
With dark matter it’s two steps removed where we’re inferring the behavior of baryonic matter and then inferring the amount of baryonic matter we aren’t observing and then calculating the existence of dark matter to get that behavior after accounting for undetected baryonic matter.
So there’s no simple way to calculate it from say looking at the Milky Way alone and extrapolating from the baryonic mass of the rest of the universe. Trying to approximate things from a representative sample is its own problem.
1. Dark matter does not meaningfully introduce billions more parameters into cosmological models than they already have, and
2. Individual galaxies' dark matter fractions are not essential to (not proving, but) strongly suggesting dark matter exists.
If the observed dark matter fractions of all known galaxies were 0% but the CMB was unchanged we wouldn’t assume dark matter exists. Thus your #2 is false. There’s infinite models consistent with any observation so finding something after a model was created for other reasons is useful as validation, but the chain of logic is still dependent on the prior observations not the model.
In a meaningfully different cosmos different observations would have happened and different models would exist. Trying to pick out specific experiments as sufficient on their own glosses over that particular limitation.
No, astrophysicists would eventually figure out something was up when they couldn't replicate the actual spectrum with dark-matter-free simulations. Why would you assume otherwise? Unless you want to dig into the assumptions of the scenario, in which case you're probably proposing a self-inconsistent universe so of course you can draw whatever conclusions you want from it.
> There’s infinite models consistent with any observation...
You can't actually believe this and still believe in science. If observations don't constrain models, then there is no point in observing. And in the long run, there's asymptotically no difference between "prior observations" and later observations. They're just observations that all go into the same model-constraining mill. Scientists are not fools, and are capable of realizing when an initial observation put them on a wrong trail.
You're still barely touching the real point. This all just sounds like rationalizations to avoid the fact that dark matter, for now at least, and for all that it genuinely sucks, is the Occam's razor explanation for the full suite of observations. Why is this so hard to accept?
There are many theories to explain dark matter observations. MOND is not a competitor with 'dark matter', because MOND is a theory and it tries to explain some aspects (spiral galaxy rotation) of what is observed as the dark matter problem, which consists of many more observations. There is no competition here. There are other theories to explain dark matter, like dark matter particle theories involving neutrinos or whatever, and these may be called competitors, but dark matter itself is not a theory, but a problem statement.
Whereas you can have many proposals for what dark matter is, provided it is capable of being almost entirely only gravitationally interacting, and there's enough of it.
MOND has had the problem that depending which MOND you're talking about, it still doesn't explain all the dark matter (so now you're pulling free parameters on top of free parameters).
Wikipedia has: "dark matter is a hypothetical form of matter that appears not to interact with light or the electromagnetic field ... Although the astrophysics community generally accepts dark matter's existence, a minority of astrophysicists, intrigued by specific observations that are not well-explained by ordinary dark matter, argue for various modifications of the standard laws of general relativity. These include modified Newtonian dynamics, tensor–vector–scalar gravity, or entropic gravity."
Even its name, "dark matter", sort of strongly implies this. If someone were just trying to refer to the observations, rather than a specific explanation for the observations, wouldn't they just say "abnormal galaxy rotation curves" rather than "dark matter"?
I'm not saying Wikipedia is an end-all-be-all source on this, I'm just asking where you're getting this alternate definition. If it is somewhere reliable then perhaps the article needs to be rephrased.
Tons of things - just like there are tons of things that can't be fit by declaring the amount of electromagnetically-interacting matter that must be present fits the data.
You can fit anything you like by positing new and more complicated laws of physics, but that's not what's going on here. Dark matter is ordinary mass gravitating in an ordinary way: the observed gravitational lensing needs to match up with the rotation curves needs to match up with the velocity distributions of galaxies in clusters; you don't strictly need large scale homogeneity and isotropy but you really really want it, etc. Lambda-CDM doesn't handle everything perfectly (which in itself demonstrates that it's not mindless overfitting) but neither does anything else.
Which MOND does: it creates huge problems fitting into GR.
Whereas dark matter as just regular mass that interacts poorly by other means does not.
Nobody probably believes MOND as such is some fundamental theory, rather as a "theory" it's sort of a stepping stone. Also MOND is used often interchangeably (and confusingly) with modified gravity theories in general.
Those are all the same thing, the shape of spacetime. The only thing DM adds is a backstory that this shaping comes from hypothetical undiscovered particles with properties that match observations.
Re: where it says "using simulations with dark matter", we can't simulate DM because it doesn't have any properties beyond our observations. All we do is distribute amounts of it to match observations. It could be "Dyson spheres with EM shields" and the results would be the same.
That doesn't take away the fact that when you work with the slightly more specific theory of "particle dark matter" it produces real results. And I believe there's a lot more work over the years in similar areas. It doesn't get talked about because it's not sexy, so people who only follow cosmology when there's drama don't hear about it. That was just the example at the top of my mind because I'd seen it recently, and the result is really quite spectacular. Did you watch it through?
For those who are interested, you can watch Freeman Dyson recount this conversation in his own words in an interview: https://youtu.be/hV41QEKiMlM
Hodgin and Huxley did ground-breaking work on squid's giant axon and modelled neural activity. They had multiple parameters extracted from 'curve fitting' of recorded potential and injected currents which were much later mapped to sodium channels. Similarly, another process to potassium channels.
I woudnt worry too much having multiple parameters -- even four when 3 can't just explain the model.
The physics equivalent is something like eternal inflation as an explanation for apparent fine-tuning - except that even if it's correct it's still absolutely nowhere near as complex or as contingent as biology.
Which makes the only truly zero parameter system the collection of all systems, in all forms.
For example, how many parameters does the Standard Model have? It's not clear what you count as a parameter. Do you count the group structure, the other mathematical structure that has been "fitted" through decades of comparisons with experiments?
You are also glossing over centuries of precedent that predate high-energy physics, namely quantum field theory, special relativity, and foundational principles such as conservation of energy and momentum.
You can change measured complexity by altering the baseline assumptions.
I'm always making the joke (observation) that ML (AI) is just curve-fitting. Whether "just curve-fitting" is enough to produce something "intelligent" is, IMO, currently unanswered, largely due to differing viewpoints on the meaning of "intelligent".
In this case they're demonstrating some very clean, easy-to-understand curve-fitting, but it's really the same process -- come up with a target, optimize over a loss function, and hope that it generalizes, (this one, obviously, does not. But the elephant is cute.)
This raises the question Neumann was asking -- why have so many parameters? Ironically (or maybe just interestingly), we've done a lot with a ton of parameters recently, answering it with "well, with a lot of parameters you can do cool things".
Continual "curve fitting" to the real world can create intelligence. What is missing is not something inside the model. It's missing a mechanism to explore, search and expand its experience.
Our current crop of LLMs ride on human experience, they have not largely participated in creating their own experiences. That's why people call it imitation learning or parroting. But once models become more agentic they can start creating useful experiences on their own. AlphaZero did it.
By spelling out what brains are doing it becomes very obvious that it's all simply a sequence of chemical reactions - and yet here we are, having experiences. Software will never have a human experience - but neither will a chimp, or an octopus, or a Zeta-Reticulan.
Mammalian neurons are not the only possible substrate for intelligence; if they're the only possible substrate for consciousness, then the fact that we're conscious is an inexplicable miracle.
And where you decide to assume that non-computable physics happens in the brain based on no evidence?
What a waste of time. You "addressed" it in a completely meaningless way.
Consider the (algorithmic) mechanical process of screwing in a screw into a board. This screw has an "experience" and therefore intelligence. So... The screw is intelligent? Very low intelligence, but intelligent according to this definition.
But we have an even bigger problem. There's the metaset of experiences, that's the collection of several screws (or the screw, board, and screwdriver together). So we now have a meta intelligence! And we have several because there's the different operations on these sets to perform.
You might be okay with this or maybe you're saying it needs memory. If the later you hopefully quickly realize this means a classic computer is intelligent but due to the many ways information can be stored it does not solve our above conundrum.
So we must then come to the conclusion that all things AND any set of things have intelligence. Which kinda makes the whole discussion meaningless. Or, we must need a more refined definition of intelligence which more closely reflects what people actually are trying to convey when they use this word.
Neither, what I'm saying is that the observable correlates of experience are the observable correlates of intelligence - saying that "humans are X therefore humans are Y, software is X but software is not Y" is special pleading. The most defensible positions here are illusionism about consciousness altogether (humans aren't Y) or a sort of soft panpsychism (X really does imply Y). Personally I favor the latter. Some sort of threshold model where the lights turn on at a certain point seems pretty sketchy to me, but I guess isn't ruled out. But GP, as I understand them, is claiming that biology doesn't even supervene on physics, which is a wild claim.
> Or, we must need a more refined definition of intelligence which more closely reflects what people actually are trying to convey when they use this word.
Well that's the thing, I don't think people are trying to convey any particular thing. I think they're trying to find some line - any line - which allows them to write off non-animal complex systems as philsophically uninteresting. Same deal as people a hundred years ago trying to find a way to strictly separate humans from nonhuman animals.
Intelligence appears in gradients, not a simple binary.
Sure, I'm in no way countering such a notion and your snarky comment is a gross mischaracterization of my comment. So far off I have a difficult time believing it isn't intentional.
The "surprise" is not that plants, animals, or even women turn out to be intelligent under the definition of "collection of experiences" but that rocks have intelligence, atom, photons, and even more confusingly groups of photons, the set of all doors, the set of all doors that such that only one door per city exists in the same set. Or any number of meta collections. This is the controversial part, not women being intelligent. Plants are still up for debate, but I'm very open to a broad definition of intelligence.
But the issue is that I, and the general fields of cognitive science, neuroscience, psychology, and essentially everyone except for a subset of computer scientists, agree that intelligence is more than a collection of experiences (including if that collection has memory). In other words, it is more than a Turing Machine. What that more is, is debated but it is still generally agreed upon that intelligence requires abstraction, planning, online learning, and creativity. But all these themselves have complicated nuanced definitions that are much more than what the average person thinks they mean. But that's a classic issue where academics use the same words normal people do but have far more restrictions on their meaning. Which often confuses the average person when they are unwilling to accept this fact that words can have different meanings under different contexts (despite that we all do this quite frequently and such a concept exists in both our comments).
I would define "intelligence" as (1) the ability to learn or understand or to deal with new or trying situations and (2) the ability to apply knowledge to manipulate one's environment.
It turns out that this is also the Merriam-Webster definition [0]. By that definition, yes AlphaZero was learning and understanding how to deal with situations and is intelligent, and yes most machine-learning systems and many other systems that have a specific goal and manipulate data/the environment to optimize for that goal, are intelligent.
By this definition, a non-living, non-conscious entity can be intelligent.
And intelligence has nothing to do with "experiences" (which seem to belong in the "consciousness" debate).
[0]: https://www.merriam-webster.com/dictionary/intelligence
Otherwise this can quickly devolve into the common useless semantic discussion.
1. The world around you 2. The experiences within your (really, the past view of the world around you) 3. Innateness of you (sure, this could be 2 but I think it's also something else) 4. The experience you find + the way you change yourself to impact (1), (2), and (3)
If you think of intelligence as all of these, then you're making the assumption that all that's required for (2), (3), and (4) is "agentic systems", which I think skips a few steps (as the author of an agent framework myself...). All this is to say that "what makes intelligence" is largely unsolved, and nobody really knows, because we actually don't understand this ourselves.
I'm going to need a citation on this bold claim. And by that I mean in the same vein as what Carl Sagan would say
Extraordinary claims require extraordinary evidenceCan't we create an agent system which can search the internet and choose what data to train itself with?
In fact, that entire notion of early stopping is due to this. We use a validation set as a pseudo test set to inject information into our optimization products without leaking information from the test set (why you shouldn't choose parameters based on test results. That is spoilage. Doesn't matter if it's status quo, it's spoilage)
But we also need to consider that a lack of divergence between train/val does not mean there isn't overfittng. Divergence implies overfittng but the inverse statement is not true. I state this because it's both relevant here and an extremely common mistake.
I think you're being too optimistic, and I'm a pretty optimistic person. Maybe it is because I work in ML, but I've had to explain to a large number of people this concept. This doesn't matter if it is academia or industry. It is true for both management and coworkers. As far as I can tell, people seem very happy to operate under the assumption that benchmark results are strong indicators of real world performance __without__ the need to consider assumptions of your metrics or data. I've even proven this to a team at a trillion dollar company where I showed a model with lower test set performance had more than double the performance on actual customer data. Response was "cool, but we're training a much larger model on more data, so we're going to use that because it is a bit better than yours." My point was that the problem still exists in that bigger model with more data, but that increased params and data do a better job at hiding the underlying (and solvable!) issues.
In other words, in my experience people are happy to be Freeman Dyson in the conversation Calavar linked[0] and very upset to hear Fermi's critique: being able to fit data doesn't mean shit without either a clear model or a rigorous mathematical basis. Much of data science is happy to just curve fit. But why shouldn't they? You advance your career in the same way, by bureaucrats who understand the context of metrics even less.
I've just experienced too many people who cannot distinguish empirical results from causal models. And a lot of people who passionately insist there is no difference.
Also LLMs just straight up do overfit, which makes them function as a database, but a really bad one. So while more parameters might just be better, that feels like a cop-out to the real problem. TBD what scaling issues we hit in the future.
[1] https://www.youtube.com/playlist?list=PLVV0r6CmEsFxKFx-0lsQD...
[2] https://www.youtube.com/watch?v=LvgLyzTEmJk&list=PLVV0r6CmEs...
Lol
> πfs: Never worry about data again!
> πfs is a revolutionary new file system that, instead of wasting space storing your data on your hard drive, stores your data in π! You'll never run out of space again - π holds every file that could possibly exist! They said 100% compression was impossible? You're looking at it!
> Paintadosi [4] argues that one parameter is always enough. He constructed a function that, through a single parameter, can depict any shape. However, in essence, this work is a form of encoding, mapping the shape into a real number with precision extending to hundreds or even thousands of decimal places. For our problem, this is meaningless, although the paper’s theme is that “parameter counting” fails as a measure of model complexity
(Seriously, though, this was a lot of fun!)
Though in that case, I would have liked for them to make it explicit. Maybe normalize it to "1", and scale the other parameters appropriately. (Because as it stands, I don't think you can reproduce their figure from their paper.)
This was a lovely passage from Dyson’s Web of Stories interview, and it struck a chord with me, like it clearly did with the authors too.
It happened when Dyson took the preliminary results of his work on the Pseudoscalar theory of Pions to Fermi and Fermi very quickly dismissed the whole thing. It was a shock to Dyson but freed him from wasting more time on it.
Fermi: When one does a theoretical calculation, either you have a clear physical module in mind or a rigorous mathematical basis. You have neither. How many free parameters did you use for your fitting?
Dyson: 4
Fermi: You know, Johnny Von Neumann always used to say ‘with four parameters I can fit an elephant; and with five I can make him wiggle his trunk’.
If I could make a discovery in my own time without using company resources I would absolutely publish it in the most humorous way possible.
Some lists:
https://academia.stackexchange.com/questions/86346/is-it-ok-...
letmeB this (F you) | you == me = B this
| otherwise = F you
letmeB this (B that) = B that
letmeB this (App fun arg) = letmeB this fun `App` letmeB this arg
[1] https://arxiv.org/abs/hep-ph/9306225
[2] https://en.wikipedia.org/wiki/Alpher%E2%80%93Bethe%E2%80%93G...
You probably have in mind https://en.wikipedia.org/wiki/F._D._C._Willard (coauthor of multiple papers, sole author of at least one).
He tried for a while to get an aerodynamics paper published on the flight of dinosaurs. http://levenspiel.com/wp-content/uploads/2016/02/DinosaurW.p...
This intellectual curiosity reminded me a bit of Feynman and his plate spinning.
A real-parameter (r(theta) = sum(r_k cos(k theta))) Fourier series can only draw a "wiggly circle" figure with one point on each radial ray from the origin.
A compex parameter (z(theta) = sum(e^(z_ theta))) can draw more squiggly figures (epicycles) -- the pen can backtrack as the drawing arm rotates, as each parameter can move a point somewhere on a small circle around the point computed from the previous parameter (and recursively).
Obligatory 3B1B https://m.youtube.com/watch?v=r6sGWTCMz2k
Since a complex parameter is 2 real parameters, we should compare the best 4-cosine curve to the best 2-complex-exponential curve.
Maybe this sort of thing would be a really good tradition. Everyone must write a very silly article with some mathematical arguments in it. Then, we can all go forward with the comfort of knowing that we aren’t really at risk of breaking new grounds in appearing unserious.
It is well written and very understandable!
Why would that be a harder problem? In the case that you get a zero parameter, you could inflate it by some epsilon and the solution would basically be the same.
In a sense, they have done their fitting using nine parameters, of which five are zero.
Not everything is continuous. Add an epsilon worth of torsion to GR and you don't get almost-GR, you get a qualitatively different theory in which potentially arbitrarily large violations of the equivalence principle are possible.
yes, a fun problem, but also a criticism of using to many parameters.