Machine Learning Can't Handle Long-Term Time-Series Data
lesswrong.com
lesswrong.com
ML operates with associative models of billions of parameters: trying to learn thermodynamics by parameterizing for every molecule in a billion images of them.
Animals operate with causal models of a very small number of parameters: these models richly describe how an intervention on one variable causes another to change. These models cannot be inferred from association (hence the last 500 years of science).
They require direct causal intervention in the environment to see how it changes (ie., real learning). And a rich background of historical learning to interpret new observation. You need to have lived a human life to guess what a pedestrian is going to do.
If you overcome the relevant computational infinities to learn "strategy" you will still only do so in the narrow horizon of a highly regulated game where causation has been eliminated by construction (ie., the space of all possible moves over the total horizon of the game can be known in an instant).
The state of all possible (past, current, future) configurations of a physical system cannot be computed -- it's an infinity computational statistics will never bridge.
The solution to self-driving cars will be to try and gamify the roads: robotize people so that machines can understand them. This is already happening on the internet: our behaviour made more machine-like so it can be predicted. I'm sceptical real-world behaviour can be so-constrained.
Deep learning is very useful, but only one piece of the whole AGI puzzle.
Furthermore, many AI problems will benefit the generality of being formulated as a probabilistic program synthesis problem [3]. In this framework, lots of program semantics (~formal methods) concepts like abstract interpretation [4,5] might become very useful. They allow to explore huge program spaces very quickly.
Lastly, Pearl's do calculus [6] is a good starting point closely related to [1].
[2] http://pyro.ai/
[3] https://web.mit.edu/cocosci/Papers/Science-2015-Lake-1332-8....
[4] http://www.concrete-semantics.org/concrete-semantics.pdf
[5] http://adam.chlipala.net/frap/frap_book.pdf
[6] http://bayes.cs.ucla.edu/BOOK-2K/causality2-epilogue.pdf
On one side we have the colloquial understanding of cause and effect where a cause is a true impetus of effect. On the other side we have "causal" learning in biology where you're not actually learning causes, just strong correlations. We can learn just about any temporal association even if there is no direct cause-effect relationship. Random reward structures are a way to illustrate this: present a reinforcing stimulus to an animal at random times and a random subset of behavior will increase in frequency. The animal develops a false "causal" belief that a series of its actions is influencing the presentation of a reward.
That's why I like focusing on "sequence prediction", even colloquially we know predictions can be wrong. Those predictions can be influenced by low-d world models, but you don't accidentally elevate that model to claim a pure/symbolic/accurate model as can happen with incautious use of the words like "causal."
Superstition in pigeons arises because they believe their actions cause the reward, it isnt "mere sequence". Any distribution over two variables observed overtime, for all time can change unpredictably given an environmental change.
Animals have rich models of objects and their behaviour over time, these models aren't "sequential", and they are brought to bare on deciding whether mere sequences should be regarded causally.
This strikes me as patently false, which means I'm probably not understanding what you mean. What does forward simulation mean if it isn't sequential?
People give lip service to "correlation is not causation" by say, well, we're not going to call our correlations causation, we're just going to use them like that. There, are your delicate sensibilities satisfied?
No! Because if you have the wrong causal relationships whatever you call them used to predict stuff, the predictions will be wrong! Totally, not 1% off.
* Tegmark and Wu's AI Physicist: https://news.ycombinator.com/item?id=18381827
This would reduce a nearly impossible job in machine intelligence to a really difficult simulation problem.
But I don't see that happening anytime soon.
Now if we could reuse cell tower signals, the telemetry system is already in place. Cell phone operators keep the location of their cell towers secret, but with some persuasion ($$$) would reveal their coordinates.
Given the existence of spoofed base stations (stingrays) among other reasons, a PKI-based solution may not be sufficiently safe. So you‘d have to overlay beacon data with sensors, at which point it’s questionable if there‘s a significant added benefit.
The author has extremely grand set of connections he developed. It ties down Buddha, enlightenment, vipassana meditation, artificial intelligence, cybernetics, fractals and neuroscience. Nothing wrong with that, of course.
Creative thinker should have these kind of crazy ideas and connections every day or at least once a week. I carry with me a notebook that is full of them.
Most ideas die as 'premature babies'. They may be interesting to think and write down, but they are not fully developed and never fit together as well as you initially thought. Filtering and piking some of them to work with is important. Giving them up is the difference between crackpot and non-crackpot.
Forcing grand connections prematurely makes this crackpottery type. Sharing the creative brainstorm in an essay that does not try make up connections would be easier to read.
I think you're being too kind here. It's just crackpottery as far as I can tell. I agree with you though that it is the type of dumb idea that should die in a private notebook.
Less thought-leadering, more actual results please.
The questions this article relates to are among the most profound and difficult that human reason has ever attempted to confront. I think one should be careful in labeling such ambitious speculation as crackpottery just because it doesn't yet amount to a fully coherent and formally testable theory.
I'll also go further and say that, while there's a kernel of validity to your analogy, it's not the right analogy with which to deliver your overarching point. I don't think the publishing method for one of the most significant scientific advancements of the previous century is a particularly good lens for analyzing this blog post.
The critical content of this post is far below the threshold usually associated with an idea sufficiently well formed to be publishable. Einstein had a minimum viable theory before he solicited feedback; and when he did solicit that feedback, it was through what we'd consider orthodox channels.
The main point I was trying to make was that given a speculative post of such breadth, which touches on such difficult issues as AGI, how the brain works and perhaps even the nature of conscious experience, and which makes some claims that are at least interesting, I think it's quite presumptuous to assert that these ideas are all nonsense without a deeper exploration of them. I certainly would not want to make such an assertion, despite being troubled by what I think are some inaccuracies in the author's description of certain physical concepts.
Now a secondary issue is that it is true that as far as I'm aware major scientific discoveries have typically been initially published in much more developed form and have thus been the work of a single individual or of a relatively small group of closely affiliated individuals. I'm not convinced however that this historical model of very small scale scientific collaboration is necessarily the only one nor the best one in light of modern means of communication.
It seems at least conceivable to me that there is a possible future in which the following hold:
* There is some kernel of validity in this author's ideas.
* A small number of other people find them intriguing and choose to collaborate with the author to further elaborate them.
* This collaboration leads to major progress in our understanding of one or more of the areas mentioned above.
For me the, admittedly very small, likelihood of such an outcome, justifies the author's post and its appearance on HN.
That "certain patent clerk" was hardly working in a vacuum; not only was he building upon the work of others (e.g. Lorentz just to pick someone) but he had been in school with some of them and was in constant communication with them, as he was hardly the only one working on the problem.
This is not to minimize his brilliance (Special Relativity in particular has that wonderful property of being completely obvious once explained...but that you could almost see before the explanation but yet nobody had previously characterized) and the mind blowing nature of his three-paper year. But when published his work fell on fertile ground.
I'm not sure how fertile that ground was. It was still 3 years after the year he basically laid the foundations of modern physics before he was able to get an academic job.
Posts like these usually want to create a strawman narrow definition of what is allowed to count as “machine learning” and the work backwards to say that subset of models can’t handle some type of problem.
Machine learning is just statistical modeling. To the extent that any kind of statistical modeling adequately solves long term time series inference goals, then so does machine learning.
I moved forward with 5 hours days / 5 days a week while keeping a full time job.
Today I am sitting on a serious "wealth machine".
We have to accept that 98% of human cannot "think outside the box"
It calls out long short term memory but doesn't mention recent (last 5 years) improvements like Gated Recurrent Networks (GRUs) or Transformers (GPT-2, huggingface/transformers) which have shown significant improvements over the traditional LSTM model. These can handle time series data much better than older models could.
I don't doubt that the things you mention involve improvements but are these improvements doing better on the same benchmarks in the same fashion or a fundamental change. I read many claims that recent changes in deep learning represent the former.
Here are some good guides on Transformers [1] and attention/ multi headed attention [2], as well as the paper that proposed the transformer model "Attention is all you need" [3]. GPT-2 heavily relies upon the advancements that transformers brought [4]
[1] http://jalammar.github.io/illustrated-transformer/
[2] https://towardsdatascience.com/attention-for-time-series-cla...
[3] Attention is all you need. https://arxiv.org/abs/1706.03762
So, but that "point" could just be that being a published academic means always using the latest thing. That doesn't demonstrate how much better these are, much less demonstrate that they represent fundamental steps forward.
From what I've seen, It breaks down at the embedding layer, because while the teams "remain the same" in name / dictionary, their actual relative relationship to each other varies season to season / week to week.
But wouldn't capturing that relationship require periodic retraining or other components to the network regardless? It may suggest that end-to-end training of a transformer is not suitable for these tasks, but that it might still capture the prediction of the long-scale time-series, if provided with extra data at each timestep in addition to the embeddings?
Sports are particularly problematic because almost all teams and statistics regress to the average at some point, meaning your generated future timestep context clues don't really help modify the embedding.
You're also dealing with variation within a season (injuries, better play, etc) and between seasons (personnel changes, rule changes, new stadiums, etc). So a team might have 4 seasons of above average performance, and then abruptly be the worst team in the league the next because they lost their coaches and star players.
Synchronously parsing the meaning a single text has at one moment in time involves no time series at all.
They're not even using transformers, just simple word embeddings.
Being able to follow multiple agents and correctly deduce their relationships at a given time t is very hard.
NLP "time-series" does a fine job at making back references within a text, but wouldn't be able to have multiple representations of a word or character through the years.
It's very hard to get the computer to say "ah, the context is 16th century, so here are the relationships" without fudging it / tailoring models via tailored corpuses.
The detection kept changing, and so the model kept going "oh, new object, restart decision process."
Lacking the ability to generate and maintain it's own context is an area where a human would do better. We might not know what the object was, but our "slow down" response wouldn't keep resetting depending on what we classified the object as.
Same as words switching meanings within a piece or sentence. It's hard, but most humans can pickup when the usage changes
The reason seems even simpler than the article. Deep learning requires lots of training data - that data naturally needs to more or less be "the same"; follow "the same" logic.
A long enough time series is going to involve a change in the logic of the real world, a change that the network won't be trained for.
It's a semantic difference, but a very important one if we're to avoid going down the road of just mindlessly throwing compute at every problem. And if we do that, we'll just wind up with millions of Rube Goldberg machines instead of actually solving problems.
The change in logic of the real world thing is absolutely spot on, though. Over enough time it becomes basically impossible to disentangle effects.
References? I mean, I know "one shot" and similar approaches but as far as I know, these involve extending a neural network that has been already trained, on massive data, to a little bit more.
Uber's engineers didn't do this for their human finder because;
1) Image recognition stuff isn't explicitly built to do this (though it easily could be jury rigged to do so)
2) Uber's engineers apparently never heard of the concept of "moving averages" and "threshholds" which would have worked just fine.
"More precisely, today's machine learning (ML) systems cannot infer a fractal structure from time series data."
-look at this idiot using words he doesn't understand. Muh fractals.
As I said above; machine learning does just fine to extremely well at long term time series data; there are entire branches of machine learning dedicated to this.