Fourier analysis may help to quickly train more accurate neural networks
spectrum.ieee.org
spectrum.ieee.org
In a lot of cases, yes. You can start with a reasonable baseline architectural guess, like “convolution should be good for vision” or “attention should be good for language”, but after that it's a lot of guess and check.
I consider it a tragedy that we throw huge hogs of models into datacenters and let them churn without giving much thought to improving performance. The climate impacts relative to the real benefits are measurable and depressing for a field so focused on innovation. Usually the 100x model built by a team of "data scientists" over a month is no better than a 1x model built by a couple SMEs over a couple weeks.
There are plenty of times when deep learning is the wrong solution, and probably most businesses out there should not ever expect to need any kind of highly compute-intensive model training. But there are also plenty cases where neural networks are the right solution as well (even if not "deep learning" as such).
Maybe I'll be convinced if you share some specific examples. Otherwise I'll assume that your comment is just arrogant derision.
> sneer at the cutting edge of ML research
I was reacting specifically to mis-specifications of what "good" means in these contexts, e.g., "attention should be good for language" from GGP comment.
The major advances of these models du jour are not the outputs, but rather the infrastructure that it takes to run them. I salute all those involved in implementation. I hope (and see!) these innovations spill into things that help people and generate real value, for instance, all the things businesses use accurate climate forecasts for.
My issue mostly revolves around the discrepancy between value-created-per-FLOPS and cost-per-FLOPS. If we run huge models for what serves for us the same purpose as the TV walls in Fahrenheit 451 do for their owners, what are we adding? If we are constantly running pixel-level segmentation networks on the cameras of automobiles that are already being driven by humans, how is that helping?
Stable Diffusion is a real innovation in optimization techniques. This is the only part of recent news that can be considered "cutting edge ML research", as it is an important step in unifying optimization theory with that of non-equilibrium statistical mechanics. The parallels have been noticed there for decades, and the cutting edge of ML research today is unifying them by cribbing aspects of each into the other.
The ideas (models) themselves are quite naive. The perceptual space of these models is essentially one-dimensional. This, in the broad strokes, is a terrible model for reality, and so these casual discussions of "intelligence" seem mis-informed, not to mention under-determined. It seems like a lot of mis-informed people speak quite confidently about the things they don't know that they don't know, especially on a site where people who value their autodidactive skills tend to congregate.
Inductive bias is the primary term of art that is missing from these discussions. The models du jour do not have any architectural components which represent language per se, only flat representations thereof. The reason LLMs work so well is because of the Transformers inside; multi-head attention is a clever trick to vastly expand the possible data processing techniques as information passes and is distilled through the network into the outputs, but ultimately it is just a fancy way of doing a technique we could call "quasi-random projections" under the domain of linear algebra.
The notion of intelligence we entertain in these discussions is hopelessly anthropocentric, and, furthermore, incredibly arrogant. The recent achievements say much more about how easy it is to simulate human-comprehensible language and images than it does about how we are birthing new intelligences. I believe the hype and deployment of these tools is doing more to hurt society in the long term than it is to help it, and I welcome more candid discussions about how simple the rules that make us "intelligent" might be. That seems to me, during my studies, to be the real question.
The biggest innovation of deep learning is that you can take a relatively general architecture and build surprisingly sophisticated models by training that general architecture on a ton of data, that absolutely wipe the floor with the older style of models.
AlphaGo/Zero is maybe an exception here: it's not "just" a model, it's a whole system based around a model, which is specifically designed to perform well on certain tasks, and succeeds where a more naive system would probably fail. You could argue that ChatGPT itself is an early kind of advancement along those lines, going beyond just cranking predictions from a model to a more holistic task-specific system. But it's still mostly "just" a huge model trained on a huge pile of data, and I'm not going to argue strongly that it isn't. What I am arguing is that people aren't as naive as you think, and it's not for lack of trying that LLMs don't incorporate detailed knowledge about linguistics, psychology, et al.
You suggest that transformers shouldn't be considered "good" for text, because they don't include enough inductive bias. Don't they? They're really clever things in my opinion, and I think they actually represent quite a lot of inductive bias compared to what came before, and have a lot of intuitive appeal in their respective intended task domains. I hardly think it's fair or correct to consider the transformer nothing more than a linear algebra trick.
I suppose you're arguing that they aren't "good" solely because they produce state-of-the-art results and beat literally everything else we have. I think your argument is that they only work as well as they do because they need a tremendous amount of computing power and data to get useful results out of them. That might be true, but that's literally why people are trying to make them faster and less costly to train!
Don't forget that the other big innovation of deep learning is transfer learning and fine tuning. A big LLM like GPT-3 only needs to be trained fully every once in a while, while it can be reused and adapted to a huge variety of tasks relatively efficiently and cheaply. The whole point of the article at the top of this thread is making big models cheaper and faster to train. If you're bothered by the cost- and compute-efficiency of these things, what could be better than making the training process more efficient?
Moreover, it's kind of weird to insinuate that the field doesn't care about making these things faster or cheaper to run, when immediately after any new model is announced, a flood of projects follows trying to shrink the model, improve inference speed, etc.
Finally, who said anything about intelligence? I didn't, and nobody else did either. That's a completely unrelated topic as far as I'm concerned, and I leave that one to the philosophers to debate.
Not unlike how you can linear regress on x1,x1^2,x1^3,x2,x2^2. Encoding prior knowledge about input data as additional inputs helps with machine learning. 1980 called it wants its accomplishments back.
But in that case it's not quite Fourier analysis because those functions aren't applied to the input; they are just constants that are added to the input because transformers are permutation invariant (without some positional encoding).
Was there a model that applied sin and cos to the input?
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, https://arxiv.org/abs/2006.10739
The world was all model-based analysis, and then neural networks came along. Model-based analysis-which requires a thorough understanding of the model mathematics-went out the window. Now, we let the network develop its own model, but it’s based on the data that is used to train it. But what if that data is biased? What if the data doesn’t contain some necessary “truth?”
I feel that I t’s like driving a car looking out of the rear window to see where you have been. This can’t be good.
However, there are many, many useful real world tasks where we haven't been able to do that (e.g. almost all of image analysis, image generation, a significant part of of text analysis and generation tasks), so for these tasks the (valid) criticism doesn't really matter, as we don't have any reasonable alternatives - if let the network develop its own model, we get a somewhat working solution (with all the risks you mention), and if we don't, then we best we can make is unusably bad.
If all the windows in your car are painted opaque black except the rear window, then driving the car by looking out of the rear window to see where you have been can be as good as it gets.
The most successful applications of ML in my own career have always been judiciously building small task-specific models that fit inside a broader system. And there is also usually quite a lot of consideration being given to the model input data as well.
Similarly, I have never seen a "just throw a model at it" project succeed in a finite amount of time.
The ML approach is a machine generating a function from test input data to test output data.
Both approaches will be compromised by biased data.
You could argue that you don't really need the data in the classic approach, if the human knows what he's doing. But that's kind of like writing code without writing tests. I'm not arguing for or against tests here, but perhaps you'd just be shifting the source of the bias from the data to the human doing the coding?
> What if the data doesn’t contain some necessary “truth?”
This is my gut reaction to a lot of data-driven-crap. "We increased our customer acquisition by-" No you didn't. "Customers are 15% more likely to-". Nope.
> I feel that it's like driving a car looking out of the rear window
I don't think this an apt metaphor here for ML in general, but I think it could apply to other HNer's utopian vision of ML, e.g. once GPT is "good enough", we'll be able to throw away the real world data and use GPT to train more models.
[0] https://academic.oup.com/pnasnexus/advance-article/doi/10.10...
Edit: I missed the quip about the Kalman comment being in jest, my reply probably wasn’t necessary but I’ll leave it up.
1. Knowing when to use the right tool, and
2. Being in the right place at the right time to offer the insight.
What’s obvious to the expert is frequently intractable to the outsider. In many cases, I wouldn’t call it so much “avoidance” as “not knowing the solution exists at all”.
99% of the time it seems they're not wrong, despite control engineering being a pretty large area of research.
http://www.ecircuitcenter.com/Circuits/op_pid/op_pid.htm
https://www.nutsvolts.com/magazine/article/the_perceptron_ci...
The next level up is obvious
maybe some s-plane control theory if you’re feeling fancy.