1,038 karma · joined June 8, 2016
In this paper, Bellard makes a comparison to the Transformer, some variant of which holds most NLP records. Of note is that in this test, the Transformer did not outperform the LSTM. Is it that Transformers need to be larger to outperform or LSTMs are more suited to compression or that more epochs are required? This deserves further investigation, what are the required conditions for Transformers to underperform LSTMs?
Because we understand how these algorithms work, we can always reduce them to just this or that. Prediction in many linear classifiers is just dot products. relu neural networks are just lots of clamped dot products. Random projections on simple count data can generate word embeddings.
Whether or not something is merely model-fitting, super-scaling, compression or AI, AI art and machine learning will depend on the field it originated from. It's indisputable that the algorithms are so reducible but I tend to think that we should care more about functional capabilities when compared to an appropriate subset of a known intelligence's abilities than details of implementation.
You're right, I noted too that the comparison isn't direct but then, I wasn't justified in calling out the gap claim as wrong, so sorry for that. I think it'd be nice however, to have it undergo an external or more neutral test of performance. I say this without at all doubting the quality of the results.
A human will also be learning vision, hearing, walking, physics, causal reasoning, and much more. This comparison just isn't well grounded. Task specific is how much training does a young brain require to learn to produce language? If the brain comes with innate advantages then rather than be resort to inefficiency and excusing our models, we should try to see if they can be bettered.
Rather than under-fitting, I'd consider this diminishing returns in the model's ability to make use of the available information. You can get within 10% of it on the challenging winograd schema task using only 2%-5% of the used data. Many here are programmers and computer scientists, rather than stoop to hyperbole we should try to understand. What class of automata can this model learn? Finite state machines, push down automata? Transformer has memory but no loops so I very strongly doubt them to be more than that and yet they perform so well on language (but then again, not simple program learning) tasks.
I also notice reported tries, suggesting some level of curation. While this level of generation is undoubtedly impressive and a sign of non-trivial levels of understanding, the ability to project along arbitrary dimensions of similarity at a fine-grained level and to learn from text instruction is more useful than text generation. Although the unicorn story was a really fun read, better than many humans already, I doubt it could have gone on for much longer. It maintains a theme but not coherently or fluently (see especially the Kennedy nanotech and recycling examples, comparing the dis-fluency there versus the excellence of the Civil War report suggest at least some over-fitting). These relatively minor caveats aside, this is unambiguously an outstanding result.
Winograd Schemas are the single metric to track if interested in understanding how language understanding is truly improving. OpenAI reports 71% and wrongly report the previous record as 63%. The current record here is at 65% https://gluebenchmark.com/leaderboard though not fully comparable. Will OpenAI be submitting? Note that you can get to 60% using about 1-2 orders of magnitude less data and compute.
It concerns me that results here are so far dependent on such large data and computation. However, based on several papers I've read, I do not believe this to be inherent even in transformers. I plan to do some experiments on this when I free up some bandwidth.
If everyone is pulled in by the glamour of working for a well funded, prestigious operation then it should be no surprise that they do not consider paths which operate on several orders of magnitude less data and computational resources.
We all should consider bringing about a group of researchers who swear to an austere computational life of a single GPU, no more than 4-8x average RAM and CPUs that do not cross 90 Watts. The Bicameral Order would be a good name for such a group.
Another view point is to note that John Holland, inventor of GAs, was more interested in the application of GAs in classifier systems than as objects of study in and of themselves. His work on the bucket brigade algorithm, a type of TD-learning, for credit assignment in complex reinforcement learning scenarios was first-rate and sadly still under-attended. In that setting, GAs were a search operator, focused on exploration while bucket brigade was for credit assignment. While GAs can be shown to be capable of doing adaptation as well as exploration, they really were meant to be part of a bigger whole by their inventor.
In fact, Deepmind's AlphaStar for Starcraft problem formulation can be seen as fitting into the learning classifier system framework where learners are themselves quite powerful neural networks. See: https://deepblue.lib.umich.edu/bitstream/handle/2027.42/2777...
He also did early work in cryptanalysis, AI (minimax chess algorithm and a learning robotic mouse). His student was Ivan Sutherland. More or less, whether its networking, signal processing, compression, crypto, machine learning, circuit design, basically anything to do with the digital age, you'll find Shannon did important foundational work there.
I'll say however, the benefit of an ML like Ocaml is less so the types than that functional languages are the most advanced in providing tools which allow for domain modeling in terms of composition and defining tailored algebras. This is something that fits both applied and experimental machine learning especially well.
If you look at the issues, you'll see the developer contemplating GADTs to deal with tensor shape matching (but even phantom types will do). This is an issue that causes headaches for just about anyone whose dealt with neural nets.
I've also noticed that a lot of the time, people are comparing GPU to naive implementations (and sometimes, implementations written in dynamic languages) instead of to highly tuned BLAS or MKL implementations. For a large swathe of problem types, using the fastest math library will reduce the CPU/GPU gap to less than an order of magnitude or less.
A parallel discussion we are having is whether the gain in accuracy is always worth the gain in complexity and loss in speed. It's something to decide on a case by case basis. It's basic hygiene to reach for the simplest model first.
This isn't true. It depends on your priorities and goals. Machine learning that spends most of its time unable to learn is not real AI. Some of us are interested in sample and energy efficient learning capable of on-line incremental updates immune to catastrophic forgetting. Not just because this is truer to actual learning but because it moves away from being dependent on a handful of companies to do the actual training.
Anticipating some replies: no, transfer learning or meta-learning methods don't really avoid this. In the case of transfer learning, you still have that high coupling between a handful of sources. The down-sides of this is its own discussion. In addition, there are times where the ability to extract local relations can be dulled by the dominant wikipedia and common-crawl representations. Meta-learning gets you fast updates but you still cannot stray too far away from the domains that were met at training time.
> What matters is prediction speeds
I'm not a fan of bag of words models either but a simple dot product is always going to be faster than many matrix multiplies and or convolutions. The implementor should always try these as a base-line and decide if the performance accuracy trade-off is worth it for them.
This algorithm is actually an instance of constraint propagation: https://en.wikipedia.org/wiki/Local_consistency and constraint satisfaction algorithms generally, that find use in say, computing graph homomorphisms.
The document also answers some of my questions. Some confusion from the fact that they are using multiple methods in parallel. The scoring networks are resnets. The database was used in training one of the networks.
The generative network section is still unclear, but it looks like it was used in one of their methods, fragment assembly step and network architecture is DRAW (trained with DB). Network generated Fragments inserted using simulated annealing, ranked by either just their conv-net or both existing methods and conv-net. Simulated Annealing hyperparameters were optimized with evolutionary search.
Their third approach is what is most similar to what you describe. It looks like they combine lots of approaches, including features, convnet scores and predictions from other methods. Still needs a lot of detail to be filled in such as more detail on how they integrate memory and how gradient descent is actually done directly on protein chain structures.
It's possible their computational advantage made a large difference in being able to arrive at this result; existing researchers need not feel too bad.
1) What is the architecture of the generative network and where exactly does it fit in the pipeline?
2) What is the interaction with the database? Is there an encoder being trained with real sequences further augmented with variations using the generative network?
3) What is the structure of the neural network that encodes the sequence? Is it a graph network, LSTM or simple conv-net?
4) The gradient descent step is very vague. Is it a physically based differentiable model (not a neural network) whose parameters are being optimized with gradient descent using automatic differentiation? Or something else? In short, there's some detail on scoring but how are the proposals being generated?
Questions aside, the results speak for themselves and are head and shoulders above all other showings. I wonder what it feels like for someone whose been in the field for years.
Despite the high score, there's still a long way to go before results reach real world utility. It's also worth keeping in mind that from a systems biology perspective, protein folding is only a small part of what makes getting clinically useful results difficult.
I might have missed something but I could not find anywhere indications of an intention to publish further details. That would be disappointing if such were indeed the case.
Near the 48th second of this video: https://www.youtube.com/watch?v=exsrX6qsKkA, you can see the bumble bee actually hesitating to work out which option will yield a reward for the least amount of effort. That kind of laziness is an excellent display of flexibility and intelligence.
It's also interesting that you mention colony collapse. If as is likely, neonicotinoids play a significant role in colony collapse then the bee's sensitivity to disruptions in acetlycholine pathways is an argument for their relying heavily on cognition for foraging and navigation. But yeah, I'm not saying bees are just as intelligent as primates, corvids or parrots; just that they too are capable of quite unique sophisticated behaviors and learning.
Within the animal kingdom, the sophistication of the honey bee's waggle dance, with predicates and combinable to a large number of phrases, is rare. The only other major example I can think of are primate calls.
Bees are also highly intelligent; on a watt for watt, pound for bound basis, their only other rival might be the portia spider. Bees can communicate angles, distance and adjust for the movement of the sun when relaying these. They can learn abstract patterns and generalize. If you asked me which was more intelligent, the honey bee or the chicken, I'd be hard pressed to give an answer.
https://en.wikipedia.org/wiki/Bee_learning_and_communication
Just as outstanding is that, in all these years, in terms of open-ended problem solving, nothing has truly managed to surpass it.
The conjunction of those two things is a bit scary because it suggests to me that current methods are probably not making the most effective use of compute. What happens when that is corrected?
The second is compactness is true only if you're chaining straightforward library calls. If you're writing your own functions or building non-trivial functionality on top, types will be of aid too.
A 815 mm^2, 250 W GPU will be 250 W / 8.15 cm^2 ≈ 31 W / cm^2.
When I do an exact search for the Justification Sentence with Google, what best matches is a quote by Rajiv Gandhi. The relevant context is: "History is full of such prejudices paraded as iron laws"
His stance is clearly opposite to what the extracted text implies. This is a common problem with knowledge extraction and one I've run into often myself.
Extracting just a phrase, or utterances of a generative model cannot be trusted because the original meaning can be opposite to what is presented. Existing models fail to preserve nuance imparted by context, struggle with negation, lack deep understanding and an ability to truly reason.