Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

> it’s ridiculously parameter inefficient

How do you know that? Perhaps with this methodology you really need those 3.12TB to reach comparable performance?

Mixture of experts architectures like this are a specific design decision to increase the parameter count but use and update those parameters sparsely. Sure, that design decision doesn't fit all scenarios, but it fits some, and it has its own advantages, like faster training time.

Google's legal team would be freaking out if any of its engineers propose open sourcing model weights. This is a mine field full of legal and privacy headaches and I guess it is infeasible to audit 1.6T parameters anyway...

Some other thoughts on what makes open source machine learning, from Debian:

https://salsa.debian.org/deeplearning-team/ml-policy

One key idea here is to use a very large number of parameters (model weights), but only use some subset of the parameters on each example. The parameters are divided up into blocks called "experts", and then some subset of experts are used on any given input. Which subset is used is chosen by the model itself in a data-dependent manner. This can be thought of as letting the model specialize different experts to handle different situations.

The advantage, as they show, is that the model can train to a given level of performance much faster with a fixed amount of computing power compared to an architecture that uses all parameters on every step. This might be because it allows you to have a very large number of parameters that can store a lot more specialized information without incurring as much of a computational cost. Of course the downside is that you end up with a very large model that literally won't fit in a lot of environments.

hah! yeah that's what I was looking for too

data is the new oil, what's the analogy for the data industry's impact on society akin climate change?

In the source code it says "I have discovered truly marvelous weights for this, which this header file is too small to contain"

If I did the math right it would be 3.12TB of weights, maybe they are trying to upload it to gdrive still. (/s, probably)

Is this because they are afraid of the model misused, like used for generating fake reviews? It is frustrating that I've been hearing great news on NLP but am able to try none of them myself.

Slightly apples to oranges comparison.

The brain has an incredibly complex architecture, which evolved over millions of years. On top of that, it then develops throughout a human's lifespan. The brain we observe is a "finished product", and even then it has ~150 trillion synapses to do computations [0].

Even massive neural networks have a relatively simple architecture before they are trained. Part of the training process is effectively learning more complex architectures, which are manifested by changing weights.

What I'm getting it is that artificial neural networks aren't equivalent to the brain - ANNs are both learning their own structure, on top of the circuits actually doing computations. They are doing the work of millions of years of evolution, genetics, developmental biology, interaction with the environment etc. Perhaps it's to be expected that ANNs will need orders of magnitude greater number of parameters than a brain.

An interesting development is meta-learning, where we separate the process for learning the architecture (this could be using deep learning, but not necessarily) with the network actually doing computation (equivalent to the brain).

> A language model must be based on the fundamental notion that there are nouns (things), verbs (processes) and adjectives (attributes).

I agree, but how does the brain represent these concepts? Some would argue that ANNs do have these concepts, just hidden away in abstract vector representations. Take the visual system, which has been extensively studied - we see the brain represents contrast, edges, shapes and so on very similarly to convolutional NNs.

[0] It's likely that this number doesn't come close to capturing the brain's complexity, as it doesn't incorporate parameters like long-term potentiation/depression, synchronization, firing rates, habituation vs sensitization, immunomodulation and likely so much more we haven't yet discovered.

If the brain is the answer, the question involves trillions of parameters. Clearly there's more to the brain than just size, but also clearly, the brain is big for a reason. In fact scale is one of the very few things we can say for certain plays a big role in the function of the brain. GOFAI notions of embedding grammars and knowledge webs are just guesses on faith—the evidence points precisely in the opposite direction—and don't really make much sense anyway.

Even so, memorisation implies word by word information retrieval and interpolation, it's not a hash table.

You could make something like an "expert system" with it. But since it doesn't seem to link back to the source, you'd never know if it was giving you a real answer or just making something up.

Google already uses a combination of NLP + PageRank to serve search queries [1].

[1] https://en.wikipedia.org/wiki/RankBrain

Maybe host and use it in a non GDPR environment, it's always going to be ambiguous and risky.

Yes, it has been shown [1] that these models can memorize information even if it only appears once in the training data. It could potentially cause issues with privacy and especially copyright. However, in this case it's not relevant because they didn't release the model weights.

[1] https://arxiv.org/abs/2012.07805

I feel like a future version of GPT will easily be able to answer 'What is the home address of ...@gmail.com', given the abundance of data it is trained on and the widespread proliferation of leaked databases.