There’s lots of work on distillation, smaller models, approximations, etc. People already have simpler forms of these running on smartphones. Models seem to be growing faster than we can make them small though :D
367 karma · joined October 18, 2007
This one perhaps
https://arxiv.org/abs/1811.00207 - Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset
https://arxiv.org/abs/1811.01241 - Wizard of Wikipedia: Knowledge-Powered Conversational agents
https://arxiv.org/abs/1810.10665 - Engaging Image Captioning Via Personality
https://arxiv.org/abs/1902.08654 - What makes a good conversation? How controllable attributes affect human judgments
And a demo of a bot that they've produced: https://www.facebook.com/Beat-The-Bot-212188996195556/
"$50,000 was divided among the human participants based on their performance to incentivize them to play their best. Each player was guaranteed a minimum of $0.40 per hand for participating, but this could increase to as much as $1.60 per hand based on performance."
So the humans weren't betting their own money, but they still made more money if they won.
You can be employed in a high paying sector like tech after going to an in-state engineering school, and still l end up with $2k tacked on in your first year of employment. So then what is more rational: pay off the loan as fast as possible, or build up an emergency fund with a generous 2% ROI.
With the cost of education, healthcare, and housing all rising significantly faster than inflation, it's possibly just harder for many to save up that emergency fund. Indeed, it's particularly difficult to justify having an emergency fund if you're also facing five figure 8% interest student loans.
You also don’t just launch that many things and them ignore it. You monitor it to make sure nothing is going terribly wrong.
But yeah there’s also the fact that if you’re Google, throwing $2m worth of compute at something becomes worth it for some reason (eg Starcraft)
By that point you know it’s going to work, it’s just a matter of how well and whether you could’ve done nominally better with different tuning.
There’s been enough research leading up to this paper to suspect that just scaling larger would play out.
For example, polymorphism could be decomposed into poly-morph-ism. Antidisestablishmentarianism, which is unlikely to appear much in the corpus, becomes anti-dis-establish-ment-arian-ism. Now the system can learn how to reuse "anti-" or "establish" from other examples more easily than trying to learn the full word's meaning from the one or two examples it might see in the corpus.
BPE is a clever way to induce these sort of decompositions automatically without any linguistic annotation, making them useful in multilingual settings. Other languages are much more morphologically rich than English, and there it really benefits.
Deep learning is very power hungry. Pretending you can do it on a battery is a fool’s game.