20B-parameter Alexa model sets new marks in few-shot learning

Also why doesn’t Siri work at all in Honda Civics and Honda HR-V’s when connected to car play and driving on the highway with no radio playing?

Google assistant works fine, for the most part anyways.

These are all subtly different problems I think, but in general most of these architectures currently assume there is a single intent.

> stop music and turn off all the lights

This is probably the easiest of the bunch because you are asking it to perform two distinct actions.

> turn off the bedroom light in 5 minutes

This is much more complex, because you are asking the application to setup some sort of workflow - after it understands what you want it to do, it then has to work out how to execute that, which will be utilising the device API's / services. This is a simple example, but there are lots of permutations of different actions here, for example you might want to say "turn off the sound system once this song finishes playing" which assumes that the assistant has the capability to then understand you want it to create a task specifically waiting for the trigger of a particular song finishing playing, and that it has the ability to setup that trigger.

> "keep the lights on” in a motion sensor system

Now this is where the orchestration gets tricky -

The assistant has to:

* Work out that the lights are being affected by a motion sensor system, which is likely outside it's own platform.

* Work out that your intent is that you want the assistant to override that.

* Understand how to connect to the platform in order to control it.

* Work out what parameter it is supposed to alter to achieve this task.

* Override the existing users settings, and presumably reinstate the settings after some portion of time.

My guess is that it's a very different (and difficult) problem to generalize that way. Interpreting intent and taking action are different aspects. Someone needs to write code to call a vendor's API to execute those actions and that's a super specialized action. Next step is probably instruct a CoPilot-like tool to do it.

I have google home and the first one worked. The 2nd did not but I don't think I'd ever say that bc I just have routines where I say "i'm leaving" or "i'm going to bed" and everything shuts off at once.

It’s sad that the only option is a voice assistant that must learn how to interpret my words through this slow, error-prone process. I would much rather have a pure speech-to-text option where I must learn the exact words to say to get a reliable result.

My benchmark for Siri will be when it learns to do "Siri, wake me up an hour earlier tomorrow". What it currently does it set a new alarm for 1 hour in the future.

> turn off the bedroom light in 5 minutes

This actually works already with Siri (and as mentioned in a sibling comment with google Home as well). I just tried that for fun a few days ago and was surprised that it actually worked.

>turn off the bedroom light in 5 minutes

This works for me, except when it mishears what I'm saying as "turn off the light for 5 minutes", which gets a bit annoying

> At what point can we say things like “turn off the bedroom light in 5 minutes”

I often get ready to leave the house and say "start the robovac in five minutes" and it just works. The Google home gadget confirms and states that it will start robovac "at $time-of-day". I've not got too many things connected but I assumed that it's generic among those things that can be turned on/off.

Anecdotal story, if you have a group of lights, you can't just say to alexa "turn lights off", she will keep asking which one you want. I ended up hardcoding "turn lights off" as a routine trigger.

You had one job, Amazon!

I swear there was a golden age in 2016 where voice assistants were magical and worked very well, and then around 2017/2018 it started to:

* Not recognize my voice/command

* Load infinitely when I give a command

* Do the wrong thing entirely

These systems have important complexity layers that may not be immediately apparent: latency and edge hardware limits.

Yes, maybe, gpu cluster server can understand you quickly, but taking whatever model you have and getting it to work quickly enough for people not to be upset is a giant problem.

Not sure how much scaling laws apply here, since this is a seq-to-seq model instead of a autoregressive causal model. It's interesting to see AlexaTM performing better than GPT-3 on SuperGLUE and SQuADv2, but it fails on Chain of Thought prompt, which is a bummer. So, is it because it's a different model or because it is positively leveraging multilingual tokens? I wish they compared this architecture to a classic GPT family model.

I can't parse that sentence.

This is the type of thing you learn to say when you're dealing with a slot filling algorithm that allows for overspecification. By putting it in one utterance, you avoid it saying coming back and saying "Where do you want a reservation?" "When do you want your reservations?" "How many people are in your party?"

That's actually an interesting thought I've been having, will audio interfaces eventually train humans to speak in a certain way?

This is a small model keeping up with the big guys. 20B parameters might fit in 2 beefy GPUs, that's a bargain compared to GPT-3.

As others pointed out, this paper tries to do more with fewer params.

But you've identified a trend does actually describe large language models for the past few years (they've been getting bigger, and bigger has been better). Like microprocessors have the famous tick/tock cycle (https://en.wikipedia.org/wiki/Tick%E2%80%93tock_model), I think models might see be seeing something similar emerge naturally (make models bigger --> make models better (shrink) --> make models bigger again --> make models better (shink again)).

Also, most of this LLM stuff is probably not trained on NVidia hardware -- at scale it's probably cost prohibitive if not also hard to set up. Google's TPUs, MSFT/Amazon's equivalent custom hardware, or other specialized accelerators are more economical overall.

How about the wakeup word even working reliably anymore or not offering me an upsell on every other interaction.

How do you define validate? These models aren't formally proven to work in all cases or anything. They're just tested on a load of data, and if it's found that they work pretty well, then they get released.

It _is_ deterministic (same input gives same output).

You typically don't "test" pairs of inputs/outputs for a model. Instead you measure its performance by defining metrics e.g. "what's the ROUGE-2 score on summarization after fine-tuning AlexaTM 20B using N examples from dataset Y"

You can test some aspects of ML models, like sync testing (if you train on hardware A and run on hardware B, their results are not always the same). But generally you test the code that embeds the model, not the model itself.

I would assume consciousness/sentience requires significant feedback loops, so the "thought process" can "keep going". I don't think any of these models have real feedback loops.

No more than you should worry about your phone, your toaster, or a boulder suddenly becoming self-aware.

Extremely theoretically, a consciousness could spontaneously form at any point [1]. In practice, there is no reason to worry about this - it's not a likely event. There is nothing about this model (or any other one in existence) that increases the likelihood of it achieving sentience when compared to anything else ever.

[1] https://en.wikipedia.org/wiki/Boltzmann_brain

Assuming the premise of this achieving sentience...

> Should we worry if it achieves sentience?

No, because it's easy to kill.

Depends on what you define sentience as.

Is it the presence of a soul, or the presence of capability to learn stuff it is not trained for?

Probably, for ethical reasons. but we should worry a lot more if it achieves sapience. I'm also pretty sure that anything (sapience, sentience, whatever) requires closing the loop so the perceiver can perceive itself perceiving, though the idea that I am merely a single Boltzmannesque forward-pass of a language model will certainly run in the back of my head as I'm trying to drift off to sleep tonight.