Its impressive for what it is, but training would be painful at those latencies (fp16, batch 32, sequence length 512 generates a ~500ms forward pass with a 22M param model)
Like for example
- did the user tap the wrong location on the screen because their device was physically jolted, and can you correct for that, considering you have access to accelerometer in HTML5
- does the user keep repeating an action (checking every box in a list of e-mails) and can you extrapolate the rest of what the user wants to do
- did the user bounce because you popped up a stupid intercom box or newsletter popup, and did you learn anything about what you need to do if you want to retain this particular user in the future
these kinds of things could be done with hundreds or thousands of parameters or less
EDIT: (I responded before your full edit with the bullet list). This next comment is orthogonal to the slow training performance topic I think, but the use cases you reference there don't seem to be well-suited at all for an autoregressive decoder-only model architecture.