I am gonna read this paper and the other latent sentence later today. I always advocated for this kind of solutions together with latent sentence search should get to the next level of AI. Amazing work from Meta
I don’t get it, isn’t this concept modelling exactly whats going on in the deeper layers of current LLMs?
What it means is a harder question. Perhaps transformers are simply an inefficient computational structure for this process? Perhaps a more flexible computational structure would integrate this step more efficiently? Perhaps Transformers are efficient enough, but our learning/densifying isn't? Or perhaps it's such a core powerful step that it might as well be built into the algo regardless? Much to learn.