HMT: Hierarchical Memory Transformer for Long Context Language Processing

arxiv.org

87 points·jasondavies··6 comments

Code: https://github.com/OswaldHe/HMT-pytorch

This looks really interesting. I've added the paper to my reading list and look forward to playing with the code. I'm curious to see what kinds of improvements we can get by agumenting Transformers and other generative sequence models with this and other mechanisms implementing hierarchical memory.[a]

Shouldn't the authors cite the work by Jeff Hawkins et al at Numenta? Hawkins has been proposing AI models with hierarchical temporal memory for a long time.[b] I can't help but wonder if there is a way, somehow, to incorporate his work and ideas in Transformers and other generative sequence models.

We sure live in interesting times!

---

[a] In the past, I've experimented with mechanisms that add memory to Transformers, but never with hierarchy.

[b] https://en.wikipedia.org/wiki/Hierarchical_temporal_memory

I thought Hawkins's book "On Intelligence" was amazing. It's a bit wild how close things have followed to the direction he laid out.

That's kinda hilarious, because I think the book was exactly wrong in its predictions. This can be evidenced by the continuous failures of his AI company, Numenta.

Well both things can be true at the same time.

I agree Numenta missed the boat, but it doesn't mean that book wasn't prescient. Numenta just didn't get there first, wasn't a fast follower, and blew a huge lead in AI. It may still end up the HTM is one of the final state solution,s but they are so far from ever being able to capitalize on it that it is unlikely to even matter if they invented the concept.

Does it relate to applications in time series? How does the hierarchy play a role in Transformers?

Not really. But questions about the work should be asked to the authors. I've only skimmed the paper.

Reply on news.ycombinator.com