lucidrains

previous systems could not compose objects within the scene correctly, not to this degree. what changed to allow for this? could this be a heavily cherrypicked example? guess we will have to wait for the paper and model to find out

Yeah I know

I've been to weddings from players who met on the site. It was magical

No, you aren't wrong. For ML people, it is quite simple and hopefully the final code reflects that

The aim is really to give a good base for follow up research / modifications, which I think there will be many for this paper

Lol, hey Danny. Indeed it was great times

I may bring it back, rebuilt in rails and svelte

just to better match the math equations in the SPIN paper

hey, appreciate the interest! repo is not done yet, but probably will be around month's end

been using the self checkout at Uniqlo recently. works beautifully

nice! will make reading papers on the phone so much more pleasant!

slow and steady wins the race

gradtree is cool! thanks for the share!

Yannic did a great interview with the authors some time ago https://youtu.be/_NMQyOu2HTo

thank you for sharing your thoughts! is there anyway I can get in touch with you, through email or some other means?

you have no idea what you are talking about

this is fixing a different issue, not the one you are measuring.

huge! thank you Tri!

got a response here https://github.com/lucidrains/soundstorm-pytorch/discussions...

can an expert in the field comment on whether the results are more or less impressive than Soundstorm?

same, I am too old to really care if others cannot see this as nearly the perfect language

Same, I love it! Having done ruby, javascript, python, I'm allowed one niche language (although I really hope it grows in popularity!)

tldr: bigger text encoder is better. SD will catch up quickly, as conditioning on a new set of precomputed text embeddings is a trivial change

besides everything that was mentioned here, what made it finally click for me early in my journey was running through this excellent tutorial by Peter Bloem multiple times https://peterbloem.nl/blog/transformers highly recommend

the code is here https://github.com/hazyresearch/safari you should try it and let us know your verdict.

or love, understand, help each of those human beings.

that line of research is still going. https://github.com/lucidrains/block-recurrent-transformer-py... i think it is worth continuing research on both fronts.

It is only linear in terms of memory, not compute. Flash attention is a big advance, but not enough for 1 million tokens

Yikes, didn't know this was an issue at Google as well. That is scary

When I was working at Uber, we had a case of a guy stalking his ex-girlfriend's every movement through the backend. They eventually found out, fired him, and stepped up auditing.

while this topic has visibility, there is also another relatively unexplored research direction, fine-tuning pretrained transformers into RNNs

https://arxiv.org/abs/2210.04243

https://aclanthology.org/2021.emnlp-main.830.pdf

that's the one I bought!