Hacker News Reader

Comment by axg11 | Hacker News Reader

It really is great that Meta released the notes/logbook. Credit where credit is due. Very few other academic or industry labs release materials like this, especially when the reality is so messy.

Some interesting takeaways:

- Meta aren't using any software for scientific logbooks, just prepending a document

- So many hardware/cluster issues.

- Hot-swapping algorithms is common and likely underreported (in this case activation functions and optimization method)

- A well resourced team didn't solve enough issues to fully utilize compute resources until >50% of the total time into the project

View on HN

If you are interested in hot-swapping, I highly recommend OpenAI Five paper: https://arxiv.org/abs/1912.06680. It includes 10 months log of how they adapted the model to releases of Dota 2 version 7.19, 7.20, and 7.21.

I agree hot-swapping is underreported, but there are some good existing reports. In fact, OpenAI Five paper today is probably more valuable for its details on hot-swapping than its details on main model which used LSTM and not transformer.

Weird they felt the need to update the version since they used a very limited subset of Dota anyway.

I wonder what software would be good for a logbook like this... I just use google docs for these kinds of things. Sure wandb and jupyter notebooks are good but they are not so good for notes and ideas and documentation

Sometimes the generic solution is just the best. No one requires special training on Google Docs and it just comes with handy features like version control and live updates.

Personally I use notion for this. Not dedicated software, but the extra options in formatting/structuring and linking can make a document like this a lot more readible.

We tried jupyter notebooks for this kind of thing once, and after the equivalent of ~50 pages it becomes unusably slow in my experience.

LogSeq? I use it for this kinda work...

Reply on news.ycombinator.com