DeDLOC – Distributed Deep Learning in Open Collaborations

Yandex, Hugging Face, HSE, Moscow Institute of Physics and Technology, University of Toronto and the Vector Institute propose a method to share compute power and ran a decentralized collaborative training of an Albert-like model in Bengali.

https://arxiv.org/abs/2106.10207

https://github.com/yandex-research/DeDLOC

https://twitter.com/m_ryabinin/status/1406988626374762503