Microsoft releases code to enable training BERT at large scale on commodity GPUs

The actual code is in this repo: https://github.com/microsoft/AzureML-BERT

Disclosure: I work at Microsoft/Azure on ML

Really excited to announce the release of the code we use to train/fine-tune BERT at Azure & Bing. Please let me know if you have any questions!

seems like the scripts have a dependency on pytorch_pretrained_bert library. Is there a way to run the dataprep/create_pretraining.py or train.py scripts on-premise?

The recipes are optimized for AzureML but the implementation can be adapted to any cluster. --aagarg [MSFT]

If you have any questions, please let us know. maxluk [MSFT]