Training a Lite Bert on AWS

Great tutorial. Now instead of the 8x V100 $24/hr p3 instance, you can use a Lambda GPU Cloud 8x V100 instance for $12/hr (https://lambdalabs.com/service/gpu-cloud). It’s crazy how much Amazon still charges for older hardware.

In addition, when you’re first starting off with Deep Learning you should just build a GPU machine under your desk and use that to train. The suggestion in the post to do horizontally scaled distributed training is a great way to horizontally scale your monthly bill too. I’ve seen dozens of people spend $500 on an AWS training experiment only to realize they could have bought a 2nd hand 1080 Ti with that same money and been able to train for the next 2 years.

BERT models are in season. Would expect more pretrained models to available from AWS.