BERT isn't dead for smaller tasks (think NER, Sentiment Analysis) where low latency is needed.