The downside is that you need to properly preprocess code, have less non-code Training Data, and can not adapt easily to new programming languages