Exactly. This is called overfitting and it's most definitely a thing.