If the quantile assignments took into account data from both the training set and the test set, then it means the training set quantile label itself contains lookahead information about what the distribution of test set values is.
This can lead to artificially increased test set accuracy since by optimizing the model during training, you are partially optimizing based on information you gained by knowing about the test set in advance.
Training-validation-testing generally means keeping a portion of the training set held out for evaluating accuracy and convergence after one full iteration of training updates. So you get some information per each training update regarding whether the model had overfitting, or performs unrealistically well on the validation set (which it’s not trained on), or for early stopping criteria.
Then, after all that, you move on to checking the final accuracy and metrics on a fully held out test set.
This validation approach is common in deep learning because of the many diagnostics you need to get information about during training. Without some information outside the training set itself, it can be hard tonunderstand how the learning rate is affecting you, how likely overfitting is, whether there is a vanishing gradient problem.
Waiting until all the way to the end of training to get any feedback on these is sometimes just too inefficient or too risky that you’ll discover an issue only after a huge time sink.