For more training data, I wonder if you could make Lego parts in Sketchup or some 3D program, then render them in a 'scene' similar to your camera setup using a renderer like Maxwell or V-Ray or whatever. Then you could maybe be able to generate unlimited numbers of sample images to train on.

I'm doing a similar experiment now to train a model to parse out an image of a blood pressure monitor that's a 7-segment LCD display. To do it I separated out each segment of the display as masks with Gimp/Photoshop and then I can create my own images by just overlaying them on top of an image of a blank LCD display. That gets me basically unlimited training photos. If you could render the 3D parts from various angles, colours, etc then something similar might be possible.

Also, you said you're doing modified VGG and into 20k classes. That works, but another thing to maybe try is use binary_crossentropy as the loss function and a sigmoid (instead of softmax) on the final activation layer, to be able to do multiclass classification. Then your labels could be a vector of shape possibilities, colour possibilities, or whatever you could divide your 20k classes into.