> I simply don’t think it is responsible to splurge on a 4 GPU machine in order to make this project go faster.
2 things: 1. You can rent 8-GPU machines on AWS, Azure or GCE. 2. The incredibly wide applicability of machine learning means that an investment in hardware might not be wasted. Even if you only use the machine for this one project, if it helps you learn more about the field it will probably still be a good investment career wise.
Also, keep in mind that the dataset is still tiny and that a method that works for large numbers of images may very well fail if you only have a few tens to maybe 100 or so images per class.
Now, if you're changing the architecture (such as by adding additional categories of pieces), as I said, that's more tricky - what people usually do there is something like lop off the top layers and retrain them from scratch, possibly while freezing the rest of the NN (the assumption there being that the learned filters and lower layers ought to already be sufficient to classify a new category, which is reasonable since the lower layers tend to be learning things like lines and corners, all primitives which should be able to classify yet another square or rectangle etc).
Since this is the obvious response any reader familiar with deep learning would have while reading complaints about how slow your CNN is to train from scratch, it'd be good to discuss it in some detail what sort of finetuning you've tried and how it failed.
I was about ready to give up on it when I decided to try to bring up a net from scratch and that worked quite well.
Do I understand correctly that a checkpoint is just a snapshot of the model at a point in time? i.e. "Here are the probabilities of each outcome given the characteristics I have observed already."
Also, what does "fully converged" signify? Are there points in the course of training the model at which it is more appropriate to "save" progress than at other times?
In machine learning/deep learning, the decrease in training loss has major diminishing returns as training continues. Eventually, training the model hits a point where the loss barely improves each epoch/iteration. (fun visualization from one of my projects: http://minimaxir.com/img/char-embeddings/epoch-losses.png)
In some cases, the loss can stop improving entirely, or increase.
Pay-for-what-you-use large-scale learning will get models trained faster than local single GPU, then you run the trained model locally.