Do the standard datasets not work too well (e.g. MNIST or NIST)?
http://yann.lecun.com/exdb/mnist/
http://yann.lecun.com/exdb/mnist/
I think the artifacts from the square borders (eg if it's slightly mis-cropped or misaligned) were also tripping up the model before I had my custom dataset loaded in there.
Edit: I should say this was my first stab at doing machine learning so I may be missing something; definitely open to ideas or suggestions!