Number plate recognition with Tensorflow
matthewearl.github.io
matthewearl.github.io
A couple of suggestions:
You will almost certainly get better accuracy with a broader array of input fonts. I'm not sure how well you can simulate all of the image conditions that might occur in a real system so adding some real images to the training set (if you can find them - but I suspect there are OCR type training sets available these days) would probably help.
For the speed problem I like the idea of using a simpler net, probably with a course scale image for pre-detection. You can set an appropriate detection threshold to avoid false negatives and thereby avoid running the full net on the vast majority of images which won't contain anything that looks like a plate number.
Wouldn't best accuracy come from obtaining the single font that is actually used for the licence plates (or finding the closest match), since in this scenario there isn't any variation as all plates are made according to a single standard?
I think that if you're going to use a net you should try to train it in such a way that it learns to generalize the fundamental shape characteristics of the characters, much like the human visual system does, and that requires training with a wide range of different type of characters, all considered to be of the same class.
If you're convinced that you only need to deal with identical characters you're probably better off using some form of template matching. But I think this is rarely the case for real world systems.
Anyone knows if confidence intervals can be estimated on probabilities for neural nets?
In my experience sometimes both the decision and the confidence will be wildly wrong (ie. wrong answer with 99% probability) but most of the time this works reasonably well.
Isn't "confidence" measuring the internal state of the net, the probability based on classes known to the net? In that case the answer can be wrong, but "confidence" is never wrong.
I don't quite follow your meaning but no, that's not quite how I think about it.
If x is the input image and C is the class selected by the classifier we expect the confidence to be representative of the conditional probability P( C | x ). There is theory that shows this to be true given certain assumptions, but as is very often the case in this field the extent to which those assumptions are valid is rarely clear. So if the net outputs a confidence of 0.9999 we expect there to be only 1 chance in 10000 that the true class is something other than C. If we test the net and find such a confidence associated with an incorrect classification more frequently than predicted then we can say that the confidence is objectively incorrect.
Confidence is simply how much confidence you have that the predicted class is correct.
This is often important because in a real system you often want to know whether to accept the model's prediction or pass the case on to a more accurate, more expensive model (sometimes a human). You then can select a confidence threshold that gives an acceptable trade-off between false positives and false negatives (ie. choose an operating point on the ROC curve). If your model just gives a yes/no answer with no measure of confidence, or if you don't know how to interpret the output as a confidence, then you don't have sufficient information to do this.
Over-fitting is often part of the problem but in my experience things aren't as black or white as you seem to imply. There are things you can do to limit over-fitting but I don't know of any simple test that will give you a 0 if you're over-fitting and 1 if you're not.
If you mean correct among the number of output nodes, then we are talking about the same thing. If you mean correct among all the possibilities to include those unknown to the NN, then I'd appreciate a link - because I've never seen that before and it sounds incredibly useful.
> ...I don't know of any simple test that will give you a 0 if you're over-fitting and 1 if you're not.
Neither do I, that is why I used all those weasel words - but I also don't know of a neural network that wasn't either being misused or being overfit that returned such results. A simple example would be the 2 inputs for the xor test data, a hidden layer with enough nodes to memorize every permutation, and 2 output nodes - each representing the confidence for the potential result of the binary operation.
I don't think this holds with a softmax output layer because I think that provides an output vector that is implicitly normalized across all classes. It also obviously doesn't hold if you explicitly normalize the sum of the output vector to 1.
For your second point I'm not sure I disagree with what you're saying, certainly not on the basis of the example you provided before since I would never expect to see 0's and 1's in the output of a net.
EDIT: After thinking about this some more I realized that I may have been mistaken in what I said above. It may be that for an unknown input you would expect to see a high entropy output distribution whose sum is close to 1 but where the average output value is about 1/N (ie. the number of output classes). If this is the case then the logistic and softmax cases would be similar. This still indicates a low confidence though, I wouldn't expect to see outputs in the upper quartile, unlike for a cleanly recognized input where you might see 0.8 or 0.9. I'm going to try this with a net I've trained for mnist using random inputs to check but unfortunately I have to retrain the net since I didn't save it and my laptop's busy with another task that I don't want to stop now so it may take a while.
> If we test the net and find such a confidence associated with an incorrect classification more frequently than predicted then we can say that the confidence is objectively incorrect.
On the test data performance is never going to match the NN's training set performance, so it would always overestimate its confidence.
EDIT: Maybe it's more complicated than that though because the experiment is itself estimating a probability, not just the value of some random variable. Maybe we need a real statistics expert to look into this.
What I mean is a confidence interval on probabilities. That is, if you did the training on 100 different training sets that follow the same distribution as your training set, they would have a range of outputs for a particular sample. The 95th highest and 5th lowest would have specific values. These values can be estimated and are the bounds of the CI.
For example, for logistic regression, you can estimate standard error per coefficient like so: https://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_... That would give you a vector of standard errors the size of your variable(feature) set. Then you can use that to estimate overall range of CI per sample.
This would be useful in the example provided in OP's link. For example, let's say that the classifier says there is 86% probability that the license plate is RBX735.
Can we really trust this 86% probability or is the model primarily relying on some information that is available in only 2 samples out of 1000 that might have been outliers?
EDIT: For my own information what would knowing the confidence intervals tell you that P( C | x ) wouldn't ?
The definition of a Q (e.g Q = 0.95) confidence interval a < X < b is just that P(a < X < b) = Q. It's not even necessary that the interval contain the most probable value of X, but when you actually calculate them to want to tack on extra constraints to prevent stuff like that, like minimising the length of the interval.
I'm pretty sure there are solutions available that works much better though, my point is that describing openalpr as 'state of the art' isn't quite accurate.
Are there any good resources that you would suggest?
The machine I'm using has an Nvidia GTX 970, but I failed to enable CUDA in the TensorFlow installation so it's not using it. When I noticed that I decided just to let it run and then, when it finishes, train it again with the GPU and compare.
if you blur license plates or text, people will still be able to read it!