Building a license plate reader from scratch with deep learning | Hacker News Reader

Building a license plate reader from scratch with deep learning

nanonets.com

230 pointsole_gooner69 comments

Great article, misleading title. It's not "from scratch" if it's using OpenCV, TensorFlow, and a web service to do all of the work.

Imagine, "Bake Cookies From Scratch," and step one is remove the Pillsbury cookie dough from the freezer...

or you just don’t have to invent the oven, the overall ingredients and how to make a dough, but still need to learn how to get from the basic ideas to something tasty which you want to show to the world...

"If you wish to make an apple pie from scratch, you must first buy Pillsbury Deep Dish Pie Crust" - Carl Sagan

For those not in the know, the joke here is a play on the real quote from Carl Sagan:

If you wish to make an apple pie from scratch, you must first invent the universe.

I guess it's a bit relative. You're not growing the wheat, growing the cane, milling the flour, etc either.

> It's not "from scratch" if it's using OpenCV, TensorFlow, and a web service to do all of the work.

To be fair, the web service (for which this article appears to be inbound marketing) is optional and presented as an alternative to building anything at all.

I work on EasyALPR which I’m putting through Startup School.

One thing often missed in license plate reading is that the tech is really well solved now using an open source stack.

There is always room for improvement so I welcome new technological approaches like this.

However when thinking about license plate data, the real trouble is in what you do with the data, how you handle duplication, and create rules and interface to make the collection useful.

I'd love to hear about your stack. I tried this with -- Object Detection for Plate localization;

-- OpenCV for de-obliqueing angled images

-- Character separation from background;

-- OCR around individual characters via Convnet Classifier

I tried this with decent results, but I wonder what state of the art open-source is?

probably this? https://github.com/openalpr/openalpr

My uncle is a cop in DC and he says >90% of the hits on plates are for the same number but a different state. Is there any work in identifying the state a license plate is from? Just the style is often enough to narrow it down, despite the fact that most license plate frames partially or totally cover up the state name.

I’m. It aware if efforts to identify same plate, different state. If the data quality is so good you’re getting exact same plate different states, that is impressive.

Instead, I’d look at vehicle type identification. There is at least one startup selling an API that determines make / model info that when combined with license plate you would very likely find unique data.

> My uncle is a cop in DC and he says >90% of the hits on plates are for the same number but a different state. Is there any work in identifying the state a license plate is from?

As is so often the case, it seems like it'd be easier to move ourselves to the machine, by including the state information in the code. If a code can only belong to one state, this problem just can't arise.

Perhaps each new plate issued should include the state abbreviation in one of the corners (such as CA for California)

It's possible but not necessary. License plates already feature non-overlap -- a standard California plate is 2AAA222, while a New Mexico plate is AAA222. Neither template can be confused for the other.

> License plates already feature non-overlap

No, they don't.

> a standard California plate is 2AAA222, while a New Mexico plate is AAA222.

And the New Mexico AAA222 pattern is currently shared with Arkansas, Indiana, Iowa, one of the optional Michigan designs, Mississippi, one of the Montana designs, some Nebraska plates, North Carolina, the Northern Mariana Islands, Oklahoma, Puerto Rico, South Carolina, Vermont, and the US Virgin Islands.

To be fair, only NC, OK, and PR share NMs use of a dash between the alpha and numeric portion.

Virginia used to issue in that pattern, with a dash between. Then they ran out and added a 4th digit. However, there are plenty of VA plates still around with the old pattern.

>> License plates already feature non-overlap

> No, they don't.

Yes, they do. I described an example of it, and you agreed. I didn't say they don't feature overlap.

Any improvement in reading plates at a angle? From my experiments it’s very difficult to ready plates from dash cameras of cars in the left and right lane, even though the human eye can identify them.

Hey,

This is an a DIY Article on building a license plate detector using attention OCR.

We deep dive in with this article on how you can automate your data entry work with the help of deep learning based OCR. It speaks about attention mechanisms, spatial transformer networks and how they are applied for any text recognition task.

This seems like an item for "Show HN".

https://news.ycombinator.com/show

(Not trying to scold or anything, but seems like this would fit well with that convention.)

I always seen Show HN for interactive things, not articles (which I would say this is), thinking specifically about this part from the Show HN Guidelines:

> Show HN is for something you've made that other people can play with. HN users can try it out, give you feedback, and ask questions in the thread.

Hi, Could you tell me how to get the charset-labels.txt file? I am running the code

it's in the attention-ocr fork of the same account. the link is mentioned in the README file (the link to number_plates.py file). go into models/research/attention_ocr/python/datasets/data/number_plates. you'll find the charset-labels.txt file there.

Nice primer on how the tech works. The visuals for weighted relationships especially made it click for me.

Musings: I've seen more articles on license plate readers (or perhaps been more aware/paying more attention to these). I understand YC has invested in a LPR-tech company Flock Safety. I'm intrigued by the rise in LPR related articles. Such technology has been around for a while, but perhaps properly modularizing it (each camera unit is wireless, etc) and making it SaaS-backed is innovative. Has anyone else noticed this trend in activity for LPR ML tech (care to explain or to discuss the trend)?

Text OCR is the cannonical neural nets challenge (MNIST) so it seems natural to extend it to be something practical like LPR, without the insane difficulty of other real world shape recognition problems.

The easy ubiquity of LPR is essentially the death knell of privacy of movement. We'd have to move to encrypted transponders that only respond to queries with the right codes, but of course the police would still be able to know where you were whenever they liked. Rolling QR code digital number plates would work. But it won't happen.

Yes, what I found interesting the growth in articles discussing the technology and even open-source projects for LPRs. It's like vehicle facial recognition. Because of this, I've always been intrigued by subversive/defense techniques that fit into the realm of ML - like adversarial patches. Essentially, surveillance in some form is becoming ubiquitous, and private companies especially in the US will probably lead the way. It seems necessary to keep these systems in check at the citizen/individual level because political systems are out-paced by the rate of technological developments.

I like the "Rolling QR code digital number plates" concept; are you aware of rolling QR codes used elsewhere?

Recognizing individual vehicles is actually really difficult from a distance, because all the features are small -- scratches and dints at the 0.1-50mm scale, mm scale paint irregularities visible in other wavelenghts, misalignment of panels, window stickers etc. So licence plates are a huge reduction in privacy.

Other digital transponders, either for tolling or just your devices broadcasting their Wifi/Bluetooth addresses, certainly cause problems but they are within your scope to control.

It would be interesting to see if you could design an adversarial LPR jammer, that would not look to a cop like a licence plate.

When I wrote 'rolling code' I was being a bit off the cuff.

A PKI based system might be something like: {nonce,E(rego_pub,(nonce,car ID, date, hour)),S(car_priv,(nonce,car ID, date, hour))}, where rego_pub is the public key of the local authority, car_priv is the private key of the car, E is encrypt, S is sign.

This is already quite complicated, requires central PKI etc. And unfortunately QR codes themselves are not a great fit, because you need to be close to read them, but you could have a variant designed for shorter strings.

Another possibility might be to use a (prefix, TOTP code). With a fixed prefix indexed on car model/year (which is already visible from looking at it), the set of all TOTP codes from that fleet at that time could easily be searched by the registration authority to identify which vehicle it was. So, e.g., all blue Toyota Camry 2018s would have prefix 'P9J' and then a 10 minute changing code, like 'Y3KE'.

So if a cop needs to look up a car the LPR reads P9JY3KE. The local terminal says it should be a blue Camry, and then sends the string to the validator, which computes the TOTP for all vehicles in the class (like, 2000 in a group) and see if any of them match. If none match, pull them over.

There are a few problems with this still, e.g. replay attacks. But possibly solved by using IFF techniques, i.e. the police car can send a signed query forcing the named plate to show more digits of the TOTP code.

Criminals just steal the plates off another vehicle (of the same model/color if they are smarter), so maybe they can just steal the whole digital licence plate computer module? Unless the smart licence plate is part of the vehicle security system, on the car bus, it is still going to happen.

--- The whole IOV (Internet of Vehicles), V2V (Vehicle to Vehicle) space is going to have to deal with these problems, and there is no shortage of protocols but I don't think it is remotely solved. Lots of companies pushing 5G approaches for this, but I wonder if transponders like for aircraft (ADS-B) or ships (AIS) won't be a simpler (and much cheaper) way, even if they are still using mm wave radios.

I wonder if openalpr is helping.

https://github.com/openalpr/openalpr

basically:

    alpr imagefile

will recognize a plate. It just as simple in python.

I've been mulling over trying to build one for our cove/neighborhood.

Ring/security cameras are fine but being able to pinpoint a handful of plates to the wee hours of the morning would allow theft issues to be handled pretty easily when cross referenced with camera tech.

Ex. Right now you get a description like "Older white 4 door" or "White Nissan Sentra"

Bounce that against plates from that time and you've got a good match to work off of, especially if the owner matches descriptions from cameras.

You don't need automated license plate recognition for that, though. If you find the motion events in the wee hours, and have a sufficiently high-quality video frame [1], you can easily check the type of car and then know the license plate by reading it with the neural network that was preinstalled between your ears rather than a software one.

I've been writing my own home surveillance camera network video recorder software, so I've thought about plugging in something like this, but I don't think it's really necessary. The one time I might use it is if there's a car that was likely involved in a burglary, I might scan the previous few weeks to find previous times they were in-frame. Then I might see their scoping the place out, potentially revealing more about themselves. It's too labor-intensive to look through that many motion events by hand.

[1] My understanding is this requires some care. You have to select the right kind of camera, place/aim/zoom it carefully, and tune it for license plate recognition, particularly at night. I think the cameras typically come tuned for slow-moving, not-very-reflective-to-IR objects. License plates on a moving car are the opposite, so you want to say decrease the aperture and increase the shutter speed from the default, position it carefully as I said, maybe tweak some other settings, and test it.

The danger of doing that systematically is that you have a record to subpoena.

So be prepared for the requests from divorce attorneys and police. Better to just have video to make it more difficult.

The primary idea behind this post is to show how you can use OCR technology to one of the simplest problems (LPR). Given how widespread Licence Plates are and how easily you can collect this data. Most other use cases are a little more niche and difficult to relate with.

I doubt LPR will be the primary utility of this kind of tech.

> I doubt LPR will be the primary utility of this kind of tech.

You're right. LPR isn't going to be the primary utility, but it's very possible that this will be one of the first areas to benefit from this tech at scale.

Police use cases are driven by federal grants to local law enforcement, and the companies tend to be smallish, product driven (vs service) and not great from a technology integration perspective.

Ditto for similar big systems like EZPass are expensive. Some read transactions cost the government as much as $2.

It’s a market where the price is going to plummet as it becomes a service delivered thing. You’ll pull into a grocery store parking lot and generate an alarm to have a clerk get your groceries ready, and when you park it will tell him where to go. You’re also going to have a lot more tolls.

For what it is worth, I'm pretty sure that Amazon is already doing this at their Amazon Fresh drive-up stores. When you drive up, a sign lights up to say that your order is coming soon. The experience degrades a little bit if you arrive in a car they've never seen before -- a human stops by your window with a tablet to check you in.

That’s really cool!

Haven’t seen these yet, but Target has a pretty cool system that is app based. Easy to see how they could hook in cars.

Huh. It didn't even occur to me that they could also be using the app on my phone to do the same thing, and do it more accurately.

Step 1: read Tensorflow first tutorial (which is about OCR). Step 2: try to apply it (license plate). Step 3: don't study any math but brag about knowing AI for your friends so you seem smart.

Success as defined by non-educated (and some from the social studies) people everywhere.

I cannot think of a use of license-plate reader technology which is not immoral.

With the rise of this kind of tech and its prevalence, I think the privacy-focused among us should start pushing to outlaw the use of this tech and eliminate the need for external identification on vehicles.

> I cannot think of a use of license-plate reader technology which is not immoral.

The parking garage I use every weekday reads my plate, and opens the gate as I have a subscription. Same thing happens when I want to leave. Is that immoral?

It's less about the technology about license-plate reading and more about what people do with the data afterwards.

I would restructure the argument as " I cannot think of a usage for storing the data of a license-plate reader for purpose of selling it which is not immoral." or something like that.

> It's less about the technology about license-plate reading and more about what people do with the data afterwards.

As someone who cares very much about probably I think this is a point that cannot be stressed enough. It is mostly about storage. Storing things is the major danger. Of course there are use cases too, and I would feel uncomfortable with the police tracking every car on the 5 freeway, but tracking requires storage.

Honestly I think your parking garage example is a great use. I can think of others. A notification of your friend or family member pulling into your driveway. Have your house do things like turn on the lights or give you a notification. It could be a good way to do parking meters. I'm sure people could think of more. Technology always has two sides to the coin, it's always a balance of using it only for ethical things. But that requires nuance.

Edit: a cool way to use these might be like we use passwords. Your license plate is a password (or let's say username). OCR is used to identify the plate, then it hashes it, checks the hash with that in the database and bingo, door opens. I think that's a level of privacy most of us are comfortable with. There's no retention of the license plate, no good way to identify who's it is, and if the database is hacked the attacker doesn't get your plate.

When I visited Seoul this was one of the coolest "daily life" things I noticed. All parking garages, even by the hour ones, operated this way. There was no ticket to lose, no ticket to accidentally de-magnetize, etc. You pulled in unhindered, and when you went to left the machine had your total up for display before you even stopped rolling to pay.

>The parking garage I use every weekday reads my plate, and opens the gate as I have a subscription.

Hmm, so would it open for me if I used a photo of your number plate near or over my own plate?

You've discovered a wonderful loophole: Fraud!

So you can make your own judgement about the morality of this, but here's a system that I worked on and I sleep just fine at night[1]. I'm a privacy-conscious guy, and nothing I saw working on this made me particularly uneasy.

Highway Weigh Stations are ostensibly there to ensure safe commercial vehicle operations. The system I worked on had a number of different sensors. A mile or so before the weigh station, there was a weight-in-motion (WIM) scale embedded in the highway; this part of the system measures (somewhat coarsely) the weight of the vehicle and counts the number of axles it has. Around there, there's also an ALPR and a USDOT reader[2]. Using the weight, ALPR, and USDOT information, a decision is made as to whether the vehicle is required to check in at the weigh station (which is indicated using a road-side sign). There's no fine yet or anything, vehicles are just flagged for further inspection based on:

- measured weight vs GVWR

- company history / safety rating

- a few other factors that I don't recall off the top of my head

There's a few more ALPR cameras sprinkled throughout the weigh station itself, and another one on the highway past the station. These ones are used to track the motion of the vehicle through the system. For example, if a truck has been flagged to stop, and it instead skips the station and keeps on going, the system knows right away that they've bypassed and can send out an enforcement officer. Alternatively, if the plate is seen at the (more accurate static) scale at the station, the system automatically correlates the weight and the plate, and if it's within range, the driver can sometimes leave without even having to interact with any enforcement officers.

[1] Note that it specifically ignored non-commercial traffic unless there was something really weird going on. Categorically was not keeping persistent records of all traffic.

[2] Funny enough, there's no significant regulations specifying the exact font/size/placement for USDOT information, so a reliable USDOT camera is significantly harder to make than a reliable ALPR.

How hard would it be to remove those bits that ignored non-commercial traffic? You might be a privacy conscious guy, but what is to stop the owner of this code base from removing this if statement? I am sure a company like Infosys would be happy to take on a project like this, and throw some less liberty minded people at the problem.

There's a dead reply to my comment that I'd like to address anyway, in a slightly glib manner:

> How hard would it be to remove those bits that ignored non-commercial traffic?

Code-wise? It'd be really easy to add an INSERT that stuffs extra data into another table. But as far as "removing the bits that ignore non-commercial traffic", none of the non-commercial traffic would be in the datasets provided by the various federal agencies (whose acronyms I forget).

Performance-wise? The whole thing is a giant mass of Oracle PL/SQL that can barely keep up with the commercial traffic. If we tried processing all non-commercial traffic through the scoring algorithms etc, the whole house of cards would fall down :D

frictionless toll collection

Huh? The whole reason license plates exist is to be read. What makes a machine reading it always immoral compared to a human? Particularly the police identifying vehicles they're searching for. Why shouldn't they use technology to help find criminals instead of sitting around watching cars go by with their own eyes?

I once had a problem with my car in a big carpark. I asked the security guard for help and he wanted my license plate number. He typed it into the computer and the camera found my car and panned and zoomed onto it. Amazing! Much easier than trying to explain the location. So there's your non-immoral example and you can stop worrying.

As for privacy. So what? Where are people going in public that's such a secret? If you want to do something secretly, do it in private. If it involves using the public streets, then accept that it's public.

> Where are people going in public that'a such a secret?

It really doesn't take much of an imagination to come up with dozens of examples, so I'll just give a couple: gay bars, abortion clinics

> Where are people going in public that'a such a secret?

It really doesn't take much of an imagination to come up with places people might not want others to know they go to.

STD clinics, abortion clinics, AA/NA meetings and gay bars would be some low hanging fruit if you really can't think of any on your own.

> If it involves using the public streets, then accept that it's public.

"If you didn't want your violent ex husband to track you down, you shouldn't have used public streets to get to the shelter"

None of those things are really private anyway. Private investigators, police, or stalkers can follow people from their house to wherever they go, and can identify you by your car's appearance even without the license plate. The difference here could be in the scale and ease of access to such data though.

Nobody seems to mind that Google Streetview shows cars. My sister visited her mother in another city at the time they drove past so now there's a permanent record that she was there at that time to anyone who knows what her car looks like. Is that a problem too?

> "Private investigators, police, or stalkers can follow people from their house to wherever they go, and can identify you by your car's appearance even without the license plate."

Quantity has a quality all of it's own.

If we ignore the obvious possibility of blackmailing people who have something (legal) to hide, there's also a huge problem of innocent people being in a wrong place at a wrong time. It's not hard to presume that police would use such database to quickly find everyone with a previous convictions who were in the neighborhood when crime happened, and would then try really hard to connect them to it. Which is great if they pick the right guy right away, but history teaches us that they often don't. Mistakes happen a lot in investigations, and this type of tool - if police trust it too much - could make it even easier for them to jump to conclusions and fall for prejudices. Even if they don't, at the best, it will still create a lot of inconvenience to people who only happened to drive too close to the crime scene in a wrong time. Having police questioning you is very sensitive and unpleasant experience, especially if you're not the white middle-class and you have previous convictions. Police coming to your work can get you fired immediately even if you're completely innocent. Or landlord can kick you out, or wife can leave you. You could easily completely screw up someone's life with this, and still not get any closer to solve the actual crime.

The problem there is the police using bad processes. That's not insolvable. Courts already have processes to protect against some of that cherry picking and prosecutor's fallacy type stuff. Avoiding technology doesn't seem like a sensible solution when safety processes already work in many areas.

If you can legally be fired for having police show up at work, then you're at risk of being fired for any reason and that itself is a problem. Solve that problem with employment protection laws like many countries already have.

You're talking about a wishful thinking, I'm talking about the (for many people rather harsh) reality. I agree, it can (and should) be solved in the ways you mention, but this technology will not wait, it's here now. At the moment it's just incredible how often cops and prosecutors will trick possibly innocent people into plea bargaining to close the cases as quickly as possible and avoid having to prove them guilty in actual trials. Tech that makes it easier to round-up "the usual suspects" will just make it easier for them to do it to an even bigger group of people.

> Where are people going in public that's such a secret?

How can you travel to any secret place without passing public areas?

Put your secret places inside private property with privacy protection. Prostitutes in hotels, and gambling in the back of a shop, for instance.

> secret

The conflation of privacy and secrecy is an issue that deserves addressing: When you walk into a public restroom stall, it's not really a secret what you're going to do in there. But you still close the door, don't you?

What I'm doing or where I'm going may not be a secret, but that doesn't mean I want the details logged and sold by some data broker.

I use a camera in conjunction with a raspberry pi to announce when someone drives down my driveway. Known license plates it will announce the name of the person driving down the driveway. Unknown license plates it announces that I have an unexpected visitor. I feel like it's pretty useful personally.

Let's see, you could use it as a supercool way to open your gate and/or garage doors when your own car approaches the entrance. You have a guest visiting? No problem, open the gate for their license plates too.

I'd love to have personal ALPR on my car: I'd like a heads up when I'm near a crazy driver who has nearly hit me before. Since I drive similar paths, even without noticing, I'm likely to run into the same cars repeatedly. And I'd like to know/log who is pulling into my driveway and when.

If you're not trying to log the movements of the public, the usage is likely moral.

Not a privacy issue--no expectation of privacy in public and you do not have a right to drive on public roads, driving is a privilege and not a right.

Reply on news.ycombinator.com