How I failed to replicate an $86M project in 1 line of code | Hacker News Reader

How I failed to replicate an $86M project in 1 line of code

medium.com

150 pointsrodgerd43 comments

The post starts with a false premise that:

"[a medium post has been] doing the rounds the last few days, describing how an automated license plate recognition (ALPR) system being developed for the Australian Victoria Police could just use the open-source ALPR system OpenALPR instead"

The original post was pretty much a tech demo, and noted a bunch of issues. I think the point was that an ALPR project shouldn't really cost $86M.

ALPR is a well studied problem, there are a number of commercial solutions. I've no doubt there are even a number of solutions already deployed in Australia. Such systems are routinely used, for example, to monitor the speed of traffic (see Traffic master).

And as the original post showed, there are even open source solutions to the problem that work pretty well.

So yea... $86M seems like a lot when there are probably off the shelf systems you can buy at a much lower cost...

The Victorian police system was first running in cars on public roads in 2012. No doubt there are plenty of solutions available today that have been developed since then, but they probably weren't considered at the time. That $86m has covered development of the software and hardware and at least six years of running costs for the program. So yea, I'm sure $86m sounds like a lot when you compare it to some random piece of software that does a small portion of the project.

The first ALPR systems were deployed in the late 1970s.

Fixed camera systems were rolled out in Australia in 2005...

So.. ALPR is not new. In 2012 there were multiple systems available. 86MUSD seems like a lot to adapt existing research or existing systems. Even if that includes acquiring/maintaining several vehicles.

Maybe the whole project is being mischaracterized. Perhaps it included requirements completely unrelated the ALPR (face recognition, driver identification? don't know...).

But with the information available, 86MUSD seems like a lot.

For a start, it's not USD (why on earth would an Australian organization talk about costs in USD?). And of course we know it includes requirements completely unrelated to ALPR, it connects to a bunch of existing databases and systems to do live tagging of stolen vehicles, vehicles owned by people with an active warrant or suspended license, etc.

In 2012 USD a Australian dollar were basically 1 to 1 exchange rate. So it doesn't really matter does it? As to why I might expect it, well... USD is pretty much the international standard, so on an international website I often expect things to be quoted in USD...

Right... "connecting to a bunch of existing databases". Does not sound like it's worth 86MUSD to me. But there we go, I guess we just disagree on this point.

In think the author of the original article really shot himself in the for by going with such a clickbaity title. I went into the post with low expectations, but I thought it was actually not bad, and as you say the point really was that the Australian government seems to be paying a lot of money for this system.

That medium post is entitled "How I replicated an $86 million project in 57 lines of code". While it does contain some caveats, it's not like some copy editor misrepresented the authors intent when coming up with the headline.

Indeed.

How I replicated an $86 million project in 57 lines of code

The thesis is that the author created the equivalent of the multi-million dollar project.

"Replicated", not "imitated", "prototyped", "created a POC", etc.

Titles bring clicks; titles bring scrutiny. Live by the sword, and die by the sword.

What if they created the 57 line project, and spent $86 million to improve OpenALPR?

Then it would be useless to the people who spent the money, because they wouldn't have the cars and cameras and ongoing support and research that they wanted to spend that money on.

Sounds like you're arguing a different point. My assumption was that 'x' == "86 million" AND 'x' == "the amount it costs to develop the software". So, I would just replace "86 million" with "actual cost of developing the software".

Yes, I was arguing that your assumption was wrong.

"straw man" - an intentionally misrepresented proposition that is set up because it is easier to defeat than an opponent's real argument

The problem isn't ALPR, the problem is the imagery. You must calibrate your cameras and take a few steps to get quality images, or it won't matter how what software you use.

I agree with this article more. Yes, $86M is sure expensive with lots of efficiencies. But the first one is oversimplifying the technical aspect, which is missing the point on the sources of inefficiencies. The project can't be using openALPR at its current level and it is hopeless to use it as a starting point to get decent accuracy.

The first article points out the inefficiency and does a good job of replicating some basic functionality. It's basically saying "with this starting pointing, the performance is not as good, but we could probably make it a lot better without spending $86 mil"

IMO, this one is saying "I'm gonna push the starting point even further back and complain about someone making a suggestion about inefficiencies."

But that's the internet I guess.

A little digging seems to indicate that the author makes a living as an engineer whose livelihood relies on customers paying for solutions provided by the company he works for, so naturally it is offensive that someone would criticize inflated costs associated with a third-party, closed source solution to a problem.

Two problems:

1. You've shown the success of OpenALPR isn't perfect, but I'm sure the closed source alternative isn't perfect either. It might be better, but is it $86 million better?

2. If we spent $86 million on developing OpenALPR, it would be an incredible product that everyone could use, without paying so much to the overhead of an executive structure.

To even begin to answer your question, you'd need to know what the project included besides the software that OpenALPR could theoretically replace. Like how much hardware has it bought? How much of that money went to paying police officers for time spent developing requirements, testing the prototypes, and learning the eventual system? How much of it went to the research needed to develop this system some five years before openALPR was available? How much went to paying the software developers a normal first world wage and benefits for five+ years?

YouTube-encoded video is going to give relatively awful results, no matter what vision software you put it through.

Good cameras, lenses, and camera placement will make or break a project like this.

I'm pretty sure you could make it realtime with more than 10x improvement:

Scan a thinner section of the video because the sky/ground is useless, and process every 10th frame.

This requires a vehicle to be right in front of the (police) car. Not very practical as most of the license plates you get to scan will be in other locations of the video - you know, since the car in front of you may be there the next 10 minutes whereas cars drive past you and are parked to the side all the time.

you could narrow it 1/3rd at least without really affecting results at all, then include the 10x improvement from skipping frames, since you really don't need that many. The world doesn't move that fast.

Actually, the world (the part of it you're interested in, i.e. the license plate) does move that fast, relative to the camera, in terms of pixels/s. You need 10fps for patrol car ALPR to work well.

Source: I've worked with ALPR for 7 years.

You just need a short exposure, blur is not related to framerate, it is related to exposure time.

Original author of the original article this person is responding to. Saw this response earlier today which made me a bit sad, but mostly excited that people are having a crack at it themselves.

It's currently Father's Day here in Australia so I don't have a chance to respond right now. On the road all day.

I'll have to do a follow up next weekend, I'm absolutely flat chat both in and out of work at the moment.

Whether the actual project is worth exactly $1 or $86 million is not the issue. Not even whether a $1/67 lines of code can replicate a $86 million project.

The real substance -- and the first posts hints at it -- is that government (and private sector) projects are more often than not many times more expensive than they should be.

Sometimes because they are overengineered (in 2005 it would be 200 Java programmers building a EJB/SOAP/XML monster that requires 10 workstations to run), other times just because they can (charge more). The same way health related projects and vendors can charge a ton for trivial stuff (even plain plastic syringes).

In many countries, it's also because the bigger the project, the more greedy intermediaries can get a bite.

I don't understand the hate that's being directed at this article. This author isn't the person basically said, "LOL. I'll string together some python in an afternoon and do it." This author tried to replicate the "How I replicated an $86 million project in 57 lines of code" article, and unsurprisingly came to the conclusion, "No, he didn't."

Well, YouTube-encoded video is a severe mistake and it was his assumption that this would be representative of video out of a commercial camera.

It is not.

The camera is also not calibrated. Text recognition does not work well on video whose distortion characteristics are not known.

I am not at all surprised that this is his result.

This morning I calibrated my dashcam and used OpenALPR to replicate this, and I got FAR better results than this blogger.

At the end of a 30-minute section of heavy traffic, I had many hundreds of license plates with 95% confidence or better. I do not have a way to verify the plates automatically, so I don't know how many of those are actually correct.

I filmed at 2560x1440 at 30fps.

And all of this is true of the original boastful post, but yet no one was suspicious of those results.

can't tell if incompetence or malicious.

everyone knows dashcam, even on 1080p, simple can't be used fir license plates! humans can't read most of them. specially after youtube compression! this is such a known fact that the more serious youtube reviews show a warning about all that before dashcam reviews.

Often time the devil is in the detail.

OpenALPR has a premium service (http://www.openalpr.com/cloud-stream.html). Has anyone tried it?

I think this is a good case for why all government software should be open source.

How many police departments around the world are spending absurd amounts of money for this technology? It would make much more sense both cost and productivity wise to have one big project that anyone can improve on instead of multiple inferior projects.

Not to mention that all of this is paid for with taxpayer money, so it is reasonable for those same taxpayers to have access to the code.

Please join my campaign at http://www.oss4gov.org/manifesto - this is exactly why we want all government software to be open source

(Shameless plug)

i think there's some gov't contractors that survive off these inefficiencies. They will fight tooth and nail against OSS in gov't, and since they are often more connected, it's quite likely they can win.

Full disclosure: I'm one such contractor. We work with small and local government. Cities and counties mostly.

I personally don't have a problem with open source software and use it extensively outside of work. However, the big problem with it is technical support. If you are a government organization and the OSS you are using has a bug, who do you contact to get it hotfixed ASAP or find you workarounds?

That's one of the biggest advantages of commercial software: it is often sold with support contracts that entitle customers to certain levels of support, which can be critical to get things fixed and minimize downtime.

I know there are companies that focus on supporting OSS deployments, but from what I have seen in my own niche, they charge a ton of money that often surpasses what the equivalent commercial package would cost.

I think the key here is open sourcing your software that you are selling, so to a vendor, it's little different from supporting your own proprietary tech.

yea, good point. There's some misunderstanding about the issue - OSS in gov't could mean both adopting pre-existing OSS rather than contracting for bespoke software, and also to open source new/existing bespoke software as OSS (which makes sense, since it was public money that developed it).

Both are worthwhile goals.

> Full disclosure: I'm one such contractor. We work with small and local government. Cities and counties mostly.

...

>If you are a government organization and the OSS you are using has a bug, who do you contact to get it hotfixed ASAP or find you workarounds?

You, or someone like you, presumably?

That's a nice theory about OSS but it seldom works in practice. It can take a very long time for even experienced and skilled government contractors to learn enough about a code base to fix major defects. And even then there's no guarantee that the project maintainers will accept that patch into the mainline, so if they don't that puts you into configuration management hell. Governments need a "single neck to choke" along with some reliable assurances of support levels and time to resolution.

It seems to me that govt agencies could still pay contractors to build these things, just with the requirement to make them openly available as part of the contract.

If others step in to offer support contracts as well, great! Competitive environments for that would probably be good.

I wonder how much better it would work if either article specified the Australian training data that is included, instead I believe it is defaulting to the US set.

As for speed, there appears to be an OpenCL option that should be investigated.

Reply on news.ycombinator.com