AI Camera Ruins Soccer Game After Mistaking Referee's Bald Head for Ball
iflscience.com
iflscience.com
Background: I was one of the original engineers on the Goal Line Technology system now used by the vast majority of professional leagues.
Heads are surprisingly ball shaped from a a lot of camera angles, and the male balding pattern conspires to give a good impression of different football graphical designs.
When we were developing the system, we thought we had ball detection pretty nailed, and we never had false positives from any of the heads of engineers on the team. Then, our boss came to test the system ... As far as our algorithms were concerned, his head was exactly a football, just as in the article! Highly embarrassing, but everyone saw the funny side. It gave us better data and inspired a few more robustness checks that ended up being crucial later on! :)
Even looking like half a sphere is enough, as the system attempts to detect partially occluded balls too.
Btw, funny that my comment above attracted so much hatred. Didn't want to sound like a know-it-all but yes, at least in the case of the referee I guess that checking for a body would have helped.
For example, in this stylised picture we can easily tell the player's head from the ball:
https://www.pngitem.com/pimgs/m/80-803112_italia-italia-90-l...
Perhaps the camera operator's new work could be sitting in a comfy chair, tuning the areas of interest and motion profiles of a dozen cameras around the stadium to find the best angle on the action, allowing them to track smoothly under servos, pausing for a minute to think about the pacing of the game and what plays might develop, instead of mindlessly following the ball from place to place.
I'm an automation engineer, I've worked with hundreds of machine operators. They're almost universally happier after their employer gives them a better tool with which to do the work. The jobs that automated/AI equipment is replacing are not good jobs. Performing the same task for 90 minutes, much less an 8 hour shift, is awful on the body, and even worse on the mind. Humans don't make great robots, and it's not great to ask a human do a job that should be done by a robot.
By the way, is that your idea of an ideal job for everyone? Sitting in a "comfy chair", rather than doing, you know, the actual job of a cameraman, a job that is arguably an art form in itself? Do you understand that people love sports, people actually enjoy the athleticism of running around with cameras documenting sports, and that what you're dismissing would be a dream job for many people?
Do you really truly think the people buying this technology are doing it so that they can also continue to pay that same person to sit in a room and control it, while working towards self-actualization??
And do you think the appropriate response to concerns over jobs being eliminated is that the newly unemployed should feel grateful for being "freed" from the work they loved and took pride in, to go and do something "more fulfilling"? And add insult to injury by telling them that the job they had was just not a good job and wasn't something humans should have been doing in the first place?
> I'm an automation engineer, I've worked with hundreds of machine operators.
You mean, the ones that are keeping their jobs? Have you spent just as much time with the ones whose jobs have been eliminated?
I mean, I can understand that you may be facing a little cognitive dissonance here due to your job, but ... try to show a tiny bit of empathy for your fellow human. You think AI won't be able to do your job someday? Are you totally comfortable being dropped onto the labor market in today's economy if your current professional skill set became entirely valueless overnight, or is that something you only expect other people to be OK with?
We are recreating and eliminating entire industries, jobs, and traditions, and the level of awareness and respect for that disruption needs to dialed up by several orders of magnitude, or we as an industry, and all of us as a society, are in for some serious trouble.
A future where people dont have to stand by a conveyor belt or follow a ball with a camera to feed their families is a better vision of the future.
While the transition also presents an immense painpoint and must be handled with the utmost care. We should find ways to keep everyone in society and with an income ASAP.
Almost nobody chooses a cameraman, truck driver, or supermarket cashier jobs if they actually had a legitimate choice between that and a well paid tech job. But most people here have well paid tech jobs and a narrowed view of the issue.
Imagine taking that old food away from a poor person because they now have the possibility to go have a nice juicy steak.
The world beyond a chair in front of a computer is a lot more interesting to most people. I know more than a few in tech who dream of having a job where they get to interact with reality in a more direct way. And I guarantee you if the pay was closer, you'd see a lot more people adjusting their lifestyle and ditching the tech grind.
Indeed, dreams and ifs. It's no coincidence that the dreams that stay dreams are for the higher end jobs, and the ifs that stay ifs are for the lower end jobs.
Cameramen make good middle class to upper middle class incomes, have good benefits, are often unionized, and their jobs take skill.
I think this might be an example of tech bubble myopia.
I think you choose to see the tip of the pyramid in this particular "cameraman" case. You pick the few with the better conditions, rather than the many, many with the worse ones. It's pretty much a certainty that most cameramen in the world do not have upper middle class incomes, good benefits, or unions on their side yet their job is just as targeted by automation as the ones you think of. The couple or truckers I know making $200+k a year are also the exception rather than the rule.
Look, I know camera operators who are expected to process and edit the material and get paid their weight in gold. Most others are "just" camera operators and get paid barely middle class salaries.
The jobs that get automated are these second type because it's easier to train an algorithm or another human to do them than for many other jobs. It also means there's less room to be choosy. And people also learn to love what they do.
The experience from a European logistics company is that truck drivers love their jobs - travels (different countries), adventure and childhood dreams.
We just said goodbye to a camerman with 30+ years of experience, from Diana's crash in Paris, to the fall of Ghadaffi, interviewing multiple US presidents one day, and Rohingya fleeing into Bangladesh (much of which was caused by the tech industry - specifically facebook) the next.
And you think that's not a choice?
Everything is a choice in the end. But I also think that out of the hundreds of thousands of camera operators in the world you had to look for an exceptional one to make a point. Before you take it personal and form a general opinion based on that one anecdote sit on it for a minute and think, is this example relevant for the discussion? Is he the rule or the exception?
What about the situation from the article? The operator who's sitting in a small football stadium of a small Scottish second league team pointing the camera at what could be charitably described as an average football match for 90 minutes? How many operators filmed Diana's crash in Paris and the fall of Ghadaffi, and how many only film mundane, mediocre things their entire career?
> (much of which was caused by the tech industry - specifically facebook)
I took the tech industry just as a good example of an all round good, well paying job in general, not to pass judgement on what it brings to the world. My comment history could tell you as much. Mass media deserves its share of jabs as I'm sure you agree. But neither of the jabs make any my arguments or conclusion any less valid.
You might find you're not that good at it, you don't like standing out in the cold, but you might find you have a talent for it, and that leads onto a more formal training course
A colleague of mine was on a camera at the weekend, he's not a cameraman, he's an engineer, he had to be there anyway to set up the camera, plug in the various cables, and get the pictures out. He was filming 3 people stood at lecterns and panning the camera between them as appropriate, which meant making his own decisions based on the flow of the conversation, speed of the pan (I don't think there were any zooms needed), etc.
It's hardly thrilling work, and you could probably automate it. I've seen that type of automation system used for radio visualisation -- hell I've set it up on occasion, but AI isn't as good as an engineer with a little situational awareness pointing a camera, let alone a creative picture correspondent.
Then you have to take the next step and ask: why do they do it? If it’s tedious, unsatisfying, lacks prestige, and is utterly demoralizing, what possible reason could somebody have to do it? Either the worker is insane, or else...
Today would be analogical if we had a population of people with average IQ 150, locked in jobs that require IQ 100. By automating those jobs away, we would open new opportunities that would fit people better.
But instead, we have a world where the average IQ is 100, and the standard advice for people who lose their jobs is "learn to code". So we get the situation where some people are unable to find a job, and other people are pushed to work overtime... and we cannot balance it by hiring the former into job positions of the latter.
(Also, historically speaking, many people starved to death when their farms were taken away. The new jobs appeared later, but it took some time.)
Most jobs in the world are not good jobs, sure.
But unless you also take money from the capitalists and elite laborers enriched by automation to provide stronger social safety nets, automating away demand for low-skill crap jobs while making the fewer remaining jobs both better and higher paying means increasing the prevalence of the worst human misery. People might hate crap jobs, but not as much as they hate being qualified for no job in a capitalist society.
I've ran for the equivalent of Congress in my country, and even got elected as a substitute, and sent 3 draft bills for consideration.
It has better safety nets than the U.S.
I feel I'm doing my part, unfortunately my work is automating work in the U.S., not in my country.
And it has bite, you get your wages garnished if you didn't vote, unless you pay a hefty fine.
In reality that person probably turned to the gig economy, with no job security, benefits, or upwards mobility, and works twice as much to put less money into savings.
Sometimes these debates on automation end up in a fight over how the economy of Star Trek works when you have a machine that can materialize almost anything. That's an interesting rabbit hole to dive into when you ask if any random person could just ask for an Enterprise type starship to be instantly generated. What are the constraints in that economy that make it not possible? Energy?
Home and elder care may employ a lot of people in future that are assisted by robots.
Historically, yes, but I think that's a fatally flawed analysis this time around.
It's important to consider the minimum skill level required by the market. Not all jobs require the same level of mental acuity. To date, automation has consistently removed lower skilled jobs and replaced them with higher skilled ones.
There are obviously physical limits to that process and I am convinced that we are rapidly approaching them. Take a look through the present day cutting edge results in machine learning, remembering that today's cutting edge is tomorrow's par for the course.
I think that an equally likely outcome is that of the ATM. After the introduction of the ATM, the number of bank tellers in the US actually increased. Why? Because ATMs didn't completely eliminate the need for bank tellers, they just automated a lot of their most mundane tasks. However, ATMs did reduce the amount of tellers needed to operate a branch, which made it cheaper to open new branches. So in the end, the elimination of positions at existing branches was dwarfed by all the positions created at new bank branches that now could be operated profitably.
Weird, why that would happen?!... I don't think that's a common outcome elsewhere...
In Northern Europe, for example, there's almost no bank branches left now... they've been closing down for years and now it's actually hard to find any branch that's open to the public (this started with ATMs but now it's mostly a result of the cashless economy and online banking).
That looks suspiciously similar to the process of raising the skill floor that I referred to. The simpler parts of the job were relegated to a machine while the more complicated bits remained. My point is that there are inherent limits to such a process. What happened historically (including the ATM example) ceases to be relevant once such a limit is reached.
Relevant quote: "Instead of shooting where I was, you should have shot at where I was going to be." -Lrrr (to Fry)
(In the ATM example, it's specifically the bits that require vision, situational awareness, and human level conversational capabilities that were retained by the human. All things that cutting edge ML is making very visible progress on. I implore you to take a serious look at the state of cutting edge ML research with such tasks in mind.)
In my country, except janitors and people working cafeteria, pretty much any job in healthcare requires some kind of degree (exception is taking care of old folks in private homes, but even those require a substancial amount of staff with some kind of schooling in it)
https://www.healthcareers.nhs.uk/explore-roles/allied-health...
https://www.healthcareers.nhs.uk/explore-roles/wider-healthc...
I never claimed they were? I said that when a job is automated away, any that replace it inevitably require a higher skill level. Thus the skill floor rises over time.
> requiring human interaction
That's the key. That's a specific task that we can't (yet) automate.
My claim is that once we're able to automate away that job entirely, any that appear to replace it will almost invariably require a higher skill level.
Some of that volatility was women entering the workforce in large numbers, but when you consider every percent represents millions of people it’s shocking how long disruptions can be.
Instead we are using the best and brightest to solve 'hard' problems like netflix suggestions and self driving cars.
There's only so many bullshit paper-pushing jobs and useless over-budget infrastructure projects we can create to make up for the lack of work.
A better way to deal with too much labor on the market would be setting government-mandated maximum work-hours, so as to spread the amount of available work evenly among the entire populace.
Secondly - for those countries where this isn't a reality yet - school and university should be free and you should receive government aid while attending them, to allow those out of a job to learn new skills or just pursue their interests.
Isn’t infrastructure chronically underfunded?
https://www.infrastructurereportcard.org/wp-content/uploads/...
Both can be true at the same time. There's really no contradiction here, the opposite in fact.
Infrastructure being underfunded may very well be a symptom of infrastructure projects becoming way too expensive.
I think you’ll find many of those actually want to work hard will bristle at that policy. Not to mention finding that hard (and undesirable IMO) to enforce on entrepreneurs.
TLDR: Often after tech makes bureaucracy paperwork easier, some bureaucrats create more paperwork to compensate that, because they want more control over what is going on the office.
Stalling progress in the name of "saving the jerbs" never works in the long run. Better to pull off the band-aid quickly and find solutions that don't require humans to continue doing jobs that can be done by machines.
All you need is to have the system set up so that it can be remote controlled by somebody away from the camera, if needed.
Imagine we don't have parfocal lenses and you need multiple cameramen with all different fixed focus lenses filming different parts of the scene, or some other contrived example of less technology creating a need for more jobs. That would surely be a better world by your standard. Should we try to eliminate some technology to create extra jobs?
Where is the correct balance? Do we currently have not enough cameraman jobs or too many? It won't be exactly the right amount by coincidence.
It's too it's all a zero sum game and the saved money will just be hidden under a mattress and not spent on anything else.
That said I don't agree. If you can afford the technology for this type of camera and the connectivity to get it online, you can afford to pay someone £20 to point the camera for a couple of hours. It's a great opportunity for people still in school to start, and far more fulfilling than stacking shelves.
This seems to be the disrupted version.
But I don't think we need to worry until the AI says "This is pointless and I have more interesting things to do."
And it's just bound to happen at some point.
I think the ML specialists often forget there's a gap between pattern matching and solving a problem
The problem here is: locating a ball and moving the camera
If you just try to find a ball on static image by static image you're going to have a bad time. Even worse if you try to find only the ball but without the play contexts (kicking, flying, being held, etc)
Balls "teleport" but not always
Yes, a ball can be on top of a player, but not all the time
So, no, if you didn't see the ball go there, it didn't suddenly appear in the middle of the field (but then again, with some exceptions).
This isn't the point you were making, but as an aside, I think converting video into static images is throwing away useful information.
Most video codecs work by encoding the changes between video frames which includes movement information. It seems hard, but worthwhile, to use this information for object recognition.
I just did a search and saw that these techniques go by the name "compressed domain analysis" [1].
[1] https://scholar.google.com/scholar?hl=de&as_sdt=0%2C5&q=comp...
Can be solved adding a "too-sexy" list. The fake ball was always in the same part of the field. When the problem starts, a human should be allowed to enter a command ordering to ignore the area around coordinates X,Y of the grid. Putting a virtual black rectangle in the area only to be seen by the camera. Coordinates could be passed as arguments. When the real ball enter this area the camera of course will lost it, but will quickly relocate it at the other side so is not much of a problem.
Their training data must have been seriously lacking.
But in defence of the software: this was a sideline ref, so probably a different kind of behaviour from the main one.
This system, to me, looks like it works on individual video frames (the camera seems to move away whenever the ball is even half-covered by a player’s body, or hard to see looking against the sun). A smarter system would know the ball doesn’t move 20 meters in a single frame, and tends to move faster than other objects on the field. That may not trivial to implement, but, I expect, would lead to a much more robust system.
It’s not like AI engineers manually run all varieties of footage against the trained model to check accuracy.
https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
Note the irony here: what people don't want is the dystopia, but what's stopping it now is that they find barcoding people creepy. They won't notice the AI tech until it's too late.
On a broader note, this pattern of rounding a square peg to fit if we can't square the round hole - it's what we do. All the time, everywhere. For instance, designing a reliable all-terrain vehicle is something that technologically escapes now even to this day, but thousands of years ago our ancestors figured out that if they flatten and pave the terrain, then a simple wheel would do.
It’s not hard to fix your software that you created. It’s much harder to get global consensus on a change that wasn’t required before your buggy code was released into the wild.
A few companies actually managed to get a production process that put a chip in the ball without affecting flight characteristics (there are a lot of regulations around this), and these balls were used in competition once (Club World Cup 2012).
Ironically, the most complicated bit is the sensors in the posts.
All GLT is Computer Vision based now.
my understanding is that QR works way better for moving images
AI for controlling a football camera, sure, move fast and break things. Why not.
AI for controlling a moving motor vehicle, maybe we should be extremely cautious about our testing and use of AI.
No you wouldn’t. But it’s not.
“Move fast, break things” is perfectly ok for certain applications, such as recreational AI. Nobody said it’s ok for medical or military applications.
You can live in a world where the bar for quality control varies depending on the application.
Experimentation will never happen if we set medical/military grade expectations across the board.
We often hear that the AI is more accurate than humans in some image matching, but it also clearly makes mistakes that no human would make. Well here is a great example of such a mistake which a human wouldn't be making.
I can see a pattern. It helps if you think strictly in terms of what visual feature a ball and a bald head have in common: a crescent of glare against the sun.
* The ball is obscured by legs, shadow, or a player's body (this appears to be majority of the occurrences to me) - this one seems to trigger even if it's only out of view for a fraction of a second
* The high kick at 2:07 is similar to the above, except it's the glare from the sun instead of shadow that obscures the ball (ambient brightness matches the crescent instead of the umbra on the ball)
* Most of the rest I can't even find the ball myself, I think it might be too small in the far end of the field to register clearly
I think it is a testament to not just human complexity, but human efficiency.
Yes, the term has become accepted, but it's a horribly irresponsible label because it massively misleads laypeople about how these systems work and how trustworthy they are.
Technically AI is correct, since AI doesn't necessarily refer to human level AGI. Unfortunately it's highly misleading because that distinction seems to be lost on the average person.
Machine learning doesn't seem to carry the same science fiction implications for whatever reason.
Time: False. Time can be included as an input. Previous system state can also be communicated forward in various ways (ex the LSTM architecture).
Conceptual context: I'll agree, since that seems like it would require general abstract reasoning ability (ie strong AI). Similarly, robust treatment in the general case of physical continuity and cause and effect both seem like special cases of conceptual context to me.
That being said, bear in mind that we already have examples of not so robust treatment for special cases of physical continuity as well as cause and effect. (Example: https://news.developer.nvidia.com/transforming-standard-vide...) Who knows how close (or far) we might be from a general solution?
I mean that they don't understand time as something that's unidirectional. (To the best of my knowledge! Would be happy to be proven wrong.)
But I don't see any reason you couldn't train a model to incorporate the underlying assumption that the flow of time is unidirectional. I expect you would just need inputs (ie training data) reflecting that fact coupled with an appropriate loss function.
(Aren't networks that predict future physical states, such as motion, more or less doing this?)
https://en.m.wikipedia.org/wiki/FoxTrax
I suspect in this instance it wasn't just the bald head, but the particular near-sunset lighting that was a contributing factor.
Great example of first iteration tech being the tip of the iceberg.
I can't imagine that the training and maintenance of such a system could be cheaper than the pay of a single camera person and I don't remember ordinary camera people having much trouble following the ball in soccer.
Which is to say, while current ML/AI may work well for some things, for other things, it will be abandoned as a fad unless there's some fundamental improvement.
Many countries have restrictions on games. No fans allowed and the personal supporting a game has to be reduced to an unsustainable low minimum with a very high-effort hygienic concept.
This may sound strange for americans but is the current norm in most countries. So if you are allowed less and less people as covid-19 gets worse you are in search of solutions. And as there are many camera men to record the game from different angles why not comply with these restrictions by replacing them with ai...
Eventually it will get better and cheaper than a human operator.
But it's like that with lots of things, for example humans.
It's like looking at a kid, and saying "it cannot even walk, it will never become a good cameraman".
What happens if there are multiple balls? What happens if the ball is redesigned with a different pattern? What happens if someone wears a shirt with a picture of a ball?
Does it make sense to center the camera on the ball? Does it make sense to also show where the ball can go?
I wonder how it's handled by people currently? I suppose it can be handled with more training data/labelling instead of custom coding.
We are looking are replacing the camera person job with AI just as we replaced other people's jobs with programming earlier and just as we replaced other people's jobs with AI more recently.
There were loads more, but that was off the top of my head. I dont think we ever mistake a head for a ball though! We worked out likely balls from movement (syncing cameras with bounces)
We have since then developed ways of analysing the inner workings of neural networks.
You can understand how each feature drives the predicted result by using SHAP https://github.com/slundberg/shap
You can analyse the individual layers and gain an understanding of how each layer encodes the input
https://ai.googleblog.com/2017/11/interpreting-deep-neural-n...
https://www.kdnuggets.com/2019/07/google-technique-understan...
Do we 'know' in an absolute way why a network thinks something? Not yet. But neither do we know that for humans. We just have a huge amount of experience with their failure modes and we work around them (see aviation).
(Or maybe you won't even need to rent the camera, if you use the free version supplied by Google which automatically adds personalized sponsorship visuals)
(Or the Facebook version, automatically posting 'hilarious' and embarrassing snippets designed to maximize user engagement)
Latency: (cheap labour is usually far away from the match). This makes teleoperation harder.
Bandwidth (TV ops run from a van in a car park - so getting a reliable video feed out is expensive).
Logistics: you still need to hire, train and utilise that person effectively.
Reliability: what if your remote operator's internet goes down?
At least they can wear hats. If you are black and rely on face detection you are out of luck.
I mean how would it be for a AI to take in a score and output "they played well"...
https://techcrunch.com/2010/11/12/automated-news-sports-stat... https://en.wikipedia.org/wiki/Automated_Insights
“Did you see that ludicrous display last night? ...”
Science is made better mistake after mistake. Better in a football game than in a shuttle, or a law enforcement AI.