Controllable Video Sprites That Appear Like Professional Tennis Players
cs.stanford.edu
cs.stanford.edu
Their models actually do a decent job of replicating true tennis strategy, and as they pointed out, even account for the quirks like the left handedness of Nadal.
However, it's still a bit unrealistic due to the lack of full data.
There's 3 things that make a tennis shot what it is: placement (covered in the video), pace (speed of ball), and spin (rpm and direction of spin). In this method, they only use placement. Probably because pace and spin data don't exist at this scale.
But there's a big difference between a slice, flat, and top spin shot to the same placement on the court, and it directly affects the return shot. For example, it's a very common and 'safe' play to return a slice with a slice
Would like to see the full extension one day
Certainly it does pace. But from the article it's hard to say if it would track spin - it's not clear to me that with the natural markings on the ball (seam lines + ballmaker logo) and the lighting that exists in the stadium, that would be enough fidelity for some algorithm to calculate the spin.
I think Hawkeye could be updated to easily do it though, and maybe if they would be willing to draw 1 or 2 black dot markers on the ball in addition to the natural markings.
Additionally spin is a big part of the trajectory model - the Hawkeye cameras are only 60fps, so the trajectory is interpolated.
However, I doubt it’s the cameras measuring spin. It’s more likely spin is a free variable when they fit the trajectory.
Very interesting read, but I'm not knowledgeable enough to really give you an accurate tldr, sorry
Certain smart racquets (I think Babolat's?) can track the rpm and spin direction based on the head movement. I think using this type of data too could make the difference in terms of realism.
edit: Well, perhaps not entirely. Trajectory is a function of a number of factors including spin. However, the ball itself is a factor. More fuzz = more drag = spin has a greater effect on trajectory. That would vary from ball to ball and even over the lifetime of an individual ball.
--
For those not familiar with tennis:
Primarily because of aerodynamic drag created by the fuzz on tennis balls, the ball's spin greatly affects its trajectory.
A topspin makes the ball dive down more sharply. This is how players can hit the ball extremely hard and fast, yet still land in bounds, as opposed to flying out of bounds.
This "trick" of topspin is also why tennis is easier than it may first seem if you've ever tried it. It's not easy to learn topspin but once you do, it increases your margin for error.
_______________
[1] Quoting from Section 10:
Finally, our work makes extensive use of domain knowledge oftennis to generate realistic results. This includes the shot cycle statemachine to structure point synthesis, the choice of shot selectionand player court positioning outputs of player behavior models,and the choice of input features provided to these behavior models.
A successful behavioural model of that kind is a contribution in and of itself, useful beyond the task of simulation presented in the paper.
technology really repeats itself
But of course Mortal Kombat's sprites were not being generated in real-time through a flexible machine learning model.
The MK analogy seems spot on.
How long before it becomes a plausible criminal defence to say “the CCTV must have been deepfaked, that’s not me”?
The indicator of reliability will be the chain of custody of the video data.
Assume the secure element is in fact secure, the issue then, as with any public/private key scheme, lies with establishing trust of the keypair. Do you trust the manufacturer that he will not be breached?
And more pressing even; How do you prevent someone modifying their internal camera video stream such that they may send any data to the authentication chip/mechanism?
And if all this is implemented, it can be done even more low level - just direct the camera e.g. to your screen (I assume solutions would then crop up to increase the fidelity of such a solution).
I am not saying such a solution would not provide ANY benefits, I am just pointing out that these issues prevent it from becoming a silver bullet.
Less about the algorithms, but more about designing applications of cryptographic primitives?
However the secret signing key will have to reside inside the camera in a way that a determined attacker cannot extract it. Sounds hard.
So you could still fake videos, but any accusation of tampering could be verified by checking the seal is intact. (of course you need non-forgeable seals too, but that's comparatively easier)
A simpler detection mechanism would be to use stereoscopic (or rather real depth detection) cameras though.
That's not to say it shouldn't be researched, or people more creative than me won't think of beneficial use cases.
From the abstract: "Our system can generate novel points between professional tennis players that resemble Wimbledon broadcasts, enabling new experiences such as the creation of matchups between players that have not competed in real life, or interactive control of players in the Wimbledon final."
I don't believe academic researchers needs to justify their work by providing real life applications. But if that were the extent of deepfake's utility, I'd be underwhelmed.
I recently lost my dog. The idea of interacting with a virtual model of her to help ease the grief is interesting and scary, and ripped straight from a Black Mirror episode.
A mother meeting her dead daughter again through VR. It's a video that fills me with all sorts of emotions. It feels so wrong but at the same time it is very touching and beautiful.
I wonder if you could use some of the recent advances in pose estimation to rig a 3D model of each player rather than the rotoscope look of clipped frames in the demo.
It's been done.[1]
If so, this could be expanded to other sports, maybe even team sports, where you can test set plays against the simulated defense.
Simulated match ups have been done on pro sports video games for years. But they're not that useful because a model is not reality.
* Hard to find all replays against that shot
* Time consuming to review
This would work as compression over that space. However, I don't watch tennis and know nothing about sport. This may be an insignificant improvement.
If so, this could be expanded to other sports, maybe even team sports, where you can test set plays against the simulated defense.
The challenge is that in sports, the opponent you face on a given day isn't "the statistical average of their past performances" - they are facing you with a game plan tailored specifically for them versus you, and their game plan will evolve over the course of the contest depending on what's working and what isn't working.
For example, "Nadal likes to hit the ball to Federer's backhand" is, statistically, true. It's basic tennis strategy. But the on-court reality is more nuanced. Nadal is going to vary that approach on the fly based on his opponent and how well that strategy is working on a given day.
Modeling this for a simulation would have to be similarly nuanced, with the simulation not just replicating Nadal's overall statistical tendencies, but how those tendencies evolve over the course of a match based on various conditions and his success or lack of it.
Of course, some aspects of Nadal's game are more easily modeled than others. If an opponent was training to face Nadal on a clay surface, I could simulate that with a single line of code: "Game Over." =)
It was somehow very soothing to watch that video. It felt like someone was telling me a bed time story about a different brighter future that never came to be. Very "Back to the Future".
[1] https://en.wikipedia.org/wiki/Time_Traveler_(video_game)
https://en.m.wikipedia.org/wiki/Holosseum
I remember my local AMC 6-plex had one.
The data might be just as “productive” as a spreadsheet or formula to inform play, but it requires someone with a more specialized skill set to translate its meaning. The HCI design, for lack of a better word, in rendering the data visually makes it not only more entertaining but easier to "see"—for mainstream users, pro players, or almost anyone. Design makes things visible.
I even started with Python & OpenCV for basic background extraction, and as I expected, the edges for the players are imperfect. But still, the result is very very promising. I'm so glad someone did it.
Why I was thinking at it is because the end goal would be to apply frame matching & transitioning to football. Tennis is the easier task, the camera angle is almost fixed.
But from this result, to football, we aren't far. Even an approach based on AI + some human intervention would suffice.