Stitch It in Time: GAN-Based Facial Editing of Real Videos
stitch-time.github.io
stitch-time.github.io
Since I read that years ago, I think its more likely that we will simply get a bombshell article a few years in the future that its been in use already for a very long time! Similar to how Cambridge Analytica information and other information warping info came out years after it was already in play on a broad scale.
I am not aware of any singular FOSS project in creative work that performs similarly to Topaz Labs’ product for all input videos (note to reader: if I am wrong, correct me— I only have second-hand anecdotes to go off of from my friends who work with this kind of work in their careers, and precious little first-hand experience from upscaling old family memories to experiment with the software). As far as I am aware, this is because different upscaling models are trained on, and thus effective / mainly used for a specific type of content. The same upscaling model that was trained on interpolated 480p videos with compression artifacts will not produce the same results as one trained on, say, anime videos/manga, e.g. the models used by Waifu2X [1]. Hence, with Topaz Labs’ application, you select the model that was trained on footage that best matches the footage you wish to upscale.
All that being said, I do know that some (if not all) of the upscaling models Topaz uses are FOSS. Much of their application is just syntactical sugar upon the models it uses, making it easier for non-SWEs to use. I’m not sure whether or not the models that Topaz distributes have any level of training done by them, in house— logically, I would assume as such, otherwise their product wouldn’t be as performant as it is.
However, the examples I saw hit the uncanny valley for me. The changes are realistic but there is just something off about them, that bothers my brain.
(the details of obtaining the marker aren't relevant, I think you'll agree that one way or another there will be some leaks or cracked phones or any one of other possibilities eventually exposing biomarkers for at least some of the population)
Furthermore, there's the catering to the least denominator. Whatever signing ability is available for a random widespread cheap third world smartphone used in Tik Tok videos will be treated as sufficiently good to assume it to be true; so if a determined attacker wanting to create a misinformation campaign with fake videos can circumvent the security of that signing process (e.g. get a bunch of valid keys indistinguishable from these phones, then fake videos will have as good signing as real videos.
Furthermore, any attacker with the desire and resources to create a disinformation campaign can simply recruit a new real person to sign each deepfake campaign, just as criminals now hire money laundering mules (I mean, that expense would be less than the actual effort of creating the media) and any intelligence agency using deepfakes for propaganda can literally create new valid identities by fiat (just issue new real passports/birth certificates/whatever for nonexistent people) that are indistinguishable from real people as far as google or anyone else abroad can verify.
In essence, what you describe would work if and only if we had a global, trusted database of all people worldwide that doesn't allow for fake people. We're very far from that, and the obstacles for that aren't technological, it's definitely not something that Google can solve.