Thanks for sharing, @yeldarb! Your blog is even more interesting to me than the OP.
related: I had an idea to make a CLIP paint.NET image editing plugin. You would make a new layer, add some green ('towards') and red ('away from') splotches to the layer, and give a prompt to CLIP. Wherever there was green, CLIP would 'nudge' the image along the direction of the gradients towards the semantic goal, keep it constant where there was nothing, and nudge it in the opposite direction where there was red. And you could do this over and over again, w/ different CLIP prompts all the while editing the semantic masks as you like. The end result would be a feedback process to use CLIP + feedback as an extremely creative and controllable image manipulation tool. But then I realized that I didn't know how to make a paint.NET plugin :) so here is a free idea for anybody who does
It doesn't use CLIP but check out NVIDIA's Canvas[1] (powered by GauGAN[2] which has a pretty poor UI but is usable in your browser).
Yep, seen Canvas - probably where I Picasso'd (read: stole :^) ) the original idea! The nice thing about making it a PDN plugin based on CLIP is that it could plug into the existing PDN plugin system, PDN is pretty solid image editing software, and also CLIP could make it extremely versatile - no more limitation to primarily landscapes - including the fact that there are newer and vastly better CLIP tools coming out, e.g. diffusion model based CLIP - but I don't think I had actually seen the GauGAN UI before, thank you!
Like most people, I've got piles of old photos that I'd like to be able to label who is who in the person. For some family photos, I've doodled sketches of the photo, labelled the sketch, and attached it to the back. But I've often thought there had to be a way to get the computer to take the photo and generate the sketch from it. This seems exactly like it!