Hypothetically, with enough information, one could predict the future (barring truly random events like radioactive decay). Generative AI is also constrained by economic forces - how much are GenAI companies willing to invest to get eyeball reflections right? Would they earn adequate revenue to cover the increase in costs to justify that feature? There are plenty of things that humanity can technically achieve, that don't get done because the incentives are not aligned- for instance, there is enough food grown to feed every human on earth and the technology to transport it, and yet we have hunger, malnutrition and famines.
Willing to? Probably not much. Should? A WHOLE LOT. It is the whole enchilada.
While this might not seem like a big issue and truthfully most people don't notice, getting this right (consistently) requires getting a lot more right. It doesn't require the model knowing physics (because every training sample face will have realistic lighting). But what underlines this issue is the model understanding subtleties. No model to date accomplishes this. From image generators to language generators (LLMs). There is a pareto efficiency issue here too. Remember that it is magnitudes easier to get a model to be "80% correct" than to be "90% correct".
But recall that the devil is in the details. We live in a complex world, and what that means is that the subtleties matter. The world is (mathematically) chaotic, so small things have big effects. You should start solving problems not worrying about these, but eventually you need to move into tackling these problems. If you don't, you'll just generate enshitification. In fact, I'd argue that the difference between an amateur and an expert is knowledge of subtleties and nuance. This is both why amateurs can trick themselves into thinking they're more expert than they are and why experts can recognize when talking to other experts (I remember a thread a while ago where many people were shocked about how most industries don't give tests or whiteboard problems when interviewing candidates and how hiring managers can identify good hires from bad ones).
They won’t train a better model just for that reason. It will just happen along the way as they seek to broadly improve performance and usefulness.
This isn't how it works. As the models are improved, they learn more about reality largely on their own. Except for glaringly obvious problems (like hands, deformed limbs, etc) the improvements are really just giving the models techniques for more accurately replicating features from reasoning data. There's nobody that's like "today we're working on fingernails" or "today we're making hair physics work better": it's about making the model understand and replicate the features already present in the training dataset.
AI models aren't complete blackboxes to the people who develop them: there is careful thought behind the architecture, dataset selection and model evaluation. Assuming that you can take an existing model and simply throw more compute at it will automatically result in higher fidelity illumination modeling takes almost religious levels of faith. If moar hardware is all you need, Nvidia would have the best models in every category right now. Perhaps someone ought to write the sequel to Fred Brooks' book amd name it "The Mythical GPU-Cluster-Month".
FWIW, Google has AI-based illumination adjustment in Google Photos where one can add virtual lights - so specialized models for lighting already exist. However, I'm very cynical about a generic mixed model incidentally gaining those capabilities without specific training for it. When dealing with exponential requirements (training data, training time, GPUs, model weight size), you'll run out of resources in short order.
Nvidia is making boatloads of money right now selling GPUs to companies that think they will be making boatloads of money in the future.
Nvidia has the better end of things at this very moment in time.
Seems an odd response to a poster who said “as the models are improved...”; the way the models are improved isn't just additional training to existing models, its updated model architectures.
The issue is practical. There isn’t enough data out there to learn the long tail. If neural nets genuinely understood the world they would be getting 100% on ARC.
Even if we added no new compute and capped the resources used for inference and training, models would produce higher fidelity results over time due to improvements in the architecture.
Normally is being strained here: yes, most finetuning isn't for things like this, but quite a substantial minority is for more accurate rendering of some narrow specific feature, especially ones typically identified as signature problems of AI image gen; publicly identifying this as a way to visually distinguish AI gens makes it more likely that fine tuning effort will be directed at addressing it.
I'm an engineer at an AI company that makes text/image to video models, but go off
I especially agree with the last sentence that models largely learn features in the dataset, but I don't understand why you would describe it as
> There's nobody that's like "today we're working on fingernails" or "today we're making hair physics work better"
If there were a business case for that, I would characterize curating a dataset of fingernails and fine-tuning or augmentind a model based on that as "today we're working on fingernails".
And the same too with eye reflections. So with the right dataset you can get eye reflections right, albeit in a limited domain. (E.g. deepfakes in a similar setting as the training data). In fact you can look at the community that sprung up around SD 1.5 (?) that fine tunes SD with relevant datasets to improve its abilities in exactly a "today we're going to improve its ability to produce these faces" kind of fashion.
Where did I misunderstand your comment? I seem to arrive at the completely opposite response from the same fact.
I also noticed that you say
> the improvements are really just giving the models techniques for more accurately replicating features from reasoning data.
You seem to refer to aspects of a model unrelated to dataset quality. But fine-tuning on a curated dataset may be sufficient and necessary for improving eye reflections and fingernails.
https://en.wikipedia.org/wiki/The_Poverty_of_Historicism
"Individual human action or reaction can never be predicted with certainty, therefore neither can the future"
See the death of Archduke Franz Ferdinand - perhaps could be predicted when it was known that he would go to Sarajevo. But before?
If you look at SciFi, some things have been predicted, but many -obvious things- haven't.
What if Trump had been killed?
And Kennedy?
This all reminds me of “fixing” mis-architected software by adding extra conditional code for every special case that is discovered to work incorrectly, instead of fixing the architecture (because no one understands it).
Yes, the models output will converge to being congruent with laws of physics by virtue of deriving that as a latent variable.
I would assume that larger models working with additional training data will eventually allow them to understand physics to the same extent as humans inspecting the world - i.e. to capture what we call Naive Physics [0]. But the limit isn't there; the next generation of GenAI could model the whole scene and then render it with ray tracing (no special casing needed).
They will all do this with a fixed seed. They just don't do that because nobody wants it.
Hinton argues that the easiest way to minimise loss in next token prediction is to actually understand meaning. An analogous thing may hold true in vision modelling wrt physics.
I guess it really depends on what the meaning of gradient decent learning the physics is.
Maybe you define it to mean that the actually correct equations appear encoded in the computation of the net. But this would still be tacit knowledge. It would be kind of like a math software being aware of physics at best.
Human innate understanding of physics is a laundry list of superficial hacks. People needs education and mental effort to go beyond that innate but limited understanding.
What people mean when saying people innately understand physics is that they have a working knowledge of many of the implications. Things like that gravity is uniformly applied from a single direction and that is the direction towards ground. That objects move in arcs or "ballistic trajectories", that straight lines are uncommon, that wires hang with hyperbolic function shapes even if they don't know that word, that snow is created from cold, that the sun creates heat, many lighting effects (which is how we also form many illusions), and so on.
Essentially, humans know that things do not fall up. One could argue that this is based on a "laundry list of superficial hacks" and they wouldn't be wrong, but they also wouldn't be right. Even when wrong, the human formulations are (more often than not) causally formulated. That is, explainable _and_ rational (rational does not mean correct, but that it follows some logic. The logic doesn't need to be right. In fact, no logic is, just some are less wrong than others).
Basic things that are not accessible to such a learning process:
- moving around to get a better view of a 3d object
- see actual motion
- measure the mass of an object participating in an interaction
- set up an experiment and measure its outcomes
- choose to look at a particular sample at a closer resolution (e.g. microscopy)
- see what's out of frame from a given image
I think we have at this point a lot of evidence that optimizing models to understand distributions of images is not the same thing as understanding the things in those images. In 2013 that was 'DeepDream' dog worms, in 2018 that was "this person does not exist" portraits where people's garments or hair or jewelry fused together or merged with their background. In 2022 it was diffusion images of people with too many fingers, or whose hands melted together if you asked for people shaking hands. In the Sora announcement earlier this year it was a woman's jacket morphing while the shot zoomed into her face.
I think in the same way that LLMs do better at some reasoning tasks by generating a program to produce the answer, I suspect models which are trained to generate 3D geometry and scenes, and run a simulation -> renderer -> style transfer process may end up being the better way to get to image models that "know" about physics.
My guess is that the models will come up with much more interesting and fruitful training sets than what a bunch of researchers can come up with.
But still:
- input doesn't distinguish what's real vs constructed nonphysical motion (e.g. animations, moving title cards, etc)
- input doesn't distinguish what's motion of the camera versus motion of portrayed objects
- input doesn't distinguish what changes are unnatural filmic techniques (e.g. change of shot, fade-in/out) vs what are in footage
Some years ago, I saw a series of results about GANs for image completion, and they had an accidental property of trying to add points of interest. If you showed it the left half of a photo of just the ocean, horizon and sky, and asked for the right half, it would try to put a boat, or an island, because generally people don't take and publish images of just the empty ocean -- though most chunks of the horizon probably are quite empty. The distribution on images is not like reality.
The latter is always easier. Not to mention that the architectures are fundamentally curve fitters. There are many curves that can fit data, but not all curves are casually related to data. The history of physics itself is a history of becoming less wrong and many of the early attempts at problems (which you probably never learned about fwiw) were pretty hacky approximations.
> Hinton argues
Hinton is only partially correct. It entirely depends on the conditions of your optimization. If you're trying to generalize and understand causality, then yes, this is without a doubt true. But models don't train like this and most research is not pursuing these (still unknown) directions. So if we aren't conditioning our model on those aspects, then consider how many parameters they have (and aspects like superposition). Without a doubt the "superficial hacks" are a lot easier and will very likely lead to better predictions on the training data (and likely test data).
Yes, this is out of reach of how we train most models today. But it demonstrates how even current models are capable of building circuits that perfectly predict (meaning understand the actual dynamics) of data given sufficient exposure.
It is important to remember that there are plenty of alternative explanations to why the "sudden increase" in performance happens. I believe if people had a deeper understanding of how metrics work that the phenomena would become less surprising and make one less convinced that scale (of data and/or model) will be insufficient to create general intelligence. But this does take quite a bit of advanced education (that is atypical from a ML PhD) and you're going to struggle to obtain it "in a few weekends".
Truth and reality cluster.
So hyperdimensional data compression which is organized around truthful modeling versus a collection of approximations will, as complexity and dimensionality approach uncapped limits, be increasingly more efficient.
We've already seen toy models do world modeling far beyond what was being expected at the time.
This is a trend likely to continue as people underestimate modeling advantages.
> doesn't actually calculate rays
> without having to understand the physics acting on it
Isn't that what AI training is in general? It has worked pretty well so far.
I dont think img-gen AI is ever going to "understand physics", but that isn't the task at hand. I don't think it is neccesary to understand physics to make good fake pictures. For that matter, i dont think understanding physics would even be a good approach to the fake picture problem.
They don't have to. They just have to understand what makes a realistic picture. The author of the article isn't really employing physics either; he's comparing the eyes to each other.
If you want to correct for this particular problem you might be better off training a face detector, an eye detector and a model that takes two eyes as input and corrects for this problem. Process then would be:
- generate image
- detect faces
- detect eyes in each face
- correct reflections in eyes
That is convoluted, though, and would get very convoluted when you want to correct for multiple such issues. It also might be problematic in handling faces with glass eyes, but you could try to ‘detect’ those with a model that is trained on the prompt.
The opposite might also be true. Just having better, well curated data goes a long way. LAION worked for a long time because it's huge, but what if all the garbage images were filtered out and the annotations were better?
The early generations of image and video models used middling data because it was the only data. Since then, literally everyone with data has been working their butts off to get it cleaned up to make the next generation better.
Better data, more intricate models, and improvements to the underlying infrastructure could mean these sorts of "improvements" come mostly "for free".
There’s no eyes module in it by default, but it’s trivial-ish to add, and a hires eyes dataset isn’t hard to collect either.
Just found eyes model on https://civitai.com/models/150925/eyes-detection-adetailer (seems anime only)
I can remember when it was hard for image generators to model the 3d shape of a table, now they can very easily display 4 very convincing legs.
I don't have technical expertise here but it just seems like a natural starting point to assume that this reflection thing is a transient shortcomimg.
Can you make a napkin calculation of how much better training should be, how larger models and datasets should be to overcome difference in widely separated but related pixels of the image?
I did that and results are not promising: 200 billions parameter model will not perform much better than 100 billions' one. The loss is already too small.
Also, the phenomena exemplified in the article is a problem of relation between distant entities in generated media. The same can be seen in LLM where they have inconsistencies at the beginning and end of sentences.
If generated eyes start to lie, there will be different separated but related objects to rely on.
Current image generators work by training models to remove artificial noise added to the training set. Take an image, add some amount of noise, and feed it with it's description as inputs to your model. The closest the output is to the original image, the highest the reward function.
Using some tricks (a big one is training simultaneously on large and small amounts of noise), you ultimately get a model that can remove 99% noise based only on the description you feed it, and that means you can just swap out the description for what you want the model to generate and feed it pure noise, and it'll do a good job.
> Better training, larger models, and larger datasets
But "better training" here is a special measure. It would take a lot of training effort to defeat this check. For example, you'd need a program or group of people who would be able to label training data as realistic/not based on the laws of physics as reflected in subjects' eyeballs.
That's not guaranteed, AI does find statistical regularities we miss but also miss some we find.
Hi, author here of a model that does really good on this[0]. My model is SOTA and has undergone a third party user study that shows it generates convincing images of faces[1]. AND my undergrad is in physics. I'm not saying this to brag, I'm giving my credentials. That I have deep knowledge in both generating realistic human faces and in physics. I've seen hundreds of thousands of generated faces from many different models and architectures.
I can assure you, these models don't know physics. What you're seeing is the result of attention. Go ahead and skip the front matter in my paper and go look at the appendix where I show attention maps and go through artifacts.
Yes, the work is GANs, but the same principles apply to diffusion models. Just diffusion models are typically MUCH bigger and have way more training data (sure, I had access to an A100 node at the time, but even one node makes you GPU poor these days. So best to explore on GANs ):
I'll point out flaws in images in my paper, but remember that these fool people and you're now primed to see errors, and if you continue reading you'll be even further informed. In Figures 8-10 you can see the "stars" that the article talks about. You'll see mine does a lot better. But the artifact exists in all images. You can also see these errors in all of the images in the header, but they are much harder to see. But I did embed the images as large as I could into the paper, so you can zoom in quite a bit.
Now there are ways to detect deep fakes pretty readily, but it does take an expert eye. These aren't the days of StyleGAN-2 where monsters are common (well... at least on GANs and diffusion is getting there). Each model and architecture has a different unique signature but there are key things that you can look for if you want to get better at this. Here's things that I look for, and I've used these to identify real world fake profiles and you will see them across Twitter and elsewhere:
- Eyes: Eyes are complex in humans with lots of texture. Look for "stars" (inconsistent lighting), pupil dilation, pupil shape, heterochromia (can be subtle see Figure 2, last row, column 2 for example), and the texture of the iris. And also make sure to look at the edge of eyes (Figs 8-10) and
- Glasses: look for aberrations, inconsistent lighting/reflections, and pay very close attention to the edges where new textures can be created
- Necks: These are just never right. The skin wrinkles, shape, angles, etc
- Ears: These always lose detail (as seen in TFA and my paper), lose symmetry in shape, are often not lit correctly, if there are earrings then watch for the same things too (see TFA).
- Hair: Dear fucking god, it is always the hair. But I think most people might not notice this at first. If you're having trouble, start by looking at the strands. Start with Figure 8. Patches are weird, color changes, texture, direction, and more. Then try Fig 9 and TFA.
- Backgrounds: I make a joke that the best indicator to determine if you have a good quality image is how much it looks like a LinkedIn headshot. I have yet to see a generated photo that has things happening in the background that do not have errors. Both long-range and local. Look at my header image with care and look at the bottom image in row 2 (which is pretty good but has errors), row 2 column 4, and even row 1 in column 4's shadow doesn't make sense.
- Phase Artifacts: This one is discussed back in StyleGAN2 paper (Fig 6). These are still common today.
- Skin texture: Without fail, unrealistic textures are created on faces. These are hard to use in the wild though because you're typically seeing a compressed image and that creates artifacts too and you frequently need to zoom to see. They can be more apparent with post processing though.
There's more, but all of these are a result of models not knowing physics. If you are just scrolling through Twitter you won't notice many of these issues. But if you slow down and study an image, they become apparent. If you practice looking, you'll quickly learn to find the errors with little effort. I can be more specific about model differences but this comment is already too long. I can also go into detail about how we can't determine these errors from our metrics, but that's a whole other lengthy comment.
When there are no more tricks remaining, I think we must be pretty close to AGI.