Neural networks need data to learn, even if it’s fake

I think "giving the rules we already know" is only a good idea if we are 100% sure that we know 100% of the rules necessary with 100% accuracy. The fuzzyness of NNs is what in many cases allows them to work as well as they do, and training on close-to-perfect data is going to preserve this aspect better than close-to-perfect rules could.

What you’re describing would be an Expert System, part of GOFAI.

Neural nets work by generalizing. This is important because GOFAI systems are brittle. We are never able to encode all the relevant rules for real world situations by hand.

Also, note that the synthetic data is generated from rules about how the world looks, not rules about what the model should do.

There may still be some direction for zapping the rules-of-the-world right into the circuitry of, say, a subnet of the model, without doing backprop. Something like a one-shot initialization of the network to approximate the given distribution.

We don't have the rules that we want the neural network to learn, if we did we could just directly use those rules, and there would be no need for ML to solve the problem. In this case we want the ML to learn how to infer spatial information from two dimensional images, and the process that generates the data it trains on cannot do this at all. It can create two dimensional images from spatial information, which is a much simpler and effectively solved process.

There are cases when we want a machine learning model to do the same thing as the process which generate it's data, like in the case of model's learning to replicate physics simulations, but even then the entire point is for the machine learning model to accomplish the same or a similar result but in a more computationally efficient way.

With (imho closely related) bayesian statistics you can add those rules into your model, the problem is that it quickly becomes computationally intractable to do exactly (in a calculate this N dimensional integral with N>>1000 kind of way).

So in the end you still end up fitting some simplified vaguely similar model/function to your data and/or using sampling to keep things tractable.

TBH, ordinary least squares can also "learn" any function[1], so maybe you are not framing the trade-offs correctly.

I think that's an insightful point. Personally, I believe an important part of ML architecture engineering is figuring out architectures that make giving "the wrong answer" to a problem impossible, or at least difficult. But sometimes just throwing a bunch of data at a generic architecture and hoping it can learn the "rules" is easier.

And of course, as alluded in the article, sometimes we don't really know the "rules", or don't know how to articulate them in a way we can feasibly synthesize data according to those rules. Then people might reach for ML to figure out those rules, implicitly or explicitly. So it becomes a bit of an ouroboros situation. But still somewhere in that loop there's some domain knowledge injected somewhere by whoever is engineering the pipeline.

But even us as humans (or anything with biological intelligence) infers the rules from data.

The only thing is — we start collecting data since birth and we also don’t have to pay for every little thing we put in front of our eyes.

If you’re training an AI, you have to pay for data. If you generate some of your training data, you pay a little less.

(Also I think we ask of AI more than we ask of humans. For example, it’s common to be face blind of ethnicities you’re not familiar with but we get pissed if an AI is face blind at all.)

Hallucinated then

agree. fake sounds prone to erroneous data, while some systems are fairly easy to simulate, generating real data from a simulation. for more complex scenarios this is trickier, and a faulty simulation moght leave subtle issues in the networks functioning. if u dont have real.data to verify it with u will be in the dark about real accuracy. depending on the networks purpose, this moght be really dangerous.

Check out neural tangent kernels and neural odes