Many thousands of incredible scientists have done amazing work over the past ~century, but cutting-edge neuroscience still doesn't have the conceptual tools to go much farther than "when you look at apples this part of your cortex is more active, so we'll call this the Apple Zone".
Sadly/happily, I personally think there's good reason to think that this will change in our lifetime, which mean's we can all find out if trading the medicalization of mental health treatment (i.e. progressing beyond symptom-based guess-and-check) for governmental access to actual lie detecting helmets (i.e. dystopia) is worth it...
When we look at the "apple zone" part of an AI model that lights up, we see it in way higher resolution than our best scans of the human brain, and this might tell us something about how apples are perceived by both systems, or how language is represented neurally, or any number of other things.
That doesn't bode well for minds being human-interpretable, not at all.
I used to think that the biggest bottleneck to understanding the workings of the human brain was that it defies instrumentation. Which could be solved by better imaging techniques, high throughput direct neural interfaces, etc. But looking at the state of AI now?
If we had full read/write access to the state of every single neuron in the brain, what would we be able to learn? Maybe not that much.
Of course, that's specifically about human anatomy. In this case we're talking about a feature that I'd bet is present in other animals too, so the factors discussed here don't all apply. In this case though there seems to be a straightforward answer -- the structures involved are very small! The post I linked is largely talking about larger structures we failed to find...
This happens occasionally where two previously thought seperate fields of study discover a common link with each being able to explain the questions the other had struggled with.
The amount of anti-education/anti-school rhetoric on HN these days is worrying.
Until we both discover everything down to the Planck length, and then prove somehow that the Planck length is truly the smallest "unit", then we have not discovered everything. And we have probably hardly discovered anything, relatively.
I genuinely wouldn't.