What did Mary know? A thought experiment about consciousness (2013)

"Jackson’s thought-experiment about Mary challenges physicalism in the following way. Physicalists claim that physical science can fully explain consciousness. However, when Mary sees the red of the apple, says Jackson, she learns something new, despite having previously learnt all the physical facts about colour vision. Thus, argues Jackson, Mary has come to know a non-physical fact; so proving that not all knowledge is physical."

I can't see how this result proves anything. Consider the analogous experiment of "Mary learns to cycle by reading books". Surely she can learn all the physics behind cycling without truly knowing how to cycle, and one could similarly conclude that there is something beyond the physical mind behind this skill. Nevertheless, we can use reinforcement learning to teach an ANN to cycle, and that doesn't mean our robot has acquired a metaphysical consciousness.

You can still interpret both results from a physicalists perspective as in "some learnings require specific physical stimuli".

The easiest way I see to parse this is by doing a thought experiment: what if the human brain was indeed a hardware/software device, how would this work?

Eg, can you take a webcam image analysis software and teach it that apples are red without ever showing it something red through the camera?

I'd say the answer is obviously dependent on the design.

Yes if you have enough control over the software. You can introduce the shape of an apple into its dataset, and tell it that an apple shape is such and that it rates highly on the R channel. Then the program will upon seeing an apple for the first time say "Yup, this is indeed a red apple" and be entirely "unsurprised".

No if the system is isolated enough. Eg, if the recognition system is locked in a black box and your only controls are a query of "what is this?" and a command of "remember that the thing you're looking at now is X", then you don't have the access to introduce the information above by any other means than presenting an object to the camera.

Both systems are still 100% physical though.

The question is what humans are most similar to and I think the second option is most likely. Think of that for instance I can't mentally measure my blood pressure, or heart beat, or make myself vomit. There are some things my body does that are outside of my direct control. Given that, there's no reason to assume then that by just reading or listening to information that I would necessarily have any pathway from there to the depths of my visual system.

It's pretty clear Mary learns something new upon viewing the red apple and it's easy to create an experiment for this. We'll continue using Mary's monochrome example as a basis.

Consider the following experimental setup: Mary is prepared with all the knowledge about colors and apples and every other odd thing just as before. Mary is then instructed to view an apple and then subsequently asked to follow the instruction printed in red in the next room.

In the next room there are two sets of instructions. The instructions in red say "Exit through the left door" and the instructions in blue read "Exit through the right door". Behind the left door is a million bucks and behind the right door is nothing.

Does Mary get the million bucks? Would Mary get the million bucks had she skipped looking at the apple? It's probably helpful to do this with several Marys and ask, is there evidence that our Marys choose the left door greater than the expected probability attributed to random chance? If we can answer yes, no, and yes, then we can infer that choosing the view the apple changed Mary's mental state and that yes, she did learn something new that helped her to identify the correct text.

What I find interesting is that this seems to be an obscure philosophical thought experiment, but it's actually highly political and turns up all the time on, say, Twitter. Specifically, how do you compare "factual knowledge" and "lived experience", and which is more important, and who is the "expert" on things.

To make this concrete, consider expertise on England. Suppose Prof. Smith has never been to England, but has studied it in detail for decades. And suppose Mr. Chav has grown up in England, but is uneducated and isn't even sure who is the Prime Minister. Who is "allowed" to talk about England? Smith, who knows all the facts, or Chav who lives there and knows what it's like? Can Smith tell Chav that he's wrong about England, or does Chav's lived experience matter? (I'm using England as a less controversial example, but this usually happens in discussions of race or gender.)

How is dualism supposed to resolve this question, any ideas? I don't really see what it brings to the table on this. The article doesn't seem to address that.

What frustrates me with dualism generally is that it doesn't actually explain anything, it just seems to say "we don't understand how this works physically, therefore woo". It's the god of the gaps argument all over again. I may be seriously mischaracterising it, and that's not my intention. I've just never come across any positive arguments for dualism having any explanatory power on it's own terms, and I'll be very happy if I'm wrong on that. What characteristics or attributes might this non-physical quantity or influence or 'stuff' have that answers any of these questions?

To me it seems a word game, as happens so many time with philosophers. What is meant by the words 'knowledge' and 'consciousness', exactly? Define them strict enough, and the paradox will evaporate.

E.g. in this case, let's define 'scientific knowledge' the ability to predict the future, and to choose scenarios where a specific future happens. Science is then the process for generating scientific knowledge. Newton's theory, for example, allows us to predict the future of an object with mass, and choose ways to put such objects together in such a way that they end up in a specific place at a specific time in the future.

In this case, Mary knows everything about 'Red'. This means she can predict what happens when something red interacts with something else. If she also knows everything about 'Mary', she can predict her own surprise at seeing something red. She can choose actions that make her come in contact with something red in the future. She can then still feel surprised at experiencing red.

The thought experiment simply contradicts itself.

The premise is that Mary knows everything there is to know about colour vision. If this is true, then she knows what it is like to see colour.

If she doesn't know that, then the premise is false.

It's just nonsense.

I have a take that I'm torn on about whether it misses the point/is covered by one of the arguments presented in the article, or makes sense -

It seems to me that one can know about all the physical properties of color before having experienced it, and still not have any expectations of knowing what their perception of seeing a given color would be like. Those are two different knowledge domains which are relevant to us, but which have no real importance in a "universal truth" sense.

Imagine we were talking about red with an individual of an alien species which doesn't have eyes or other light sensing organs, but is technologically advanced enough to know about photons and the physical properties of light and use them in their navigation systems (maybe not unlike we understand something about radiation without being able to feel it). We'd agree on the existence of the "red" wavelength, and we'd agree that certain surfaces reflect it and certain bodies emit it. But it'd be rather futile to try to explain seeing the color red to the alien even if we could, because it's irrelevant to them; it's just some being's perception of a physical quantity, a delta in the chemical balance of our brain soup. To them it's no more special than the feeling of hunger, a crude reflex which evolved so our bodies could tell us we need sustenance. Importantly, there's nothing of practical value about color itself that they could learn from the discussion. In that sense, does Mary really learn something new about color, or does she just learn about her own perception of it? Are those two things really related?

The experiment is flawed in that “Mary knows something new” is very loosely defined. We know new things constantly, to begin with. Our neurons change continuously.

As an example- She’ll immediately have a new memory: “I saw the color red for the first time”. Does that count? What about the emotions she might have felt? Elated. Disappointed. Does that count?

You are “knowing new things” just by reading these words.

There’s also the fact we don’t need a thought experiment for this one. There are rare cases where people have been able to gain sight or hearing via operations or devices. Ask them directly! All of them will say that they had many new thoughts when this happened.

That doesn’t imply that qualia exists at all though:

> she has gained a non-physical fact

She now knows how it feels to see the color red. I don’t think that disqualifies this as a “physical fact”. It could be that the subjective experience can be explained in physical terms, but we humans just not have figured it out. In that case Mary would not have that on her notes, would her? After all, this is a complex problem, I don’t expect conscience to be easily explainable in 2 or 3 lines, the same way that “how does the Linux kernel work?” can’t.

It seems that people who were born blind dream in visual images[1]. So certainly Mary had experienced colors in her dreams.

[1] https://www.wtamu.edu/~cbaird/sq/2020/02/11/do-blind-people-...

I've had a handful of arguments w/ smart people about Mary's room lately.

The conclusion I've come to is that the disagreement typically rests on how you define "knowledge." To many, "what the color blue looks like" does not constitute knowledge, because it doesn't enable you to make any additional quantitative statements beyond what you would know from an objective study of color/light. But in common usage, we say things like "I know what that feels like," meaning "I've felt that before." In the end, this is just a semantic disagreement.

But there are still philosophers like Dennett who take an even harder position, saying that functional knowledge is identical to experiential knowledge. I still can't quite wrap my head around this idea.

Is this another ill-conceived argument for the existence of classical Christian metaphysics? They'd have a much easier time concealing the bias if they didn't construct it exclusively from the themes of genesis... Woman... Knowledge... Apples... Who forgot to bring the snake?

Well, seeing some new color for the first time will cause some sensory input signals to become active which have never been active before, and that will cause some neurons patterns to emerge which have not been there yet, and this will surely have influence, e.g. synaptic plasticity, i.e. in other words the brain has learned.

But then, the brain also learns basically all the time. It's constantly changing. It is not something static.

But is that really the question?

This "thought experiment" is pointless. We might as well debate the mechanics of Harry Potter's magic spells.

> she grew up entirely in a black and white room, she has never actually seen any colours.

White is all colors, or in other words, there is no colour "white", the subjective experience of it is an illusion caused by perceiving multiple frequencies of light simultaneously.

This human guinea pig raised in a black and white room will have been seeing red, and green and blue, all her life. In fact the "whiter" the room the more frequencies of light she must be experiencing.

When she finally sees an apple there is no new information at all, in fact, she is seeing a kind of shadow. It's the absence of other colours that makes the red noticeable.

What if it was a green, yellow, or pink apple, or an apple painted violet? She would also need someone to tell her the common name of the new visual sensation she experienced. In any case, it is just another phenomenon in the bucket of 'being Mary'.

I understand the experiment could be reframed to fix this "bug", but it's flawed: when Mary sees a red apple for the first time, some optical/neural pathways get excited that weren't before. She knew everything there was to know about the color red, but a specific physical experience was missing (physical, with observable physical changes that can be physically measured).

So this isn't about metaphysics or anything. Had Mary stimulated her optic nerve with an electric current to simulate a red apple, we'd be closer to the intended message of this thought experiment (and I'd argue in that case that a physical apple and a simulated one are indistinguishable).

"consciousness is all about the brain; or more specifically, it is identical to the brain"

That's easy to disprove with the computer analogy. If transistors are simplified neurons and CPU chips are brains, then software is consciousness. Just like software is an immaterial something that uses magnetic medium to run on CPU, human consciousness is immaterial something that uses magnetic medium to run on neurons. Mind is parricularly fast type of hardware, an equivalent of GPU, that runs imaginary experiments. One thing that's certain is that software doesn't emerge from transistors and barely can it be understood by inspecting those transistors.

When Mary sees a red apple for the first time, her mind records the relationship between "red" and "apple", while her consciousness registers the new waveform produces by red receptors in her eyes. The richness of this waveform is limited by her eyes, whether they capture basic redness or the entire spectrum, and by her conscioussness that may or may not be able to register waveform of such complexity. She gets to experience a surprise because she hasn't seen such waveform before, and her mind will put the red color into its proper place by recording relationships between observed objects.

Mary can know what it's like to be a bat because bat's consciousness is much simpler: any waveform that can exist in the bat's consciousness can also exist in Mary's consciousness. The reverse isn't true.

You can learn all the biochemistry about a strawberry, but actually _tasting_ one is going to be a new experience. I don't see any need to bring in metaphysics to explain this (in fact, doing so indicates a confusion of ideas to me).

When we describe sensory experiences to one another, we usually do so in comparisons. "It tastes like chicken." "They look like little blue bananas." "It smells like almonds and paint thinner."

Of course there will be a gap in understanding if Mary has never tasted chicken, seen the color blue, or smelled paint thinner.

> ...despite having previously learnt all the physical facts about colour vision...

Well, we _could_ simply include scratch-and-sniff in our physics books...

If Mary never actually had examples of the different colors, then she wouldn't know how to recognize a color when she eventually saw one. But that's because she (and pretty much everyone else) can't synthesize accurate sensory simulations given information like frequencies or wavelengths.

One could imagine a machine that could take the physical description of a color, taste, smell, and stimulate Mary's brain to simulate a sensory experience. No metaphysics necessary.

Seeing color(qualia) does not provide any additional information. They are just a tool your brain uses to speed up processing. Treat is as quick tagging system. It's done inside your head, we don't even know if all people tag colors in the same was(is your red, my red?). Qualia are derived information. Your senses get information, and you get some processed result. There is nothing added(beyond what your mind may add, but that's not part of the original information). You could swap red for green, green for blue and nothing changes. Imagine a wall of text, a very big one. To read it you have to carefully go line by line, focused on the content. It takes a lot of time to do. But now imagine you evolved to read such text walls. Now it's not hard, because nature developed special structures to process them, and now you have dedicated, and optimized region in the brain to handle them. You just glance at that massive text and "see" all the content, it sings to you, you understand it instantly, maybe even feel parts of it. And seeing is the same. The mass of color information hitting our eye, that for someone without innate ability to process it would look like a bunch of hard to plow through data, for us is just instant. We process multiple frames per second, detect objects, project movements. All of it without even thinking about it. Incredible. All thanks to specialised machinery in our brains. The specialized part does the heavy lifting so we can see our qualia. But it's just a tool, it does not provide additional objective information, as it's only your subjective view of the information that other parts of the brain push to you. It's processed information, enriched, but because of that, it's subjective now.

Can words and word structure capture all ideas?

(I'm using 'words' in a very broad sense as packets of information that can be communicated sequentially).

It's a broader and easier to answer question that covers this specific case.

Another interesting question is, are there disjoint sets of words and word structure (meaning that they cannot be 'transpiled' 1:1 into one another) that can describe and predict physical phenomena just as accurately as another? Particularly the set of words and word structure that we've come up with so far as a species?

Some recent experiments seem to suggest, yes, there are combinations of variables that can fully characterize certain physical phenomena that cannot be translated into our current understanding. This means there can be parallel and equally valid and precise models of reality.

https://www.nature.com/articles/s43588-022-00281-6.epdf?shar...

What is a color image? An RGB image of pixels. Put another way, three images with red, blue and green filters.

If Mary could only see black and white, reading every book about color would not give her the experience of red. Like a child could read every book on cycling, but not know how to ride a bicycle.

However if Mary could interface a color camera to her visual cortex, she would experience red over time. She could add a bigger spectrum of infra red and ultraviolet and be able to experience more than a normal human being.

Similarly for cycling, if someone had VR goggles, and could simulate forces on their body, they would learn how to ride a bicycle.

Words and writing are a highly compressed version of shared experiences. When we read "red apple fell from the tree", our minds are creating simulations of apple, tree, falling symbols from the raw experiences we've perceived.

To understand an "apple" we have to see many different kinds of apples, taste them, touch them, feel their texture and weight, smell them, the sound when we bite them. All of that is compounded into the symbol "Apple". But we also disambiguate the internal symbol {apple} from the fruit Apple and company Apple.

This thought experiment is a wonderful demonstration of the limitations of thought experiments in philosophy.

The premise of a thought experiment used in this way is that the truth of the matter is ultimately to be found in your intuition. One just needs to properly ingest the experimental setup, and the right conclusion will lay itself bare.

This experiment is as absurd as the Chinese room experiment. You will be told that you have only properly understood the experiment's setup if you accept that Mary can somehow have learned all physical facts about a chosen color without experiencing seeing the color. It begs the question, because it explicitly claims that knowledge obtainable only through experience is "not physical" (Mary somehow "knows" all physical information about the color before experiencing it). "But wait, if experiencing red added new information, then experiencing red must not be physical, because... we assumed experiencing red is not physical."

edit: left out a space

I have been very interested in Buddhist non-dualism recently. Although the philosophical implications are very profound assuming that is true, but I don't.

For me consciousness clearly emerges from our physical body and not the other way around. However, I admit that is fascinating and learning about how self-identification can lead to suffering is eye (mind?) opening.

I’m fully in the physicalist camp. I think this is a flawed thought experiment because humans are incapable of knowing everything about seeing red until they actually experience it. Here’s what it’d like be if we actually could without experience:

Suppose you take a perfect snapshot and learn the exact position of all atoms in a persons brain after they see a red apple. Now imagine that humans have the ability to move all the atoms in our brain to be in those exact same positions, at will. So without the red photo receptors in your eye actually detecting anything red, you can make yourself see red apples at anytime. Mary could do this even before ever actually seeing red or even when seeing something that’s reflecting a completely different color to her eyes.

Since we don’t have this magical ability, having an understanding of all the underlying physics and biochemistry of color perception does not really provide everything there is to know about seeing red.

My current understanding of recognition (I think it was pulled from a Kurzweil book I read recently, so grain of salt etc) is that humans have on average 150k recognizer circuits[1], which are long columns that permeate a few layers of our brain. Signals come in from various sensory inputs, and it lights up some subset of recognizer circuits, and that data is then passed on to other parts of the brain to analyze the context. Every letter of the alphabet, and every word you use regularly, and every object you see, has a handful of associated recognizer circuits which will trigger given the shape, smell, or the sound, etc. When one of these circuits goes unused for a while, it can be repurposed.

Personally, it kind of blows me away that we can function as well as we do while only being able to immediately recognize 150 thousand total concepts. It's such an easily comprehensible number, yet completely defines our perception of reality.

If Mary had never seen the color red, so no matter how hard she studies, her brain will not have the circuit capable of recognizing the color red, simply because that particular signal has never entered her brain. She might be able to acquire it later though. Colorblind people get the protein injection that allows them to see the third color, my understanding is that it takes a few months[2] for it to take effect, which lines up well with my understanding that it would need to rewire a bunch of recognizer circuits through repeated exposures.

[1] https://numenta.com/neuroscience-research/cortical-columns/

[2] https://www.npr.org/2009/09/17/112897277/thanks-to-gene-ther...

>The ignorant are cocksure, while the wise are full of doubt.

If we assume our abilities are limited and finite, we can never be 100% sure of anything. However, we can be 90% confident of things based on: (a) Shared wisdom and learning (b) Scientific research and experiments. Philosophy of knowledge is a tool to get knowledge; it is not knowledge itself. Going back full circle, we believe our abilities are finite and limited. We believe we cannot know everything. We can only be confident based on our limited shared knowledge and scientific discoveries. We expect future knowledge to contradict old knowledge. Philosophy underpins or guides this quest for knowledge. Philosophy are fundamental assumptions or beliefs. e.g. Truth cannot contradict itself. If truth contradicts itself, it is our limited abilities that deceives us.

"Many black and white books and TV programmes have taught her all there is to know about colour vision. Mary knows facts like the structure of our eyes and the exact wavelengths of light that stimulate our retinas when we look at a light blue sky."

This is a serious hole in the argument. "All there is to know about color vision" is an abstraction---it's "about" color vision; it's not vision. The experience of color vision is a different set of data than wavelengths and eye structure.

A thought experiment seems to be exactly the opposite of an experiment: no experience there.

I understand the concept, but every time I see one of these, the most compelling points are for me the ones that show how the setup makes no sense at all because, for starters, there's no way to make a real experiment that's truly equivalent.

Edit-> please, think about this: can the "result" of any thought experiment be surprising, not what you expected?

Mary would probably not be able to see that the apple was red.

Just like I can’t hear the difference between sounds in many Indian words Mary would not be able to differentiate colorized things from non colorized things. Her brain would have learnt to ignore the color signal that the eye provides. Just like my brain has learnt that the different sounds in Indian words is not worth processing.

I wonder if this experiment could be ran in real life today with EnChroma Color Blind Glasses and a cohort of color blind subjects.

A lot of these thought experiments are just semantic arguments in disguise.

We think we disagree about physicality of consciousness, but we're actually disagreeing about meaning of the word "all". She knows all there is to know about colors.

This might be a good linguistic discussion, but it's unlikely to move us anywhere on physicality of consciousness.

This does not seem a complicated question. No amount of words or verbal explanation can convey the full sense of qualia. You cannot describe the phenomenon of sight fully to a blind person. This seems more a limitation of language than consciousness. I mean... surely a pig sees just fine, but you cannot explain the physics of color to them.

> If we were to show Mary a blue banana instead, she would not be fooled; she would know that it had the wrong colour, argues Daniel Dennett, in what he coined the ‘blue banana trick’.

Why won't she be fooled? Is it because she already knows that a banana has a different color than the sky and she knows the sky is light blue?

This paradox only shows that the definition of knowledge used to produce it was wrong.

Mary learns nothing and the physicalists are correct.

Since Mary knows everything about human physiology and physics, she is able to simulate a neural vision system within her own brain.

The experiment ask "Has Mary learnt anything new about the colour red upon seeing the colour for the first time?"

Yes, she has learned to detect red colour using her eyes.

I'm not sure if this experiment means anything. Consider a much more mundane(?) example: echolocation. Bats can "see" the world around them via ultrasound. Humans can research everything about how it works, including acoustics, physics, neurology, bat anatomy, you name it, but we will never experience it the way a bat does.

So... does that prove that human consciousness is non-physical?

...I'm sorry, what? That sounds like a complete non sequitur to me.

Which is exactly how I feel about the original argument.

It's sad to see so many people focusing on the color aspect of the experiment, completely ignoring the argument in the process.

Look at it this way - switch out "red" for "parental love" and then re-evaluate the argument.

Mary has read and currently knows everything there is to know about the bond a parent forms with their child[1].

And then Mary raises a child ...

[1] An adopted child would do - then there's no claim of biological hormones affecting the knowledge.

"Consciousness is what keeps philosophers awake at night."

Unless they're dyslexic, right?

Loved this. Thank you for sharing.

Another way to rephrase this is: Does the person who has read absolutely everything about love, know what it feels like to love? This thought experiment relies mostly on how fuzzy our understanding of consciousness is, and how vague our definitions of it are.

It confuses conscious 'knowing' and conscious 'seeing' (or 'feeling' in the love example), which are two different things. How can I state this confidently? Because we do have a wonderfully detailed computational model for consciousness, which explains a ton of data!

It's a real shame that it's largely ignored or unknown even in the field of neuroscience. I've written about this here: https://saigaddam.medium.com/understanding-consciousness-is-... As I mentioned there, I also co-authored a book (https://www.goodreads.com/book/show/58085266-journey-of-the-...) where we go into this theory in greater detail. Stephen Grossberg and his work really do deserve greater attention.

Most discussions of consciousness are frustratingly incomplete because they leave out a crucial aspect. Who is the “experiencer”, that is consciously “experiencing”? How is that "I" put together? Only a clear answer/theory that attempts to define and construct both the experience and experiencer will allow us to see through these cutesy philosophical constructs like Mary the color vision expert, philosophical zombies and more.

Here's a definition, based on the computational model I mentioned above, that makes a lot of sense.

Consciousness is the constellation of past experiences experiencing the present, assimilating it to act and prepare for future opportunities.*

So you are nothing but a bundle of experiences.

Each new experience, after it is had, is added to this bundle, and changes you. With that in mind, here's how Mary's experiences make sense. Mary could read everything there is about color vision, but all of these are just semantic experiences. They become a part of her. When the first visual experience happens -- when she actually sees red -- she has absolutely no visual experience to fall back on that explains this. Reading about it, does not in any way reconstruct the unique neural patterns going all the way from her retina to her inferotemporal cortex. The experience is utterly new, and as with all new experiences, it is memorable and retained and becomes part of her. That sadly, also means that all subsequent experiences of red are not as vivid, because they are now matched against what came before. (This invokes the idea of predictive inference; there are reasons to do this -- it is metabolically cheap to just classify and forget, rather than retain the full pattern.)

*-this definition of consciousness leaves out how an experience is constructed. To answer that very briefly, it is when the sensory data streaming in is matched locally and globally against past experiences and a broad agreement (a “resonance”) is achieved. This is obviously a very lossy summary. You can read this for article for a complete picture: https://www.sciencedirect.com/science/article/pii/S089360801...

TL;DR: Plato's Cave meets The Giver.