The Engineered Student: On B. F. Skinner’s Teaching Machine
thereader.mitpress.mit.edu
thereader.mitpress.mit.edu
I learned the basics of linear algebra in high school largely from a "programmed instruction" book inspired by Skinner's work. The Little Lisper family of books is the best example I've seen of this in recent decades.
It's a shame that Watters is committing so many malapropisms in this article ("operating conditioning", plus incorrect uses of "ascribing" and "negative reinforcement"). You'd think she could hire an editor.
Do you know anymore?
- punishment usually comes too late and also makes the trainee dislike the trainer and so makes future training harder
- rewards should come immediately at success
- since most rewards can't be given immediately, develop a cue (a "clicker" is popular) which you give at the immediate moment that informs them they're about to get the reward. This bridges the time span between act and reward, and the cue itself becomes valuable to the trainee later
- variable rewards often work better
- trainees in a rut can sometimes be jolted out of the rut by giving them an undeserved reward jackpot
- teaching a trainee to not do something is harder but one strategy is to teach them to do it on command. Then simply don't never give them the command. Most behaviors die out once trainned to command except when commanded
- or train them to do something that is incompatible with the undesired behavior
- bizarrely, stopping doing an undesirable task can be reward enough to train them to do the undesirable task. I'm not super clear on how you do this in practice
- giving cues and rewards too late is a common error. It's counterintuitive because you often must interrupt the behavior you are training. Rewarding your dog for sitting will cause the dog to stop sitting, but you must do this. Waiting until the dog stops sitting is too late since then you are rewarding the stopping of sitting, not sitting itself.
- all of this applies to humans (if you don't let them know what you're doing) and also to yourself, supposedly
I'm trying to train my misbehaving cat. We're making some progress but it's not as fast and magical as the book makes it sound (my inexperienced training skills likely contributing).
I'm also learning Mandarin and a lot of what I've learned is through apps like Anki. Feedback is instant. But it doesn't really feel that rewarding to get a card right, so I'm not sure how successful it is in a Skinnerian sense. Teaching machines are here and work but I think they're still suboptimal.
A lot of users use addons to provide positive feedback in Anki. You may want to give them a try
https://ankiweb.net/shared/info/1722658993 - Puppy Reinforcement
https://ankiweb.net/shared/info/579111794 - Anki Killstreaks
https://ankiweb.net/shared/info/41708974 - Leaderboard
https://www.youtube.com/watch?v=UkveLkAgXiM - Anki Add-on Guide: Gamify Your Reviews
I’d say the pieces that are best for influencing how to work with people are (and obviously much of this does come from Skinner) are variable reward systems and immediate feedback of good/bad behavior. That’s about it.
I’ll also close by saying that Pryor’s methods are most relevant when there is ample time for the training and when consequences of misbehavior are relatively low. If the behavior being extinguished is for example randomly attacking other dogs, Pryor’s methods don’t effectively address that kind of thing, despite her claims to the contrary. And while Koehler gets a lot of justified criticism, in my experience with potentially dangerous animals at least like large-breed working dogs, a mix of Pryor-like and Koehler-like approaches are required. The happy clicky feedback isn’t going to convince an aggressive Malinois not to shred someone’s arm, much as we wish it would.
Earning a Ph.D. in this area, I was always frustrated that - as utterly brilliant as Skinner was (e.g. he wrote a dozen books, including best-sellers, in the middle of the night after a short rest) - he always overlooked an obvious fact: people saw themselves as special from other animals, and would not accept any theory or technology that treated them as something less.
While 'operant principles' could explain everything from the emotional lives of people to the conditioning of neural networks, Skinner's approach was, once again, too rational to survive in the public sphere.
As far as I know, he never engaged with the obvious arguments against his work - particularly the poverty of the stimulus argument. I don't believe he was a particularly rational individual - he was simply convinced of his simplistic model, and didn't want to see all the ways it didn't succeed in predicting all animal behavior.
It's not that humans are special, by the way. There are human behaviors that can be trained with behaviorist approaches. There are also animal behaviors that are obviously not trained in similar ways. For a good example of the latter, look at how long it takes quadruped animals to learn to walk - many do it within seconds or minutes after birth. Deer are leaping over obstacles hours after birth. That's not operant conditioning.
[1] And to be contrasted with those idiots who think that humans are “special” since they think that simple theories are not sufficient... or you know, maybe that organisms in general are complex machines or whatever you want to call it and that, yes, there are differences between a maggot, a bird and a human.
and see articles such as: https://www.sciencedirect.com/science/article/abs/pii/000579...
Your replies in this thread are quite predictable.
Emphasis mine. Human behavior is not physics. Even in that domain though, Newtonian physics is still useful - we just happen to know its limits.
For example, a basic study allowed animals to self-administer cocaine. Other animals were administered the drug in the brain on the same schedule but without 'choosing' to take it. The findings showed that the areas of the brain affected differed depending on these two conditions. The role of behavior in this system of reinforcement is determinant. This is a study with rats, yet the effects of a seemingly small environmental/psychological difference changed everything about the reward experience.
Another study on electrical brain stimulation showed that while animals would self-administer it, when it was automatically administered, the same animals would respond always to turn it off. Again, this is agency and it is demonstrated via a Skinnerian model.
Again, as a basic example, operant conditioning can't explain how a newly born animal can operate extremely complex biological machinery like quadruped legs. Also, studying animal behaviors in the wild shows other mechanisms of learning, such as curiosity, self-directed play, or imitation. Imitation in particular can show one-shot learning of complex behaviors, without the need for reward mechanisms (for example, chimps and birds can learn how to operate certain mechanisms such as latches after observing someone operate it a single time).
Bringing human learning into all this is mostly a red herring. We can refute operant conditioning as the complete model of how behavior is determined without mentioning humans even once. That is not to say that operant conditioning is not a piece of the puzzle - it absolutely is. But it is far from the final word.
There is a behavioral literature on what's called 'stimulus equivalence' - it shows how emergent, novel behavior evolves out of training the underlying relations (e.g., train a>b and b>c, then you get the emergent c>a). This happens with humans down to the level of mild retardation, but it's very hard to demonstrate it in animals.
Such emergent behavior is a kind of variability - the same genotypic variability that the environment 'selects' from to reinforce complex stimulus-governed learning.
If animals show observation learning, as you suggest, it's a mistake to conclude that it's not learned. As dozens of studies have shown, when the reinforcement is contingent, not of a fixed behavior, but on a conceptual behavior, that's what's learned (eg. an animal doesn't get food unless the sequence of three behaviors on a 3x3 grid are totally unique relative to the last 100 responses). Such contingencies are natural in all animal environments.
An animal or person can't execute complete motor performances according to some computational scheme, unless by this you just mean neural-networks (which don't compute, they just respond). If you look at primitive reflexive responses in insects, you see that the underlying neural pathways are, at the micro level, different for each individual. In other words, the brain develops according to environmental feedback (the first matrix movie has much to teach us about this). This is not learning, but the same notion applies to learning: a four legged animal has to learn to walk, then run, and gallop, and that neuronal-based learning is governed by feedback between the organism and the environment. You don't have to call this operant conditioning, but that's all that's involved: a complex biological system adapting from environmental feedback.
We overlook the power of conditioning because we think in simple terms of behavior being reinforced. As noted here -- https://psycnet.apa.org/record/2000-08385-002 -- behavior is not what's reinforced. What is selected is a stimulus-response relationship. Think of memory. Memories are not stored, if they were, the brain would have to remember where they were stored, ad infinitum. Memory is just a tendency to respond based on history: we see a movie, we remember a movie. It's all about remembering (a kind of behavior), and you won't think of what was the first car you owned if you don't read this sentence. Without stimuli, including interior/private thoughts, memories don't really exist. Sensory deprivation shows this impact of removing stimuli.
A problem with Skinner was he often simplified things too much. But not always. As seen here -- https://www.jstor.org/stable/27759371 -- his offering of a 'selectionist' model where the operant was the constructed product of cultural conditioning, anticipated a number of influential philosophers that few would think to tie to Skinner. As well as a number of post-modern philosphers
Let's take a look at this video [0] of a newborn foal learning to walk. It does so in approximately 30 minutes, without any explicit reinforcement from anyone (its mother is supportive, but doesn't explicitly encourage or discourage any particular behavior - it is simply caring for the animal). It also shows the ability to interpret visual input from its eyes, to recognize objects such as its mother, and the ability to produce and execute simple plans (get up, go to eat). If you look at a million million such foals, you will never see one behaving differently in any significant way - for example, not a single one will start grazing, or try to eat it's mothers teats instead of suckling on them, or try to suckle at thin air and so on.
The only possible explanation for this type of rapid discovery is that it was born with these abilities "built-in", trained by evolution over countless millions of years. There is no adaptation going on: the animal is just born with the "computing infrastructure" to perform these tasks. Of course, there is further learning to be performed later, and the algorithms for that are also part of the genetic endowment.
Note that "computing" is an extremely general notion, and artificial neural networks at least are absolutely computers. Whether biological neural networks are also computers is not settled science, to be fair, though it is hard to even imagine today what else they could be.
> Think of memory. Memories are not stored, if they were, the brain would have to remember where they were stored, ad infinitum.
By this argument, computers would be unable to store information as well. That's not to say that I imagine animal memory is equivalent to RAM or HDDs - they are vastly different. But RAM is definitive proof that it is possible to store and retrieve information without reaching an infinite regress of "who remembers where the information is stored".
> Without stimuli, including interior/private thoughts, memories don't really exist.
If internal states of the brain can be considered stimuli, than the entire concept of operant conditioning becomes a trivially obvious idea, a metaphor for almost anything. By this notion, CPUs can also be said to work based on operant conditioning: electrical pulses are "internal stimuli" that shape the behavior of the integrated circuits. Atoms themselves "learn" to decay when receiving stimuli from a neutron particle hitting the nucleus.
> An animal or person can't execute complete motor performances according to some computational scheme
Again, we have definitive counter-evidence to your statement from the world of computing. Modern robots that are able to walk decently in rough terrains are in fact doing so based on sophisticated (statistical) computational schemes. If you want to claim that ANNs are not computation per se (which is wrong), we can fall back to industrial robot "arms" which have existed for decades and are able to perform sophisticated environment-based movements while being manually pre-programmed using mathematical control theory (systems of differential equations).
(And that influence likely is not just one of ideas, but also of people. “The Dream Machine”, the book on Licklider and the early days of computing mentions that some of the early computer pioneers shared the same office building with Skinner.)
RL is useful for certain constrained problems, such as game playing, but it does not generalize well.
Additionally, all current ML methods are extremely slow to converge (they require enormous training sets compared to most biological learning), and they generalize pretty poorly outside their initial domain. So, their success is not a good argument in favor of Skinner's models being a good model for biological learning.
In an audio recording he dismissed Chomsky’s critique of his work. He said that he hadn’t read it. He then went on to agree with people who had had read and disagreed with the critique... but how could he agree with them if he hadn’t read the critique.
But that’s not the point.
Say you haven’t read X:
1. It is rational to say “I haven’t read X so I can’t comment on it”
2. It is less rational (or dishonest) to say “I haven’t read X but I agree with these other people who have read it and dislike it!”
As far as I can see, education is full of really perverse incentives which are, if not created by, greatly exacerbated by the prevalence of technology. Gamification (in general) ended poorly. MOOCs never changed the world. And so much of edtech forgets that education is about making better people, not better test scores.
Take something relatively simple like learning to read. It's relatively straight forward to test, it's usually learned in relatively tightly controlled environments (schools) and yet we har no idea what is the best way to teach kids how to read. It would be comparitavely easy to conduct large scale experiments: make several groups of schools use different approaches to teach reading over several years, conduct standardised tests and compare. The populations could easily be large enough to average out other variables and after some time we would have a reasonable good idea which approach works and which doesn't.
Instead we have a bunch of deductively derived theories paired with small scale qualitative studies ("it worked for me") , with dubious statistical significance.
It is quite frustrating really.
There are large differences between school education and higher education though. Both my children's school used some software called TT Rockstars for learning times tables which is very much a gamification of that and it did seem to work and motivate the children fairly effectively. A lot of schools obviously experimented with various online systems in the lockdowns here in the UK (and I also did as a parent!) and it was interesting seeing which engaged my children's attention and which didn't.