Can GPT pass my programming courses?
lemire.me
lemire.me
I think it will be very interesting to see what the teachers, and institutions, do about it though. It's not like students couldn't find these things on the internet before GPT, but it's so much easier now. Over all I personally think that a nice shake-up of the way we educate people might turn out to be a good thing though. That is of course easy for me to say, as someone who doesn't actually have to do any of the shaking-up.
How about this: test actual understanding instead of desperately making trivial tasks needlessly challenging.
It was fine. It tested actual understanding of the problems at hand.
No one should be asking someone to write long pieces of code by hand, but a dozen lines of code to implement a simple algorithm is fine.
I took tests by hand all my life and even in college I think most computer science tests that weren’t “implement this big project” were all done by hand. Professors didn’t give a fuck about fancy test software: bring pencil, bring paper, here’s a test, good luck.
Your hand isn’t going to cramp up writing double linked list or binary heap functions. Plus you’re going to have to do this anyway for whiteboard interviews to get your first job.
If a CS student can’t do this by hand, then just get out of the field. We do not need more intellectually weak engineers, especially as the amount of money one earns in tech seems to increasingly attract money hungry brogrammers that write shitty code. And I’m sure ChatGPT will empower an entire generation of slackers who think software engineering is just writing the right prompts in plain English.
Or they could improve interviews so they test more than memorization.
> If a CS student can’t do this by hand, then just get out of the field.
Why? Should construction job applicants also dig holes using old shovels to prove they aren't weaklings depending on modern machinery? Completely pointless.
> We do not need more intellectually weak engineers
But that implies current "intellectually weak" engineers were good enough for handwritten whiteboard interviews, so what's your point again?
It can also write CloudFormation, convert existing CF to Terraform and idiomatically correct CDK code in Python, convert code from Python to all of the target languages I’ve given it and convert Python AWS SDK (boto3) code to bash/AWS CLI.
Friday we decided to split our EF migration into multiple context files, something we should’ve done from the beginning to be fair. Anyway, we deploy with Azure DevOps and we migrate for our more permanent testing, staging and production slots through a VM agent on our VNET. We don’t have a DevOps engineer, and we don’t completely know what we’re doing but it works. I figure I could copy paste the code four teams, giving us 5, one for each context, but I didn’t want to do that. So I asked ChatGPT and it gave me an answer involving Bash, which doesn’t work in our DevOps pipeline, as in, it doesn’t exist in the context. I told GPT this, and fed it more information and it proceeded to give me the exact same code four times before it gave me something different, but also unusable code. That’s one of the good examples of it failing. Because it made things up and they didn’t work. So no harm done, aside from the wasted time.
A few months back we tasked it with writing something for our Odata client library, and it did. It also worked, only, it didn’t reaaaaalllyyy work because it was flawed in a way you might not notice if you weren’t an experienced developer. Basically what it had build wouldn’t scale effectively, or at least not in a cost efficient manner. It would have been efficient enough to run for a long time without anyone noticing, however, but it would have been rather expensive and it would’ve broken down completely under “normal” loads, if we ever grow to twice the size we currently are. But it worked and it looked fine, and that’s sort of the scary scenarios. Or maybe they aren’t scary because we’ll likely be well paid to clean up those messes for years to come.
I do think it’s likely to become more and more useful as time passes. I mentioned documentation in my GP, and I am genuinely amazed at how good it is at writing code documentation. You obviously don’t feed it sensitive code, but it’s been able to accurately describe functions from their name, input and return values alone. Seriously, it’s taken a function name and written a piece of JSDoc that was miles better than what I had written, accurately guessing complex context (which it only saw maybe 10% of) from the function and variable names alone.
I wanted to just generate one big .sql instead of one per context, but maybe you just can’t.
Highschool level questions aren't an issue for a bot trained specifically to answer such questions. A google search query could pass with flying colours.
I see that some teachers will no longer allow projects printed on paper, it needs to be written by hand to force the student to read the content, I think this is bad for some subjects where the student does not care about, they will just forget instantly what they wrote.
Many of the institutions have billions of dollars put back. Harvard for example google tells me 53 billion. So I don't think they are just going to go away, they will adapt somehow.
Education has been fundamentally broken pretty much my entire life. The cracks have been showing through at an increasing rate since the internet went mainstream.
If I had to pick between the current gulag and a self-paced world where nearly every child has access to the library of Alexandria with a personal AI tutor, I’d chose the later in a heartbeat.
I have a very strong feeling that the current system isn’t going to survive the AI + Internet 1-2 punch. I believe that what replaces it will be significantly better and waste less of my descendants’ childhoods.
Go on Wikipedia, Khan Academy, MDN, etc and open ChatGPT right now.
It's become glaringly apparent in the last century with progress vastly outpacing generational swaps. Old geezers in charge, trying to control a world they don't fundamentally understand anymore.
I don't think mortality was selected for - but rather, that it's longevity that never was.
The default state of any organism - or any temporarily self-sustaining / self-perpetuating chemical reaction or temporarily-stable dynamical system - is that something will eventually kick it out of the stable region, and the reaction breaks down, the system muffles itself or shakes itself apart, the organism decays and dies. The way the evolutionary process works, natural selection will strongly promote organisms that survive to the point of reproduction. Past that point, the influence weakens significantly. It's more work that affects reproduction weakly, if at all.
Humans are one of those few special cases, where it's strongly beneficial to keep the older individuals around, due to the positive impact of accumulated experience and thinking on survival rates of the group. I think in our case natural selection would actually favor longevity, or at least extending the part of the lifespan where we are capable of thinking clearly. Alas, from evolution's point of view, the entire history of humanity - from first tales and invention of writing, all the way to spaceflight and the Internet and GPT-4 - happened faster than in a blink of an eye. We're iterating and improving couple orders of magnitude faster than natural selection - we've decoupled ourselves from natural evolution, and we're stuck with the shitty hardware we have, that's optimized for barely above reproductive age.
IMO, the right answer isn't to throw up our hands and say "oh, science/politics advances one funeral at a time" or "evolution clearly thought longevity sucks" - but rather recognizing that we're on our own with this, evolution doesn't matter to us, and maybe we should come up with a better solution.
Mass instruction effectively treats students as the average of their cohort, without really getting to know each particular student. This is the fundamental break. It treats students more as blank states to be filled, than as collaborators in their own development.
As a specific anecdote, when I was in 4th/5th grade I didn't work ahead in math much because the assignments were to do every even problem on a page (one page, the potato page, as it had pictures of potatoes on it, had over 100 problems, I still remember it), which was tedious as heck. Because I didn't work ahead I was mainstreamed in math in 6th grade, and felt absolute horror at repeating yet again what I had already tediously repeated so many times.
Miraca Gross, in her literature on the exceptionally gifted, has a person with a 4 S.D. IQ who dropped out of high school and ended up in a store management track.
One explanation that occurs to me immediately is that this person was really good at scoring IQ tests but otherwise not especially smart.
My point is - sure, if you have more specialized attention smarter people will learn even more and even faster, I don't dispute that. But this idea that there's super smart people out there that end up doing store management because they weren't spoonfed the right information is just self-contradictory.
Quoting his mother: "At school he never had to learn to focus and concentrate to get over a difficult piece of work because everything came easily. Now he finds it extremely difficult to concentrate. Also, there have always been other things which he has deemed more important."
His education was mismanaged as a young child (e.g. being accelerated, but then being held back because a new principal "wasn't comfortable" making exceptions for him). Gross' book has a lot of specifics on a few kids (including the future Terrence Tao, pseudonamed Adrian Seng in this book), I'll try to excerpt and summarize a few pertinent ones:
---
> At five years of age Rick was handling, with ease, division problems such as 3721 ÷ 6.
---
A principal made the decision to mainstream him in math and revert him to Grade 2 math even though he had already successfully completed Grade 3 math. This was because the principal was concerned that he wouldn't learn how to be "happy" or "creative".
> the principal’s insistence that all children, regardless of their level of mastery, should use concrete materials in mathematics. Rick’s teachers tried hard to discourage him from working out maths problems in his head instead of using the materials provided to assist him.
> I have tried to explain to the teacher that to Rick all maths becomes a sum to be worked out mentally, even if it is a worded problem-solving task, but she still becomes so openly concerned when he does this, and speaks to him sternly in front of the other kids, that on occasions he comes home in tears. The change in him since last year is really disturbing. He has gone from being a happy, confident, stimulated little boy to a child who is bewildered and uncertain of what is expected of him.
> Next, the school began to encourage Rick to moderate his reading achievement so that it would be more in line with his classmates.
> His teacher allows the children to bring books from home to read in class, but for the last few weeks when he has arrived at school with the novel he has been reading, she has told him that she would rather he bring along shorter books with more pictures, which would be more ‘enjoyable’ for him. I’m sure she thinks that I am interfering with Rick’s choice of book, but I’m not. He is now frightened of taking his own choice of book to school in case he gets into trouble. (Jan Ward, mother of Rick)
---
Following social hazing from teachers (not just the above, but active, IRL shunning) and the effective exclusion from playing with other boys who were his age, when his parents transferred him to another school in Grade 5 he made it clear that he didn't want anyone to know he was gifted and ended up mainstreamed. He learned to mask his giftedness to fit in.
At this new school he was still bullied and begged his parents to transfer him to another, which they did for his Grade 8.
> At his request, his parents did not mention his high ability to the new school and he chose to ‘cruise’ academically, doing enough work to get by but not enough to stand out from the crowd. He was content to excel in music – a subject in which this school not only permitted but expected students to excel.
Around this time he realized he was gay, became involved in a long-distance relationship with another boy, and became seriously depressed when the other boy ended the relationship. A few weeks later he started skipping school and dropped out entirely the following year (Grade 12).
---
The book ends when he's around 20, and sums things up then (he was then currently in the management track at the supermarket chain):
> I know that I will be discriminated against in some ways because I don’t have a Grade 12 Certificate [he says], but I don’t think, at this stage, that I could try school again, and if I went to university I have no idea what I’d want to do. I don’t want to have wasted the last two years, working for this company. I have to establish a profession to stay in. I still disagree with a lot of company policy. I can see so many ways of improving things both financially and in terms of customer service. But I’m not even going to try to explain my points at this stage. I can wait.
> Rick’s parents are glad he has found something to which he can commit. They understood, only too well, his early restlessness, even while it worried them. He has never learned how to study [Jan said]. At school he never had to learn to focus and concentrate to get over a difficult piece of work because everything came easily. Now he finds it extremely difficult to concentrate. Also, there have always been other things which he has deemed more important.
> A recent synthesis of research on issues in the social and emotional development of gifted students concurs. Most of their school days are spent relearning material they have already mastered or could master in a fraction of the time it takes their chronological peers. Therefore, many never learn strategies to cope with the challenges related to effort and perseverance that other children encounter throughout their childhood and later lives. In addition, the maturity of their personal outlook, which is, in many ways, similar to that of older students, may result in a mismatch, not only with the curriculum, but also with their classmates . . . They may have difficulty finding friends who share their understandings and far too often they endure not only the burden of loneliness but also enormous peer pressure to 'be like everyone else' [1].
> I believe that, in Australia at this time, it may be somewhat more acceptable to be gay than to be highly gifted. This may be why he is accepting his sexuality while holding much more ambivalent feelings about his extraordinary intellectual gifts.
> I recently asked Jan and Tony whether they have ever doubted that Rick still possesses the capacity to excel academically. They answered immediately and emphatically: The ability is absolutely still there. We have no doubt of that and whatever has happened in his life we have never doubted it. He has an astonishing capacity to learn. Recently, as part of his training in the supermarket, he did a theory of management course, which was supposed to take a year – but he did it in six months!
[1] - Robinson, N.M., Reis, S.M., Neihart, M. and Moon, S.M. (2002) ‘Social and emotional issues: what have we learned and what should we do now?’, in M. Neihart, S.M. Reis, N.M. Robinson and S.M. Moon (eds) The social and emotional development of gifted children. Washington, DC: Prufrock Press, pp. 267–288.
I was skeptical of your argument, but the specific example you came up with is so stupid I'm starting to think you're trolling.
You know what's also stupid? Making policy decisions based on anecdotes. Sure, I could invent a system that would work great for a specific kid but terrible for society. So? That's no basis for making policy decisions.
Surely you're trolling.
You are trolling, so I flagged your comment.
For a topic as personal and social as educational policy, anecdotes serve to highlight systemic points of failure.
Miraca Gross' entire book on the education of exceptionally gifted students does this quite well. You may not find the parts I excerpted convincing, but you might find other convincing. This book is titled "Exceptionally Gifted Children: Edition 2". You can find articles for free online about the research if you are interested.
I believe her work generalized outside of the exceptionally gifted range (because I'm certainly not one), and outside of Australia.
There's a good chance that either we evolved to it, or it evolved to us, over that time span. Much less so with respect to mass education.
Mass education abandoned Chesterton's fence. For reasons, but pedagogical adaptations were still lost with the abandonment of it. And now educators need to relearn and reimplement structures which fulfill the purpose of those lost adaptations. The process of reimplementing these purposes (and adapting to novel purposes that come with non-lifelong-employment) means that students are still falling through the cracks.
How AI could save (not destroy) education Sal Khan • TED2023
https://www.ted.com/talks/sal_khan_how_ai_could_save_not_des...
> with good mentors
Most people don't get mentors until the latter part of undergraduate, at best. They don't even get people asking pertinent questions as to what they (the students) want to do, or are interested in, and what problems or issues they are having. Asking these questions is not an easy process, as the asker doesn't know about the student in order to ask pertinent questions, and the student often lacks the life knowledge to give the most pertinent answers. But even a bad processes would be preferable to nothing.
I know now what guidance I should have sought. But other students aren't me. It's a sad fact that it's pretty much only those who successfully navigated the system who are in a position to give informed advice and mentorship to students today. Our teachers are the successes. Those who failed are, at best, object lessons who we never hear from.
Some pedagogues try to research successes versus failures to generate lessons on how to instruct and credential better. And I wish them well.
Of course it's not a great system for teachers either and a lot of them work really hard.
But I expect a lot of people would drop out of the system in the absence of structure.
Especially once they've been socialized through years of instruction to not be self-motivated learners. I've read about child development that most children actively want to be engaged with adult work (such as chores), but harried adults can become impatient with the best efforts of the children. This can lead to the children no longer being interested in helping, and when the tasks eventually become chores assigned to the kids they dislike doing them, and don't have the willpower to stick with doing them. I think this process would generalize to general education.
Things can be done with individualized or mentored instruction within the current system without using too many additional resources.
Sure, you get access to the library of Alexandria, but every book you need is wedged between copies of "The Big Book of distractions" and the "Encyclopedia of young girls with barely any clothes on". Some of those books play loud music, too, not to mention the books written by hacks and disguised as the book you actually want. You'd have to have the tenacity of a monk to get anything done.
It is possible for a person to be self-disciplined and learn by themselves. It is also possible, and even likely, to get lost in an ocean of distractions and/or depressed due to the lack of human interaction. As a physical person living in a physical space I wouldn't discount the positive influence of bringing your body somewhere with a purpose.
Lost in all of this is the notion that learning is a coop game, not single player; it’s more fun with friends! That’s the whole point!
At the risk of over-asserting your stance, I’d say you’re accusing introverted self-learners of throwing the baby out with the bathwater, when you are mistaking the bathwater for the baby.
The social aspect of the craft doesn’t require taking on 5-6 figures of debt (or being lucky enough to have a social structure from the previous generation that can foot that bill upfront). Most venues for education don’t even provide that great of a social network compared to the internet (elite universities withstanding). Not to mention most of this only applies to collegiate level study and not pre-k through 12th grade, which IMO is more broken then collegiate education.
As an introverted self-learner, I’d rather meet my social network while practicing my craft and take out a 5 figure loan to live on my own for 4 years and “figure it out” before moving into a big kid job. That doesn’t require our current education system.
... which is to say Neural networks are very, very good at being a social network. This hasn't really been explored so far, but it will be. And neural networks have one advantage not even parents will be able to match, never mind friends, colleagues or teachers: they're available, 24/7, 365/365 every year of your life. They never move, never take vacation, never ...
And this is before we do more with them. They can be made to be 100% reliable, to never place their own self-interest over yours, always be nice, or if you prefer that, never be nice. Etc.
However, in most classrooms it's the teacher doing the asking. "Why are you asking me if you already know the answer?" is a completely valid response. A teacher should try to avoid that and instead ask e.g. "what are you having trouble with?"
With regard to ChatGPT passing tests, IMO the root problem is the overemphasis on tests with these kinds of redundant questions (and constant grading) that have always had limited correlation with learning.
Yes, but these statements are often posed as questions though.
> A teacher should try to avoid that and instead ask e.g. "what are you having trouble with?"
The student who is sufficiently lost answers “I’m having trouble with everything!” It takes some guided questions first to figure out exactly where the problem is.
Regardless of how badly lost some student may be, a question like "do you understand what I just said?" directed at the class in general, will almost always solicit zero responses other than everyone nodding and making approving noises. This is driven by social sense of the students, involving many contributing reasons, such as "I don't want my classmates to think I'm stupid", "I don't want my classmates to think I'm brown-nosing", and "my classmates will kill me if I make this lesson last any longer than necessary".
In my experience, this kind of thinking holds well into university years - it takes some years of adulthood for people to unlearn this pattern, and take responsibility for their own education.
What is weird is that 4 years later my brother was in a class at the same teacher and the atmosphere was completely different, nobody was working on optional stuff, nobody was asking questions, nobody was offering to go to the blackboard and participate. It was same teacher, same school , same level of smart students. Not sure if maybe there is one ot two students that with their enthusiasm or the opposite can cause this drastic change.
is is a data point so it could be possible there is soem variable I do not know about.
Professional use, on the other hand, saves time, at least in some cases.
The "hundreds digit" being the third from the right, whereas the "hundredth digit" would be one hundred digits from (presumably) the left, as in "the hundredth digit of pi".
This is English as I would use it. YMMV.
Having written the above and with my hand hoverig over the "send" button, I wondered what GPT would think so I asked it:
'What is the difference between "the hundreds digit" and "the hundredth digit"?'
It answered:
'"The hundreds digit" refers to the digit in the hundreds place of a number, while "the hundredth digit" refers to the second digit after the decimal point in a number.'
An interesting interpretation that hadn't occurred to me!
“also omitting numbers whose hundredth digit is 2 or 3”
I know this kind of post is frowned on, so downvote away if that’s your thing, but I think the point could also be relevant to GPT’s performance here, although it generally seems to handle typos pretty well.
One flaw of transformer models is that, while they are able to provide good approximations of additions and small multiplications, they are unable to handle more complex formulae and functions, yet will provide an answer. They should really be fine-tuned to use an external calculator, which hopefully plugins will normalize.
This seems to work well enough for LLM's too.
ChatGPT fails it, as does Bard.
(We cloned the language part.)
OpenAI announced that ChatGPT Plus users will (finally) get Plugins in a week, so you'll be able to integrate WolframAlpha.
Or even ask it what a formula might be that incorporates an Air Fuel Ratio using the Mass Air Flow Sensor. Or even a speed density algorithm. Things like this for which I know no formulas. Or how a injection time in milliseconds is converted to milligrams of fuel.
For example, I asked ChatGPT to solve a function, in which is suggested to use Newton's method and provided some Python to do so
Copy and paste into repl.it and add some quick assertions on known solutions.
This is all stuff I learnt early in my undergrad (and quickly forgot).
Integrating a Python interpreter into ChatGPT would be awesome
(Firefox, not Chrome)