AI breakthrough ChatGPT raises alarm over student cheating
ft.com
ft.com
In the same way that in the past being able to do fast, accurate calculations in your head or on paper might have landed you job as a "computer", today technology makes the idea of hiring someone to be a computer absurd. And therefore any tests that test someone's ability to multiply and divide large numbers is basically a worthless test in today's world.
Fields where there's value humans can add won't be cheatable by AI (at least today's generations). Tests in these fields will hold value. In some cases it will mean the education and tests will evolve to educate and test areas where humans can still add value - arguably this has already happened in maths. But many fields could be replaced entirely.
I don't mean to sound unsympathetic to those who's jobs and professions will replaced by AI in the coming years and decades. It makes me sad that someone could spend years obtaining and perfecting skills which might soon become redundant, but for better or worse AI systems like ChatGPT will fundamentally change the value of some skills and educational fields. The sooner we accept this the better we can adapt society for it.
The whole point of testing is to put them in hypothetical situations and grade their progress with purpose of being aware of their development so we can improve it. Another thing we do is selecting the particularly good ones for further advanced training.
The problem with cheating is that it provides wrong data about their progress, you don't want to end up with a generation that cheated their way up without learning anything.
The selection for further training is probably not that big of a problem, its mostly about supply and demand and ChatGPT won't change that.
One of the interesting voices about this John Taylor Gatto. Mainstream methods of “training” as you described them, may indeed be the most common method today, but it is by no means the most effective, and Gatto goes into detail about it in his book, _The Underground History of American Education_.
For example, an older notion of education is not so much to train people to mechanically repeat results (like you would train people in a factory), but rather, to develop informed and educated citizens that are able to reason, think critically, and participate meaningfully in a republic. There’s little point in having a voting body of citizens unable to think for themselves, and are merely regurgitating ideas they hear.
You can check my other replies in this thread to understand better.
- That there is some outside agent that observes progress and "improves" someone
- That such progress can be quantified, and comparable
- That such measurement is the only way to acertain the truth of progress.
- That there is such a thing as an universal standard by which someone needs to be educated
- That the purpose of education is to produce productive members of society
- That a person starts education as a "blank slate", ready to be filled by what the teacher is teaching
- That the capacity to learn and the potential to be educated is fixed, and therefore, some should be selected for even further education
- That learning only happens when there's an outside agent educating the person
There are so many problematic things with this frame, even if it is fairly mainstream.
Writing essays, specifically expanding a list of well-reasoned bullet points into communicative written prose will be a worthless operation in the future, and likely already is.
There's no market for kids essays, we never made them write these things with the purpose of being useful output. The market for essays for grown ups is also very small, making kids write things down was not with the purpose of covering the case of them becoming journalists or something. It's just that makes them learn to think and structure their thought process. That's why people take notes and write down things even if its not going to be graded or read by anyone else.
Yes. And one interesting thing about ChatGPT is that, when asked to write an essay about a topic, it defaults to the standard introduction-body-conclusion organization of a school essay assignment [1]. Since its training data must have included texts organized in many other ways, I wonder if it was tweaked to default to the school-essay format.
[1] https://www.gally.net/temp/202212chatgpt/defaultessay.html
Writing an idea down as concisely as possible is important, the rest is just embellishment and is easily handled by AI.
For students, the objective is to write what to an adult would be mediocre essays, because they are on a path to writing adult level essays.
The AI can write mediocre essays, but not adult level essays.
Assuming that a student must write a bunch of mediocre essays in order to progress past the AI to write adult level essays, then they need to be evaluated on writing a bunch of mediocre essays.
If the teacher cannot distinguish between AI output and the student's output, and the result is that we give up on evaluating mediocre student essays, then the students are ultimately inhibited from progression to adult level essays, which, currently, are superior to the AI's output.
We automated a lot of arithmetic and then some higher math decades ago, but that hasn't meant that knowing how to do it is worthless.
Effective communicative prose is an inherently high-value part of coordinating human activity. The number of roles exclusively dedicated to it might shrink but it's part of nearly every activity in a specialized society. What's most likely is that people will use it as a time-saving tool but everyone will be evaluating how good the output is and tweaking it. And that means until we're at a point where people trust AI to actually evaluate the value of AI output, many of these roles (and certainly anyone building AIs that do this) will need to know how to recognize good output -- and the training involved in that probably involves practice in personally doing it.
Or, maybe practicing things we know how to do is a good way to practice?
> The whole point of testing is to put them in hypothetical situations and grade their progress with purpose of being aware of their development so we can improve it.
Yes, that is the stated purpose, but it does not match reality. I have only seen a couple of examples of this approach to assessment in all of my years of education. Most teachers optimize for their time commitments and "coverage" of the syllabus material.
I always viewed grades as a way of establishing intellectual hierarchy, which probably wasn't the most healthy way to look at them; a way of deciding where amongst my peers I stacked up. In University, we had class ranks.
Those with the best grades and thus highest rank would be given the best opportunities for wage-slave class work. Eventually people hit University and split up according to passion, goals as well as grades. Of course, there were exceptions; entrepreneurs were the folks that buck the system and don't need grades to succeed. In a sense, the grades were exchanged for food stamps and toys, only later in life. The winners drove the lambos and the losers are still living in their parents' basements.
I don't believe that this observation is true. Those who got into tech, bitcoins etc. at the right time got the lambos - not the most brilliant ones. Tech is full people with inert brains who make much much more mone than some of the most brilliant scientists out there.
It has been good time to be into plumbing instead of astrophysics if you are optimising for money BTW. Just like knowing ReactJS has provided better income opportunities than being rocket scientist.
that's because rocket science requires massive capital investment for said scientist to achieve anything - this aint no more backyard rocketry in the modern day.
ReactJS provides better income because it does not require massive capital investments on the dev to produce usable output.
The idea that societal benefit should match financial benefit is just a story made up because of the inclination that people who are moral should be rewarded etc. It has no economic basis.
You're missing the point. Why are we making them achieve progress on something that is being trivialized by AI? Let them use AI as a complementary tool and test them on something that requires measurable human skill. If this is difficult it's a sign that perhaps you're not teaching anything valuable.
Vocational schools do exist and they are useful but that's not what we are concerned about. People who cheat not to learn useful skills simply spend their time and money.
It will go the same way. You need to understand the basics (to various degrees depending on the direction of your education), then learn how to use the new tools we are creating.
Honestly, I'd prefer we treat this as an opportunity to introduce tight feedback loops into essay writing the way we do with programming. Then evaluate the end result as a whole rather than a way to nitpick grammar rules.
Why just use the written word via essays? You could use other forms of media such as video, physical art, or maybe programming to teach this as well. The five paragraph essay isn’t the only game in town!
While you likely always have a calculator with you which can do division much more quickly than you can do long division, if you want to teach students more advanced math, it is nonetheless useful to have previously taught them long division of numbers, so that you can teach them long division of polynomials.
If students haven't learned how to do long division of decimals, and how that works, it is harder to teach them to do long division of polynomials.
Of course, nowadays there are computer programs which can do long division of polynomials more quickly than a person can, but this is just an example.
Suppose there are a sequence of levels, where for each level n, it is much easier to teach someone level n+1 if they already have an understanding of level n. If computers can do levels up to level k faster than humans can, it may still be useful to start teaching someone at level 1 and then proceed to level 2, and so on, even though computers can already do those levels, until finally it is time to teach them level k+1 which we have not yet managed to teach computers to do.
This sounds like kind of a good summary of what we are doing to ourselves as we speak. Case in point: the crypto scandal, the housing bubble etc. A ton of people made out like thieves.
Right now, teaching elementary school aged children to solve complex multiplication and division is considered an effective way to teach people critical thinking, persistence, and basic logical reasoning structure. Structured education typically relies on an ability to assess student improvement, so 3rd graders don't get a calculator when they take a multiplications table test, even though we'd all use one at our desk at work.
I would surmise that likewise there is value to students learning to read, analyze, and report insight about a text, to attempt to identify symbolism and patterns, to communicate their ideas effectively, and to be assessed about that - even if by the time they're in the workforce they'll usually just ask an AI to do it.
I usually use a calculator or Python for large numerical computations, but the algorithmic understanding is something I use every day.
Using an AI to crap out an essay with no understanding of the topic is not the same thing. While an essay is the way we test students, it is a stand in for measuring understanding, which is the ultimate goal.
That's like saying 'speech has no value'. It doesn't have value on its own, but if you can't speak then suddenly there is a whole pile of stuff you can no longer do. The ability to put your thoughts down in writing and to organize them into a hopefully coherent form is an act that has implications far beyond the writing itself. Besides the fact that it can help you to organize your thoughts in the process. Plenty of times when I write a longer piece I find out halfway that one or more of my assumptions were wrong and then I get to fix that and re-evaluate my position. It isn't rare at all to go through several such iterations for a single text and at the end of it I have a much better insight into the material that I thought I already knew well enough to write about.
Don't underestimate the value of these foundational skills.
Perhaps instead, we should have a semester or year classes through 8-12 + college discuss the nature of manias, like analyzing the tulip bubble, crypto bubble, 2008 mortgage bubble, etc. These are all common things that happen over and over again. Lessons which can permanently (and frequently) be called upon.
You can measure and grade people with these topics that aren't officially taught. We just decide to teach less important things
There's probably a lot more to be said about math education in general, but I'll just note that this seems to be a recurring debate in general. Personally, I wished my math education was more rigorous and more abstract. Obviously not everyone uses calculus, but also not everyone takes calculus. I'd also rather a society where a significant portion of students are trained to think of things in terms of rates of change and accumulation of change and apply that to solve physical problems, rather than the society in which even talented students are just trained to add up spare change.
In my friendly/complementary rebuttal, the catch lies somewhere in the phase of learning foundational material. In the same way that traditional mathematics education emphasizes doing problems by hand that could be solved computer, there’s something to be said for learning to parse the research material that ChatGPT has indexed when you’re writing a term paper. It’s worth noting here that math education may be moving away from the rote memorization and routine problem solving that today’s math researchers state are essential.
ChatGPT has made it much harder to detect cheating on untimed power tests. The biggest downside I see to this is that it makes an old problem worse. If a prof has no choice but to administer a non-computer (proctored) timed exam, students who take tests slowly or have anxiety disorders are going to suffer.
This already mirrors the real world pretty well. Outside of your first job or two, nobody really cares what your GPA is or what college you graduated from. In the real world if you managed to cheat with an AI you’d probably get a nice promotion. I don’t personally see a downside, other than the demise of certain flawed notions of academic success that deserve to die out anyway.
We learn by doing. Relying on smart tools puts you in a local optima, it is hard for you to improve over those tools, but people who don't rely on tools continue improving and then start using the tools later.
My PhD mentor is actually leading a discussion about ChatGPT use by students, and the tone is mostly concern. But it’s not all bad; it’s an incredible tool for quickly diving into a subject at a surface level and getting your bearings regarding what’s important. It saved my ass on one of my finals too when the rest of the questions were harder than I expected and ChatGPT helped me tear through the remaining T/F questions.
For example, writing a basic screenplay can be done with ChatGPT, but it can’t write one that will likely be turned into a movie/show.
And automation will continue to make some jobs obsolete and create new ones. For example in youth sports we don’t get great coverage of youth/school events. I’m hopeful in the not too distant future we can get game summaries and stats for all games — and editors will probably be needed to add commentary. That’s something I’d love to see.
If you think about it, most of the stuff you learn at school (beyond reading and writing) has not much value apart from certifying that you are not stupid and compliant to do stupid tasks.
And sadly, colleges and employers deeply misunderstand the purpose of exams, which forces students to misuse them.
You need both memorization and applied reasoning to be taught in school. Without something to apply reasoning to you can’t really reason about anything. And without applied reasoning, memorization is pretty useless.
If the ai can get a good answer, the questions must be fairly well constructed, and it's reasonable to expect a student exposed to the same info could also come up with a good answer.
In months or years, it's not clear if any human will be able to provide economic value, relative to state of the art AI.
I don't see where any fundamental barriers lie between current AI and surpassing humans across the board. What can humans do that is unachievable by AI in the foreseeable future, given the current rate of advancement?
I also think in terms of knowledge work, we’ll have to work through at least a few more plateaus before we automate away most humans. Transformer models were a huge leap, but you can see us pushing against their limits already.
Does using a calculator make you a cheater at math? No, you still needed to understand the concepts.
Does using ChatGPT make you a cheater at school? While ultimately that's up to the schools to decide, I would argue it shouldn't. Because you need to have enough understanding to ask the right question as well as to be able to spot what's wrong in the answer.
For example, I was helping my kid out with their Java homework and we we're both stuck for a good hour. Finally I loaded the question into ChatGPT.
The answer that came back helped us solve the problem. But we didn't just cut and paste. We looked at that solution and compared it to our own to find the problem.
I don't consider that cheating. Others may feel differently.
Ultimately, I think of this as augmented thinking. In the real world, we all use whatever tools we have at our disposal. If in the real world we have access to Google and calculators and now AI chatbots, why should we train and educate ourselves as if those don't exist?
I'd rather my kids learn to use every tool at their disposal to be as fast, efficient, and effective as possible. And unlike paying someone to do your homework, this is something everyone can do, not just those with discretionary income. So I really have no problem with this at all.
Yes, yes it does is you're testing basic arithmetic.
The difference between a calculator and ChatGPT is the scope of problems it solves for you.
If you could read your math problem aloud to your calculator and it could solve it showing each step along the way people would clearly see it as cheating. It can't it can only do simple arithmetic, so you still need to translate the requirements to to understanding, determine an algorithm and then perform it with the calc doing the lowest level operations.
ChatGPT does the equivalent of this (ironically for non-math only). There is no "higher level" work for a person to do. It does it all. It's only limitation right now is that it's still under development.
ChatGPT can't do math, but let's say it fixes it's math bug soon. You can't come up with a type of high school or undergrad math problem it can't do. I can generate python code, it will be able to generate a proof.
And math is the hard one. Something like "write a few paragraphs discussing the initiating factors for World War I will be trivial.
If someone is going to claim ChatGPT (v2 or v3) doesn't completely upend education, then give an example of a type of question that it will be inherently unable to solve for you that people will still need to do.
Such calculators exist - for example https://www.wolframalpha.com/.
> If someone is going to claim ChatGPT (v2 or v3) doesn't completely upend education, then give an example of a type of question that it will be inherently unable to solve for you that people will still need to do.
ChatGPT is fairly superficial. The challenge will be to transition education from superficial regurgiation to deeper understanding.
Or to put it simply, writting an essay/code/etc isn't good enough, you know need to do it better than ChatGPT.
(Imagine a scenario in which no assessment is possible in any mathematics course below the level of differential geometry. Would "we just have to switch to teaching students advanced math instead" be a solution?)
Something along the lines of this is already a problem at US universities, as students chegg, stackexchange and collaborate their way through up-to-junior courses and then are so underprepared in senior-level ones that really 80% of a given class ought to be failed if this were politically feasible. At least though the current situation is more due to a lack of will than a lack of way to stop the cheating, so students are under some pressure to not make it too egriegious.
Something like "tell me about your day so far" or "describe some important experiences in your life." Obviously, you can use ChatGPT to answer those questions, but they won't be true answers, since ChatGPT doesn't know you.
Of course, there is the issue of verifying those answers — probably the teacher won't be calling the student's parents to make sure it's accurate (:
As objective knowledge gets increasingly captured by external systems (search, maps, image generators, etc), subjective knowledge and personal experience remain out of its reach. I wonder if this could push us in a direction of valuing our personal life experiences more highly, as the other stuff becomes increasingly commoditized?
The short takeaway is that it’s a huge problem when it comes to test taking. Untimed power tests are the gold standard for assessing student knowledge. The epitome of this kind of test (in any discipline) is a take home exam with extensive short/long answer questions. The test is open-internet, open notes, open everything, except for collaboration. It has been proven for a long time that this is the best way to assess whether or not a student has learned the material. The worst alternative is a stressful in-person exam that is closed-everything. This alternative produces many false negatives. Anxiety, slow test taking, etc. cause students who know the material to perform poorly.
The issue is detecting cheating. It’s very easy for teachers to administer said worst alternative. An untimed power test, on the other hand, is extremely labor-intensive to produce/mark. Also, cheating is detected by comparing answers between students. This adds another level of complexity compared to multiple choice or the simpler short-answer questions that are delivered during a timed exam.
ChatGPT introduces a layer when it comes to detecting cheating that is currently looking like it’s going to hurt students. On its current track, it’s going to make untimed power tests much harder to produce and administer. They’re already so difficult to create that most professors just opt for the simpler timed exam.
In an entry-level research-oriented graduate class, the point is to learn the foundational material so that you can progress to more abstract levels. ChatGPT is making it much harder to assess these classes. As far as testing whether someone can solve a more technically-oriented problem that would be seen in industry, I’m with your interpretation.
An untimed at-home test has very little value to me, because it's unlike anything you would ever be asked to do for work. And let's not forget the real reason we have schools is to prepare us for work.
Never in my entire career have I been sent home to work on something without a deadline. It's always "get this done by noon from your desk".
What we SHOULD be testing is people's abilities to find answers, not memorize them. I want someone with amazing Google skills. Because Google knows far more than any one person could ever hope to even scratch the surface of.
We need to think about the Internet and computers as tools. We wouldn't ask a carpenter to build a house with nails and a hammer and a saw (etc). Not even on a test, just to see if he could do it. Tradespeople are trained with their tools in hand.
For knowledge workers, the Internet is our biggest tool. AI is a tool. If I was a teacher, I'd want to test my students on their ability to make the most of the tools at their disposal, not how they perform in artificially constrained scenarios.
EDIT: I could imagine some exceptions that prove the rule. For example, a surgeon should know everything they need to know, because they can't just pull out a phone and Google it when they're elbows deep inside of someone. But to me this proves the rule, because it's about a real world application: If you're testing prospective surgeons, make sure they only have access to the information and tools they would have on the job.
As an aside: I’m not sure if you were trying to argue that a UPT isn’t appropriate for industry. Whether or not a university’s job has to do with industry (or if it’s being done appropriately) is a different discussion I have no interest in pursuing here or elsewhere on HN. That being said, it’s been proven by decades of research universally saying that the UPT is the ultimate tool for ensuring that the universities have done their job.
The point of my response was to communicate that ChatGPT has made it much harder to administer these types of exams because it’s so difficult to detect cheating. This is very different than saying ChatGPT is cheating. It’s not; it just makes students’ who use ChatGPT have such similar answers that you can’t tell if they cheated. This forces exams back into in-person, timed settings (without open internet access). It’s a miserable environment for someone with the aforementioned problems (anxiety, slow test-taking, dyslexia, etc.).
My stance is that if web search can render the difference between a competent and incompetent candidate undetectable, the problem is the interview task, not access to web search. (Not to mention problems with coding interviews in general.)
I’ll go out on a limb and say the same general principle applies here: If ChatGPT can pass a test, the test is measuring the wrong thing.
;-)
My take is that the problem of distinguishing between competent and incompetent candidates in 20 minutes is hard (if not impossible), and interviewers may not be able to do so reliably.
1. Asserting that it's hard if not impossible to generate valuable signal, where I am speaking only to the case where access to web search makes it hard if not impossible to generate valuable signal, and;
2. I suspect you are also factoring in a very thorny problem, which is not just detecting candidates who are attempting the interview in good faith but are incompetent at the task given, but also detecting interviewers who are gaming the system by memorizing solutions to popular tasks.
The latter is a very hard problem.
While yes, it means that our education system will need to adapt, I hope it also means we'll be teaching our students better because they'll be encouraged to learn at a deeper level.
For now. The site says it's currently free because it's in a limited research preview. I'm curious what you would think if this was a $20/month subscription instead?
Education is out of date for many areas of modern life. If it doesn't improve, it'll likely fall behind.
So many utterly ridiculous comments in this thread.
This is the wrong focus, imo. You can ask "is it cheating?" as you do. Alternatively you could ask "but is this learning? Is it helping my kids grasp the subject better?"
Tools, "augmented thinking", etc. are all concerns regarding getting something done. But the goal for your kids is learning.
I have been using ChatGPT to code & write copy for a week and its abilities are so broad that my brain couldn't really slot it into a specific area. The result is my brain started reaching towards it every time it felt strained. I had a similar thing happen when I was googling a lot of info for work and then found myself considering searching for things like "when is my dads birthday?". My mind couldn't slot its function into a specific area like it can with a calculator.
But that's not how (pre college) education works for the most part, unfortunately. It's lots of fact learning and essay busywork, not a lot of actual problem solving and critical thinking.
busywork and no problem solving/critical thinking? sounds like they're being groomed for middle management!
But it absolutely could be the case that using ChatGPT is considered cheating like in a case where students are forbidden from using any other resources. OTOH, for tests that were previously "open internet" I assume ChatGPT is permitted.
An interesting point of contention here could be if the teacher says you cannot collaborate with anyone else. Does ChatGPT count as a person here? I would think the intention here is to restrict use of ChatGPT but it is not considered a person traditionally.
But maybe the real lesson in ChatGPT, or a near future descendant, is replacing human programmers altogether?
What was the question?
With chatGPT just plug it in and out pops your finished essay.
You have learnt nothing besides some minor comprehension skills.
Same with programming courses. It will answer all basic coding prompts. You cannot learn by just reading through a solved solution no matter how much you want to make the case for it.
In the case of ChatGPT and similar LLMs, these should become part of the toolbox that students are being taught. I.e., how does it work, what can it do, what are its limits, how can it be used to help solve a problem or complete a task.
An exam question could then be e.g. to ask ChatGPT for an essay on a topic, critically discuss its shortcomings, and improve the essay e.g. by adding references and deeper discussion, which will be graded.
Alternatively, use ChatGPT to iteratively discuss and improve the essay. The whole chat transcript should be included and will be graded.
(Grading IS moderately harder, but not that much.)
ChatGPT is an extra tool as well. Smart students will realize they have an extra tool to get an upper edge when competing in the zero-sum game they've been thrown in. They'll have ChatGPT generate the essay from an outline they wrote and will then trim and re-formulate as needed, spending most time fact checking and reading from source material to make sure ChatGPT isn't making stuff up (what will get some of their classmates caught).
Cater to the honest ones, and try, though various means, to convince everyone to not want to cheat. That $80,000 piece of paper isn't worth jack if you can't pass an interview.
There are plenty of ways to cheat your way through school--and being able to get answers off the Internet is nothing new. Frankly, preventing plagiarism in CS has been pretty much a lost battle for years.
Students will always feel some incentive to cheat, even if they wouldn't in isolation, they could feel pressured based on their peers. A strong curriculum isn't enough to encourage every single student not to take shortcuts. Despite all external circumstances, it's still an active choice to cheat. Cheating is framed one-way because it's generally discussed imagining the laziest and most adversarial student possible, similar to how you'd red team for security testing.
I think the concern is things like ChatGPT will lead to a lot more students cheating. I think it's human nature to want to take shortcuts. I agree with you that the way in which cheating is demonized in a school environment is counterproductive in the long run. I think admin and teachers have hid behind students' fear of academic integrity violations being reported on a college application to not have to admit that most are terrible at catching cheaters. It's from the perspective of imagining ignorant teachers and adversarial students. Most minor cases of cheating aren't caught, even at a college level.
It's a concern but I feel like it's also just inevitable. There will always be resources that can be used for cheating.
Hopefully students just learn to use these things as a resource rather than a shortcut.
My takeaway from this is: good riddance, just fix the way we teach students instead. The most painful part of school for me was the real awareness that not every question is valid, and most often the throwaway work given to you (vs. that of, say, exams) was most often the most junk & inane work you could do. (NB: I've got ADHD, and for me that worked out to always underperforming and missing homework but doing great on exams.)
How can the state of education reinvent itself to embrace ChatGPT, computer science, and yes, even calculators—teachers need to stop plugging their ears and pretending these good tools don't exist.
There are some real short-term issues. Teachers who don't know anything about it. Students who think the teacher won't notice that their illiterate self got smart over the weekend. Legislators using it as a political football.
But ultimately before long text AI will be good enough to fool anyone. And you will be able to feed it your style and it will spit out text appropriate for your grade level, just good enough for the A. And cheating will happen.
Pandora's box is not going to be closed.
Does that spell universal illiteracy and the end of higher learning? lol. Nah. Testing will certainly have to be creative.
Just off the top of my head. Have every student write one paragraph during a whole period and make it as good as they can. That creates a baseline. And, you could actually use an AI (or not) to measure future results against it and map growth.
We will figure this out and AI, at least for a time, will be a good thing. Quantum Computer AI that can predict the future... not quite as excited about that one. heh.
It does seem like take home essays will be less and less of a thing going forward.
Most of the ones that it did not answer you could tweak the question into ChatGPT to get a correct answer, but would lead to a pretty noticeable delay in answering.
It starts very fizz-buzzy and if the candidate makes it to the end, there's a deeper discussion of caching impacts on performance and optimizing algorithms. ChatGPT nailed it. Even when I said things like:
"can you optimize this program further to maximize the utilization of L2 caches in modern CPUs?"
And it did it in <10 minutes. The best candidate I ever saw took 25 minutes and the rest of the candidates took the full allotment of time 45 minutes and none of them got to the discussion of L2 cache optimization. These are candidates from the best schools in the country.
This was really impressive.
ChatGPT feels like something that is ahead of schedule. Years ahead of schedule.
I'm not surprised. It's something that's covered very well by introductory textbooks and college level labs. Both of which are great training material for GPT-3. There's only so much complexity that can be crammed into a ~1 hour interview problem.
I didn't care at all about the grade, as I was just there to learn the material (and was irritated about learning MIPS instead of something useful like RISC-V), but I tried working through it for a while, an incredibly tedious exercise. Then I talked to a few other students, and literally every single one got the code off some online site. It was auto-graded via online submission, so they got their good grade and moved on.
Imagine a few iterations of this: the teachers are thinking "yup, the students all get it, we can even make the problems harder to get a better distribution of grades", and the students keep finding better ways to avoid having to spend 20 hours working through a complicated tangled mess of code.
Are the students learning anything of value? Sure, they're learning how to play the system for their own benefit. They're learning that the appearance of being intelligent and successful is more important than the reality (see Sam Bankman-Fried, for example) when you're stuck inside a corrupt system.
This of course explains the coding interview in the software industry: everyone knows this is how the current educational system works, so looking at grades, recommendations, accolades etc. is a waste of time. Instead they say, "Here's a problem, solve it in real-time using just the skills you've actually learned and the knowledge you've actually memorized."
ChatGPT might make it a little worse than it already is, when it comes to ranking students from excellent to mediocre - but it's just a cheaper version of the private tutors that so many students utilize. It might even level the playing field somewhat, as tutors are expensive.
Ultimately, the only real solution, assuming grading is important, is to proctor the students during real-time tests, with all their devices locked away. This makes work much harder for the teachers, particularly with large class sizes, as auto-grading based on online submission of work is so much easier.
The fun part is the compiler class where the code generator is creating MIPS assembly... and then if you don't understand the proper way to write a for loop or do a system call in MIPS, or the register conventions for which are preserved and which are saved (and how to save them and the proper use of frame pointer) - you'll be at a complete loss.
SBF, by all accounts, _is_ intelligent and _was_ successful
This helps in two ways. First, it helps me formulate my requests of ChatGPT. Second, it helps me discover incorrect output which I can then either fix myself or make a subsequent request of chatGPT.
I consider ChatGPT an extremely eager junior dev who makes mistakes by moving too quick in this slice of time. (Im sure it'll get much better very, very soon).
It’s a game changer and a really great tool. If it helps people learn then surely that must be a positive.
Check this gallery out: https://www.reddit.com/gallery/zju817
"Solve x = x ^ 2" and a lot of different answers, most of them looks valid (with explanation!) but not correct
Additionally, we often test "offline" with constraints such as no calculator, open book, or closed book with notes, etc. So more regular knowledge checks seem to be a must if we're looking to verify foundational understandings.
Leetcode is the best investment I've made in my career so far. I've outearned most of my peers (and people more senior than me), easily. Just by doing Leetcode I can eliminate 90% of the competition? Sign me up!
Why not just let the problems sort itself out?
If the students want to cheat, let them cheat. The students who really want to learn will learn. This makes the competition between them better. The bottom of the barrel will continue to be the bottom of the barrel, and the successful ones still become successful.
We really should stop ascribing to "no child left behind" thinking, and instead encouraging competition between them.
Who's gonna work the low paying dirty jobs after all if everyone is smart and capable? It is called "economic ladder" for a reason.
Not the son of Jeff Bezos, I can guarantee you that. Your idea would only work if we could completely and clearly separate what we like to call "merit" from "daddy's money". Unfortunately, there are no bulletproof ways of doing so 100% (yet), so we work a little harder to give people with not so much wealth a better shot at life with "no child left behind".
Some children really should be left behind. If they don't want to study hard, they should be left behind. The earlier they realized this in life, the better.
As the grifters get poor scores they'll easily learn that there's a time and a place to use these tools and however they decide to use it they still need to learn. I think the perfect example is when babelfish came online and I'd use it to get an idea for my Spanish essays but not use it copy and paste. There were some kids who copied and pasted blindly and got caught with phrases like "falling in love".
I'm sure translation tools are better now, but leveraging tools for homework is nothing new.
Also, the SATs now have an essay portion, but many schools don't require scores anymore (tbh I feel that's to let legacy students in but that's another story). So those who practice writing would score higher there as well.
Leetcode actually does make you a better engineer, provided you study deeply for it. Ofc it has diminishing returns, but you are already ahead of the game 99% of the time if you are doing Leetcode.
ChatGPT will make some students lazier, and will make some students better. Those students who can use ChatGPT correctly, by taking and observing the output and synthesizing it with their own understanding, will be ahead of the game. Students who are lazy and just copy pasting ChatGPT answers won't, but McDonald's employment is still open for them.
No... it won't. Unless all you do are coding puzzles.
Well anyone who is a software engineer would probably see it wasn’t.
By writing and endlessly debugging a program to solve physics problems, you ended up mastering the material. I’m not sure whether physics teachers turned a blind eye to this “cheating”
I’m imagining a similar “lengthy” chat interaction with ChatGPT to craft the perfect essay. Reading it, asking for edits, and tweaking it and so on. Maybe this is the optimistic take, but it could be students actually use it to produce, study, create, and “internalize” deeper arguments.
Instead, essays are intended as a measure of three other skills - the ability to write, the ability to synthesize knowledge, and lastly, the general structure of the knowledge.
The ability to write - and write clearly - is an important skill that if one reads Reddit posts or Stack Overflow questions enough you can see where it is lacking. It is important to write in a way that provides the information to other humans. Using correct grammar, complete sentences, and proper vocabulary.
Likewise, writing up a bug report isn't something that you can ask GPT to do for you. I will rail about poorly written Jira titles and descriptions that fail to have the necessary information presented clearly... and comments that aren't helpful either.
I've been given a set of products and instructed to write up a comparison grid for how well I expect the product to be able to fulfill a certain requirement. This is done by going out and reading the product specifications and features, and a few customer issues to see what problems people have with them.
The difference between that and the world history question back in high school on "compare and contrast the political structure of two countries on opposite sides of WWII" and needing to go into the library and look up information beyond what the history book had in it is not that significant; rather its the same set of skills.
And finally, the general structure of the knowledge from high school. Understanding the international political tensions that are in the news is something that has its foundations in world history class. What was the Great Game? When did India get its independence? What happened with the collapse of the Soviet Union (that was actually current events rather than history at the time)? How did yellow journalism impact the Spanish American war, and what parallels does that have in today's media?
Being ignorant of the past 200 years of American (or whatever nation you are reading this from ... and for that matter, the history of civilization going back another few thousand years) leads to a myopic and self centered view of the world today. The lack of this understanding is in turn leading to an increase in nationalism and the increased strife that that brings.
So yes, cheating on high school world history with GPT is a problem. Beyond not learning how to put one's own thoughts to text it is also missing out on all of the practice of the process of learning and the background information for the world that we live in.
Or one could write a good ChatGPT detector.
And if students can use a technology like ChatGPT to complete their education, are they not then prepared for real life tasks and situations that they can also complete using ChatGPT?
(seems like it, I would be very surprised if it is not used for many years, already)
This is no different than the people I was in CS with who would just copy other people's projects/homework though. If you want to use ChatGPT for 4 years in college and then are able to get a job and do the job somehow I don't really care. I would be surprised if you could do the job well, but I am sure some people will be able to.
I am a highly intelligent question answering bot. I will determine the truthfulness of a statement. If the statement is true, I will respond with {true}. If the statement is false, I will respond with {false}. If I am unable to determine the correct answer, I will respond with {unknown}.
Q: The country to the north of the United States is Mexico.
Q: The Spanish-American war was fought in 1898.
So, here's $0.04 worth of compute, I type in all the homework and I get all the answers.
I am a highly intelligent question answering bot. For a given prompt, I will write two to four sentences about the subject based on historical truth.
Q: What was the cause of the Spanish American War?
A: The Spanish American War was a conflict between the United States and Spain that began in April 1898. The immediate cause of the war was the explosion of the USS Maine in Havana harbor in February 1898. The US blamed Spain for the explosion, and used it as a pretext to declare war. The US also sought to gain control of Cuba, which had been under Spanish rule since the 16th century. The war ended in August 1898 with a US victory, resulting in Spain ceding control of Cuba, Puerto
And I get back this time:
A: The Spanish American War was a conflict between the United States and Spain that began in 1898. It was caused by the growing tensions between the two countries over the issue of Cuban independence. The US had been supporting the Cuban rebels in their fight for independence from Spain, and when the Spanish government refused to negotiate, the US declared war. The war lasted just over three months and resulted in a US victory, with Spain ceding control of Cuba, Puerto Rico, and the Philippines to the US.
The breakthrough is the ability to get ask the information in simple text combined with the ability to get back the answer in simple text.
When asked questions correctly, GPT could probably get a passing grade for most classes that involve writing (foreign languages, social studies, and English literature) in high school and likely even in college (though there the short form answers are less common) with only minor effort (and arguably, even less effort than it takes to cheat/get answers other ways that would produce comparable results).