Coursera's Functional Programming - Cheating Discovered
pastebin.com
pastebin.com
The point of Coursera (to me) isn't to pad my resume, it's to learn from courses not always offered at my university.
Cheating degrades the quality of education at any level. Cheaters are trying to get accreditation when they shouldn't. While Cheating is all over in education (we had a couple of cases at my school), it is obviously going to be more prominent in a place like Coursera, where cheating is easier and looking for cheaters is harder.
One thing they could do is create an algorithm that checks answers much like what they do for journals, but again that's an extra load they "shouldn't" have on their employees.
A: "Oh hey I know how to program Scala"
B: "Ok program Scala..."
A: "Well I only got the cert, I don't actually know."
B: "Sorry I guess you're not qualified then. Why don't you learn?"
Yeah that sounds like a great interview. How long do you let them go, half an hour, a day? Do you think you can adequately judge someones abilities like this?
The university has the onus to hire credible teachers, and to maintain a high reputation. You're paying for the standards they apply. Some universities aren't worth their weight and some are. The information that they have is out there. This has not changed. Textbooks have been arround for a while. Additionally professional teachers have the ability to communicate the material affectively.
Online classes can't guarantee a level of understanding of the material. Additionally online classes will never be able to guarantee that the person who signed up for the class is the actual one that gets the credit.
It wouldn't work at Coursera's scale and for a price point they can afford.
Not in the least! You can argue that it degrades the prestige or something like that, but the education remains the same.
I mean, if there are 5000 students in the course, for even odds of it happening, 0.5 = (1 - n) ^ 5000 -> n = 0.0139% chance per student, which seems like an awfully saintly ratio to assume.
Won't they be making the problem worse by drawing borderline cheaters' attention to it (http://en.wikipedia.org/wiki/Streisand_effect)?
This kind of thinking really bothers me. They've gone to some pains to ensure their certification has some level of meaning, and now they want to chuck it all out the window based on the actions of a tiny minority.
Collective punishment is usually the product of lazy thinking.
I suppose I've always been of the opinion that they should offer the courses in a fashion similar to that of Khan Academy. I don't expect a certificate when I complete a section there as I'm there for the purpose of educating myself, not for accrediting myself. I think it's hard for Coursera to fulfill accreditation, but wish them the best if it's what they want to try for.
If learning is truly your motive and you are unable to do the assignments, well just don't do them. Why even go as far as cheating.
Also - cases will be handled on an individual basis - I guess it's not a standard sized class. There is a few hundred people signed up at least. Putting up a free course in functional programming seems like a lot of work. On top of that, trying to find people who are cheating and handling them more or less individually is a bit of a time waste in my opinion.
While I'm quick to condemn certification, I can see the other side of this argument as well. To a lot of people, Coursera represents the opportunity to overhaul a broken education system. In order for true change, Coursera must not only demonstrate that it can educate people, but that it can do so credibly and consistently.
However, I do take most certifications as mild negative signs. E.g., SCJP, CSM, PMP. A lot of for-profit certifications are worthless. A notable exception are some of the Cisco certs, which really put people through their paces. Which makes sense, as Cisco benefits more from having expensive Cisco products work well than from certification revenue.
It'll be interesting to see what bucket Coursera certs fall into. They need to jump on the cheating thing right quick, though.
I took the first AI class last Fall, and if you've never been exposed to the material, it was a very good survey of Artificial Intelligence.
I'm not sure about the actual comparative amounts, but Cisco makes a lot of money off of certifications. Of course, in the long run making them actually represent something is best for both sides of the business.
80% of Cisco's revenue is hardware. And even the service side is something close to $2 billion. I suspect the amount of money they make off of certifications is a tiny slice of that. Even if there are 10k CCIEs each paying $1500 for a yearly test, that's only $15 million, which is nothing to Cisco.
> Of course, in the long run making them actually represent something is best for both sides of the business.
That's not true for many certifiers. Take a look at the CSM certificate. It is meaningless, but popular and therefore profitable.
Present market value of a common certification - ~$100k.
Which aspect would you attempt to build a business around?
Certification is second after ability, as it is in most arts and activities, except scams, such as SAP, Oracle, Cisco or other pay-walled artificial hierarchies.
To put it differently - certification is for those, who are unable to prove/show his abilities second time.)
The decision we have to make is the level of resources to put in. We've tried emphasizing the honor code and putting resources into teaching quality. This leads to endemic cheating. We've tried cracking down very hard on detection, and this leads to massive work for the instructors, lots of busted students and a general "meh" from the administration. Finding the right balance is really tough.
I don't get it - it never occurred to me to cheat as an undergrad. But then I had UK-style end of year finals instead of homework-heavy classes and constant grading, and I was never optimizing GPA from class to class. There are problems with mega-finals for some students, but they sure make cheating hard.
The online services can't possibly defeat cheating. The main defence they can have is proctored exams at physical locations, which kills some of the online advantage, and even then they have the ID problem. The Open University has worked out how to deal with this pretty well but it's not cheap. The new online services can and should be different to this, and focus on the massive ultra-low-cost learning opportunities. It's going to be interesting to see if they come up with something really new, and not just distance-ed 3.0.
I think the bottom line is that a year after you "ace" a <3 month course you can't be caught because you can't be expected to remember all that much. You may be more suspect; but there are also plenty of innocent suspects.
A hierarchy of classes that truly builds works, but they are rare. Really, we need spaced repetition for as long as necessary to master the material, not a virtual version of a broken semester design. Then there is no point in cheating.
As a potential employer, I care more about the candidate's ethics (or lack thereof) than their ability. I could always teach them, but I could never trust them.
Coursera should just publicly identify cheaters so I can cross check against resumes or certifications. Problem solved.
But I don't think that should allow you to go and post code / answers in a public forum of sorts.
I have also worked through some iOS courses from Standford on iTunes U and I would not think twice about helping someone with one of the problems. Is that cheating?
Also, do you have any evidence that the no-cheating, no-publishing-homework policy is buried in a TOS? If Coursera hasn't made it obvious enough, then sure, people could have made a mistake. But Odersky's a smart guy so his reaction makes me imagine he made the requirement obvious.
A website that is providing you with completely free content from some of the top universities. I think it's fair for them to dictate the terms.
That being said, the onus is on the student to make sure they aren't submitting someone's work as their own. I'm not really sure if they understood what they were doing was serious (as in if Coursera was just a bunch of robot drones and they didn't give it much credibility). I know this incident legitimizes Coursera (and it's certificates of completion) in my eyes.
Really I believe all they need to say is, "Look, we really put a lot of work into this to make a good experience for people. We'd appreciate it if you took your answers offline and help make Coursera a respectable name in learning by submitting original work". The whole "We'll start expelling!" doesn't really strike a chord with me. I've been expelled before [not from coursera] for really dumb reasons.
# ALSO, they should take their Honor code out of the ToS. Put it somewhere that people READ!
I think the expelling is totally fine. Martin and his team put a lot of time into this, developing the course and the tools to automate grading and if someone is going to put this up publicly and not take it down they should just boot them from the course.
I don't understand why anyone would actually turn in the same work, though. It's not like a completion certificate from this class will get you closer to a degree or anything. I'm sure there must be some reason and I hope it's more than just "I wanted a 10/10 for me to see!". I suppose that an employer might tell someone "learn Scala and get that certificate and get a 10/10 on every assignment" but I somehow doubt it.
There it is, really obvious too.
https://twitter.com/blaines/status/255762571704020992/photo/...
This introduces a lot of problems, all of which are difficult to solve and not directly related to the fundamental vision of Coursera. If I cheat under the full name of my sworn enemy, I can have them publicly blacklisted from any employer who uses this list. Identification seems like the logical next step, but it's no easy task especially when your students come from hundreds of different countries with varying forms of ID.
If you're serious in your suggestion, you're going to have to explain how these kinds of problems can be solved. You're then going to have the same problem Google+ had with online identities. Why should one be forced to give their real identity if they're simply trying to learn, not to be recognized for it?
My thought on this comes from a good chunk of direct experience: I've taught a number of courses to undergrads at NYU, as well as having about 5 summers of teaching experience in programming and cryptography at summer camps including CTY.
Imagine all students are given a button that says "give me an A," and they can press it if they choose, or do all the course work and possibly get a worse grade. Even if you chose to do the work because you were self-motived, you'd still be tempted to press the A button to guarantee a good grade.
That's an exaggeration, but the point is the same. The easier it is to get _any_ kind of validation without real work, the harder it is to learn.
It is harder to stay motivated, and the feedback received becomes less meaningful. It is even disappointing to simply know many of your fellow students probably cheated without consequences.
I do believe in seeing answers after doing the work, but it is just common sense to avoid posting them before things are due, and to support clear communications between the instructors and students about the details of posting answers.
First of all, cheating is really easy on Coursera without having to appeal to public solutions posted by somebody else. You can get private help without anybody knowing anything about it and you can do so in a really efficient way, since there's absolutely nobody watching you.
Coursera courses are for people willing to learn something, providing certificates was stupid in the first place, poisoning the well for everybody that's there for the right reasons. The topics are not exactly PhD-level, you can probably find 80% of all the answers you need by doing stupid Google searches and even if we were talking about PhD dissertations, cheating is still possible, the only difference being the difficulty threshold.
That cheating is so easy on Coursera, "give me an A" is precisely what's going on for people that are there, not for the learning experience, but for an A.
Which is why the value in these courses is only for personal growth and personal gratification. I would never look at such certificates on a resume, because such certificates are even less valuable (from a market perspective) than a piece of toilet paper, because at least a piece of toilet paper is useful for something while at work. And no, interest in subjects such as "Scala" is not enough, because such proxies for performance can be easily discovered and hence gamed.
Also, the gratification comes from knowing that you worked and that you passed the tests. The certificate only holds personal value if this condition is met.
And in the context of Coursera, that certificate does not make sense anyway. If you follow the forums, you'll get a lot of people asking if they can still join the course with late submissions, providing reasons for why they are late in the hope that staff will make exceptions (they've taken long trips, the dog ate their homework, etc...) - which is just a mindbogglingly dumb thing to do.
Maybe the 'A' or nice cert is there for some people, but no one that I know personally cheats for the grades on coursera. OTOH, all 7 of them are currently well employed software engineers, so maybe the incentive to cheat just isn't there for them in particular. Until the signal gets gamed, I think it will be useful for hiring. Further, it's pretty verifiable if the interviewer has also taken the course. What did you think about the programming assignment about X for course Y, how did you do Z? Throw in fake X for real Y if you are particularly suspicious. If you've cheated through it, it's gonna be obvious.
So, whilst you cannot be prevented from posting it somewhere public, you can (theoretically) be thrown off the course for doing so.
I think there's a unique opportunity for GitHub here. They could give an educational license to verified students in college/high-school/on-line schools. For the duration of their class, they can have as many private repos as they like. When the semester ends, you've got a week to grab your code before it goes public, like a free account. First, it would get students used to GitHub and they would clammer for it in the workplace because they love it so much; Second, it would position GH as a great supporter of education.
I suspect this would catch on better if the repos were destroyed or stayed private. Professors reuse assignments year after year and wouldn't be too happy about public solutions to all their past assignments.
(fwiw, I use Github exactly in this way, only with public repos)
edit for link: https://github.com/edu
Maybe Coursera should host private repos?
Edit- Apparently BitBucket launched a new version of their site only moments after sharing this link. Looks great!
That's like using free speech as a motivation to tell your boss to "Go fuck himself"
I don't have any actual suggestions other then when I was in school the assignments were worth about 10-15% of our final grade because of this very reason (shared solutions). Actual knowledge was tested in other ways: presentations, quizzes, exams, etc.
Revoking the completion certificate for everyone else is NOT a way to build trust with the community at large - use those large academic brains and come up with a better solution.
So each attempt, even from the same student, would get a slightly different set of questions, and the dumbest attempts at cheating were discouraged[1].
I don't see why this couldn't be done for the programming assingments.
For example, the first assignment in this functional programming was "functional sets": empty set, singleton, contains, union, intersect, filter. It would have been trivial to add a few more (unionAll, containsAll, difference etc) and select a subset for each student, or something like that.
[0] alas, they don't seem to do this anymore.
[1] also, a "random answer + replay with proper solution" approach wouldn't work.
If someone it stupid enough not to even change variable names then they'd be caught easily. While with Millions of solutions it would be easy to get some replication, most of the time there wouldn't be any and this is when manual intervention by a person could be triggered.
This in conjuntion with your solution would work pretty well especially if someone submitted a wrong answer for another question that they haven't seen.
Use an algorithm to cross-check homework submissions, checking for structural copying. If a high correlation is found, as with human intervention, you suspect cheating but cannot prove it.
So, only for the potential cheating students, issue an extra quiz at some scheduled time, and discard the homework score. These quizzes should be all issued in parallel batches and last an hour or so, so solutions can't possibly be copied among cheaters. Make sure the quiz is actually harder than the homework, conceptually.
In other words, rather than trying to punish potential cheaters, just keep testing them until you're sure they're not cheating. This is better IMO than producing a 10x size problem set and selecting random subsets, because the number of suspected cheaters will probably be a small-ish percentage. Therefore, rather than having to write a ton of etra test material for everyone, you only have to write a few extra quizzes for a subset of the class.
I use github as a central place to store almost everything I write outside work, so it comes pretty natural for me to version code for problems I've been working on there.
Disclaimer: I'm not involved in the Scala course, so I don't know how the actual assigments were structured.
Given two resumes, one with a big bullet list of unverified skills, and one with a bullet list of some completed coursera courses, which would you choose?
Obviously coursera will not be replacing real college any time soon, but it can definitely serve a role in verifying at least cursory knowledge of basic skills.
I would accept neither as evidence of anything other than the skills a candidate advertises.
IOW, I trust "I know Scala" on a resume exactly as much as "I know Scala and completed a Coursera course on it".
The big problem with your logic is assuming that one of these two options must be meaningful. Those two resumes are either both good enough to bring in for an interview, or both not good enough to bring in for an interview. "I have a worthless piece of paper" changes nothing.
If they can't use the course certificate as a signal mechanism then there will be no incentive for employers to partner with them
I'd recommend that they get rid of the certificate. There is no way to prove that 1) the person who took the course is who they say they are, and 2) they didn't cheat. Copying and pasting from a solution online is only one way to cheat. You could also have a friend finish the assignment. Maybe some day we can get past the point where education is simply a list of achievements, and more about who you have become as a person.
Rather than trying to fight cheating (which is impossible), Coursera should emphasize the point that these courses are not resume busters but for learning.
I've taken several Coursera classes, but I don't mention them in my resume and I only care about the certificates as an incentive to complete the course, but from the very nature on how these classes are graded, it would be very difficult to prevent cheating (or sharing), barring having the exams taken in real-life locations (something that Udacity optionaly provides for some of its courses if I recall correctly).
With the ubiquity of free (public only) github accounts, I'm not entirely sure what else they expected might happen.
These courses are meant to be a segue into free courses offered online that people can use in lue of a 200k four year degree. While this form of education isn't there yet, it's use builds the reputation of these online courses.
If cheating runs rampant in the courses (more so than in person classes) it has a chance to negate the reputation of these sites. This could set back "free" education for years.
While I don't think that Cheaters should have an affect on what we can learn and take away from these classes, but it will have an affect on how much employers (like edw519) trust the courses. If they don't trust the online classes, then people who don't have a formal education won't be able to use these freely offered classes to raise their earning power.
It's sad how much a few bad apples can spoil.
"Most of the solutions for this assignment can be written as one-liners. If you have more, you probably need to rethink your solution. In other words, this assignment needs more thinking (whiteboard, pen and paper) than coding ;-)."
Is it not entirely possible that they're getting some false positives?
class Rational(x: Int, y: Int) {
def numer = x
def denom = Y
}
def less(that: Rational) = ???
def less(that: Rational) = numer * that.denom < that.numer * denom
def less(that: Rational) =
numer * that.denom < that.numer * denom
def less(that: Rational) = numer*that.denom < that.numer*denom
But it's a moot argument as we need to do a real research (or ask coursera / Martin Odersky for the data, which I'm sure HN community will be happy to run some data mining on)
The people who submit exact copies of assignments they find online should absolutely be penalized. To expel those who publish their work seems overly harsh.
This seems like pretty clear language. I doubt people who accidently used a public repo will be punished for it.
It's my understanding that, in this case, this is not possible. There are real students at EPFL who are using the Coursera system to take this very same course for credit. If some random resume-padder creates a public repository on github for doing the homework, then a simple google search by an actual student at EPFL could easily find it.
I don't see how they have any choice but to bring the hammer down.
I've signed up for this class because I was interested in learning something. Cheating would obviously not teach me anything, so why even take the class in the first place?
The Coursera certification currently holds no or little value. Maybe if they gain enough traction, someday, having many coursera certificates could have some value when applying for a job. This is not currently the case. I understand that they want to be in the position at some point in the future and that if they don't enforce these rules now, doing so later will be hard. Nevertheless, threatening to ban users in not a winning move for any startup.
A better solution would be to offer an alternative to Github, maybe even a partnership with Github. If they hosted their own git repository and used that as the submission system, they users could get source control, as well as a simple mean of distributing assignments.
This felt like a poor PR move from coursera.
Then you're violating the conduct. They ask tons of times to not share your code, including on GH. So you should either use a private GH repo or host it elsewhere, privately.
Don't whine about not liking the "threat of expulsion" when it's perfectly clear that you shouldn't be doing that.
They shouldn't have to offer that, anyone who wants a private repo for free can use bitbucket, for instance.
Perhaps in a perfect world, you could publicly post all of your homework and whoever is administering the course would be able to detect someone using your code for their homework - that is incredibly difficult in the best of circumstances, and in this case, effectively impossible. Doing something about low-hanging fruit - publicly available answers to the questions - seems like a reasonable mitigation step.
Really? That sounds like something I would not expect to get away with in a number of situations. Like, for most of the stuff I've done as a student and as an employee I have considered stuff like that to be totally out of the question.
(And in this case, it doesn't even feel that much like "my" code. It's mostly written by them Coursera course people, with a few blanks filled out by me. I'm not sure how cool I think it is to upload other people's code to publicly available places without checking with them first.)
Do people really care about online certifications anyway?
For me, the main advantage of the weekly assignments, is that it stops me from procrastinating, like I might if I were to learn by myself with a book... I don't really care about the certification itself. I guess I can add it to my resume, like the SCJP, but I don't think employers will care much.
The long story about why companies should use work-sample tests to hire job applicants, including programmers, can be found in the online FAQ I post from time to time here on HN, last posted five days ago.
http://news.ycombinator.com/item?id=4613543
Of course, because some (many?) people cheat, work-sample tests work best if the companies using them make sure that cheating is not possible during the test. If you want to get a job at a company that doesn't hire cheats or fakers, be ready to show what you know in a work-sample test.
To be a good programmer, you need to be efficient. Writing 100% unique code is not efficient. It's not even good. It's bad. Good developers research solutions to solved problems. Good solutions have been used and tested by dozens, if not hundreds, of other developers in just as many different use-cases.
So, as a DEVELOPER, it's GOOD to research and find an existing solution to your problem. It's even good to go on places like Stack Overflow and post your problem and let someone else solve it for you.
But, as a STUDENT, doing these things is BAD. Taking credit for someone else's work isn't good. And letting other people do your work for you is cheating.
So, how do you resolve being a STUDENT DEVELOPER? Do you follow practices that would make you a bad developer? Or do you follow practices that would make you a bad student?
What I see is a complication in proving that the student understands the concepts they need to understand for the course. If you just go out and copy other people's code, but never take the time to understand it, then the course is worthless. Remember that the knowledge and understanding gained is the purpose of the course. Assignments, quizzes, whatever - these are just measurement tools, they are not the purpose behind the course.
I feel that we probably just need a different way to determine whether or not a student understands the concepts of the course. Is there a way to determine a student's understanding or comprehension outside of assignments, tests, and quizzes? Something that isn't replicable student to student?
1) Figure out the number of students taking course n.
2) For the first assignment, figure out the number of questions m you want.
3) Make a question bank of size M.
4) Satisfy the constraint n < (M-m)!
5) You now can uniquely identify uploaders via the set of answers they upload.
If you're willing to kick people out of the class and mark them as cheaters, you can now do so without worrying they're trying to frame someone else.
This can be refined--maybe make it so that the question bank is large enough that several people conspiring will be unable to randomly generate a sequence of questions that's been given to a real person, so they can't randomly screw people for shits and giggles. You could also make the questions differently phrased, making it harder to Google them. Depending on how many cheaters there are, you can also just make a question bank that's large enough that any given question is unlikely to have been uploaded by a cheater, though that rapidly blows up the size of the necessary question pool if you do have a significant proportion of cheaters.
You could experimentally calibrate the question bank, seeing which questions trip students up the most, but that requires asking the same questions a bunch of times to different students, which is exactly what using the question bank was supposed to avoid! And if you want to reuse the calibrated data every year, you'll have to ask the same questions every year, which we prefer to avoid. I suppose with a sufficiently large student population an effective calibration might be possible - maybe Coursera will be the ones to do it!
In the spirit of the Platonic ideal of the Academy it seems beneficial to freely share and exchange ideas.
I am enrolled in the class, and when I got the e-mail, my first thought was "Great. Now I have to worry about whether or not my work will look too similar to someone else's, or to the solutions that have been posted."
Granted, my first submission for assignment #2 was a third of a point short of 10/10, so maybe I don't have anything to worry about.
The job of coursera is to provide access to learning and there it should stop. Coursera should not provide certificate of any kind, most people go there to learn not to become certified on anything, Cheaters will go way if there is no certificate since their purpose is not to learn.
For connecting top students with recruiters, they're already looking at contributions to forums as well as homework/test scores. It shouldn't be too hard to see who's really engaged with the material.
Getting uptight about people sharing work seems counterproductive and hopeless.
At the least, if someone contributes a ton of code for an assignment in one shot that's identical to somebody else, the professor could compare commit logs there too. Tons of code is out there on the internet and plagiarism is something that'll be tempting on the job under a deadline, too, with potentially more serious legal consequences.
I keep my code private until the class is over, then open-source it. I consider this stuff to be my resume.
the people who share? I'm sure most of them simply put in as they put any code they do on their GH account. which is by default public.
solution: people should use bitbucket, it has free private repos.
Surely the only point in doing the assignments in the first place is to satisfy yourself that you have understood and can apply the material?
(on a side note - did anyone get their CTF tshirt yet?)
The perspective employee would login and take a test based on the courses that they have completed. The score would then be sent to employee and employeer and ranked along side others who have completed the course.
Wait, so they didn't know this was going to happen?
Early questions are things like: if membership of a set is a function mapping between an integer and a boolean, how would you define the membership of a singleton set? Modulo the spaces and variable names, every student's answer was probably the same!
So detecting who has copied their solution is probably difficult.
that's kinda amusing. [edit: thanks for the reply (have an upvote); missed that. that's more reassuring.]
"For those students for whom it's been discovered have uploaded similar or duplicate solutions, cases will be handled on an individual basis. Though beware, that cheating in any capacity not be tolerated."