For Online Courses, Questions Over How Success is Measured (2014) | Hacker News Reader

For Online Courses, Questions Over How Success is Measured (2014)

texastribune.org

171 pointsreedwolf78 comments

I am always amazed when people make comments like this:

"The results of the University of Texas at Austin’s first full-semester foray into massive open online courses, or MOOCs, are in."

"Professor Michael Webber’s “Energy 101,” which had an enrollment that peaked at around 44,000 students, had 5,000 receive a certificate of completion — about 13 percent of the roughly 38,000 students who ultimately participated."

So let's unpack this a bit. Professor Webber created a class called "Energy 101" and processed 5,000 students through it to completion. Your typical 100 level undergraduate class might have anywhere from 50 to 200 students in it.

UT Austin this year had 8690 freshman total.

So assuming the largest possible class of 200, this professor in one semester taught the equivalent of 25 semesters of 200 student classes, in one semester.

Why should we care that 32,000 people signed up and then said "Woah, really don't have the time to commit to this right now?"

It is one thing to move to a campus dorm and start studying some topic. It is quite another to click a few links on a web page and say, "Yeah, that could be an interesting class."

Also, taking but not completing MOOC shouldn't count as a failure. I've enrolled in ~8 courses on Coursera and never completed any of them, but I still gained a lot of value from what I learned in those courses.

Exactly. I commonly enroll in Coursera classes just to get access to the video lectures for later perusal for information relevant to my work and interests.

Indeed, the sharing of information and knowledge obtainment has scaled. What are the traditional metrics of impact of a brick-and-mortar course? Is it just grades? Or is it deeper: students who go on to use the knowledge gained to find new ideas, new careers, etc? Reaching order(s) of magnitude more students probably means more of the latter effect/metric. For something like academics, I would think that the more people who have been exposed to useful information and peers interested in these ideas the better.

@turadg recounts how Coursera created statistics for partners: """The completion “rate” does not matter because the denominator is a very noisy signal. It’s a count of enrollments and what does “enrollment” signal in an online course, when it takes two clicks from a tweet."""

>> https://news.ycombinator.com/item?id=20975443

And remember that certificates of completion are now for paying users only. I’ve personally got a passing grade on some courses that didn't translate into a certificate because it's not interesting to me professionally and therefore don't pay for it. I still wouldn’t call it a failure.

I think this emphasis on completion rates are a problem. I take a lot of MOOCs, but not all the content in said MOOCs are important. I just watch the parts of courses that are important or relevant to what I am doing, and call it a day.

For people learning from non-conventional sources, they are used to using bits and pieces from different online sources and building up their own courses and their own paths.

This fascination with completion rates is simply our propensity to value what we measure instead of measuring what we value.

Very much this. I think a large issue with MOOCs is that they're too focused on emulating the structure of a university class - focus on lectures, community built around specific classes, focusing on the metric of course completion rather than on attaining skills, etc. People who successfully teach themselves things (and right now, there are many people using the internet to teach themselves things) often have a very different approach.

It's interesting that so many people have trouble even considering that there might be other ways to approach education.

I think the Microsoft edx moocs have solved this problem pretty well. They are typically focused on a more narrow topic, they are shorter (usually around a month), and they are very explicit on the weekly time required and this actually matches.

I've used msft edx to learn bootstrap and their azure machine learning platform and they were there best moocs I've ever taken.

People who "successfully" teach themselves are overrated (good examples being Trump or Zuckerberg). They might be successful in a few dimensions (purely out of self discipline and above avg energy levels) but totally miseducated in many others because people don't know what they don't know. Without the right guides, feedback processes, environment and time(lots of time) most people usually end up taking all kinds of wrong turns.

Education not something intuitive, even if there are many paths. If it was we would just be leaving kids in libraries.

Your point that self-education often tends toward narrowness is well-taken, but I strongly disagree with the implication that any significant number of autodidacts are overrated in the way that your two examples are.

Trump graduated University, so how are you defining self educated?

Zuckerberg received an honorary degree, which suggests the University approved of what he learnt.

If you're pointing at education as the cause of the problem for these 2, structured education bears at least some of the blame because it should have been teaching the tools for further learning.

The rate of 12 years old kids who can barely read suggest that current school systems don't work too well either, so what do you suggest?

I think the problem here is the framing of the headline.

Considering MOOC signups are completely free, a percentage alone tells you nothing. 4% of 100 signups would be appalling. 4% of 60,000 signups is still thousands of people successfully completing courses, and is actually a success story.

I was recently watching some courses on AWS, I passed the actual exam but I only watched a few videos where things were a bit ambiguous for me. So the completion metric is not very useful or accurate IMO.

I too have signed up to a mooc, simply to watch a particular lecture or two on a topic - as recommended by colleagues or friends. No intention of completing the course, and in fact very happy with the content.

Perhaps they need a box on the signup or viewing form, to indicate "just checking it out to see if I'm interested", or "just here to watch part of the course".

Here in Australia, when students finish high school and enter university it is common to hear them given the advice that it is "okay to change" after a year of your course. Get a feel for it, decide if it is actually for you, calibrate against the teaching/learning style, measure the enjoyment vs effort etc. I imagine MOOCs are starkly in contrast (provide an email address) to the complexity of signing up for a university degree.

Who knows, maybe students even form opinions by partially completing MOOCs in topics they might want to study at a tertiary institution!

How about making all the content viewable without signing up for the class instead of a checkbox.

Yep, exactly this. I'll also add that there are just a lot more signups. If I sign up for 20 courses and only really engage with 1...that's still one more course than I would have gotten to learn from otherwise. Certainly my eyes are bigger than my stomach when it comes to learning in many cases, but that doesn't mean that great value is not being had.

You hit the nail on the head and I'm going to shamelessly steal the last sentence because it's a great turn of phrase.

Imagine you ran a class with 90% casual users wherein 80% of serious users finished and 10% of casuals. Out of 1000 users 170 would finish or 17%.

Charging increasing amounts of money would drive away increasingly large number of casual users while increasing completion rates.

By the time you get rid of all your casual users you will have an 80% competition rate while halving the number of people who take the class to completion and decreasing the number who learn something by 90%.

Maybe we can measure the relative preparation of incoming students who participated in moocs vs those who did not to better understand how much people benefit?

I have done research on moocs. I think this article is a relic of it's time (2014). While some people are still focused on completion rates, I think many researchers have moved on to instead focus on other topics.

What are the current metrics?

We grappled with rates when I was engineering manager for the team at Coursera that created course statistics for our partners.

The completion “rate” does not matter because the denominator is a very noisy signal. It’s a count of enrollments and what does “enrollment” signal in an online course, when it takes two clicks from a tweet.

There are other complexities but completion rate in the “are MOOCs worth offering” question is silly and tired.

What matters is people getting value they otherwise wouldn’t.

How did you end up measuring the course? Completion by students who logged in at least once? Hours spent in course? Cost of course per completion?

It seems like we would want to compare what is learnedC that’s how I value. But not sure how you would test that.

These are for MOOCs, not online degree courses, though.

There's no certifications with these. You can't get an accounting, law, or medical degree, bachelor/associate certificate, or anything you'd need professional accreditation for.

Maybe the completion rate is similar for people who buy books. I'd end up getting a book thinking it'd be an investment as it would be good as learning a subject. The next day I didn't really want to spend 12 straight hours learning Visual Basic.

Also these aren't like auditing a class and listening in on a real lecture either. Just from the list for UT Austin (a very good school): https://www.edx.org/school/utaustinx

There are university lecturers people listen to all the time though, through the Great Courses / Teaching Company. Those sort of fit the leisure-learner path and don't require logging into some system for it.

> Maybe the completion rate is similar for people who buy books.

There's an even lower bar to entry: all the MOOCs I've participated in have been free.

They might be free, but all the MOOCs I've participated in have required a weekly time commitment to complete them, which is a sort of hidden cost completely separate from money. I've started the Stanford ML course on Coursera three times now, and each time some real-life thing has come up and taken me out of action for a couple of weeks.

That's just what free means, isn't it? The cost is separate from money.

I don't think it's hidden that you have to study in order to learn.

Studying isn't the problem, deadlines are the problem. If you could work at your own pace then I'd guess the completion rate would be substantially higher.

this is has a lot do with our attention span. it has slowly been in decline ever since scrolling took over. the running animation over content tires us out and it is hard for the eyes to follow and track lines.

whenever somebody asks me how do people read on the web?, I say: they don’t. people rarely read web pages word by word.

instead they scan for information with keywords, uninvested and disinterested, ala, lowered attention span, less commitment.

this behavior immediately translates to poor results on everything and MOOCs aren’t any different.

I've many Udemy courses and my observation is this:

Most course creators do absolutely no video editing. Sure they slap in an intro but that's it.

So you'll get a lot of umms, & silence, waiting for the creator to gather their thoughts.

There's also a lot of filler content. So a 2 hour course might have 1.5 hours of history, reasons for the course, setting things up and 30 minutes of useful stuff.

So I watch most videos at 1.5x speed or more. I also download Udemy videos for offline view - therefore, Udemy doesn't know which courses I've left untouched, started, abandoned halfway or completed.

The fact these classes dont allow one to preview the first few courses before registering has to influence these stats.

I’ve done a number of them over the last couple of years and the fact that I can’t preview has caused me to abandon so many of them. If I don’t like my professor, the way the course is organized, or whatever, I’ll abandon the course and go find some other source for the same information which does a better job resonating with me.

And I know I’m not alone in doing this, off the top of my head, I know at least 3 workmates and at least a number of friends who do the same thing.

The completion rate would have to shoot up much higher were they to allow people to know what theyre getting into ahead of time. And strangely, these companies have a much better ability to do this than traditional schools.

Can I bounce an idea off Hacker News readers?

How about doing live events at which you simply play -- on an overhead screen -- the very best videos or courseware from the web on selected technical topics. You get the best possible presentation and the chance to mingle with people who are keen on the same subject. You'd still need a person to act as a moderator or host, but there would be no official speakers. Would you go to such a thing?

This idea came about because I was at a seminar recently with multiple presenters on technical topics. At least 90% of what was said is available on the web. And the average presentation was awful. Yet, the auditorium was packed. Two hundred people showed up to hear publicly available information presented by mediocre speakers, none of whom were experts in the field (just people interested in the topic).

This is a regularly held seminar and people come. A lot people like attending a talk even if we could have stayed home and watched a better video on the same technical topic. I think part of it is that some people absorb material better in a classroom setting and the other aspect is the chance to socialize and talk with like-minded people.

This summer there was a "reading group" at my university about physics foundations based on watching the videos from this lecture series on Scientific Realism https://www.youtube.com/playlist?list=PLtIs3eEC6pzL1v_haWfzn... Structure: Tuesday in PM ppl watch together, then Wednesday PM group discussion (moderated to keep on track).

There was a dedicated volunteer who prepared a 1-2 page summary of the lecture before the meeting, and a note taker for the discussion.

The lectures in the playlist are by the top people in the field (various viewpoints represented) so it was a great learning experience. Groups were 10-15 people and discussions were all worthwhile. Not sure how this would scale to 15+, but the watch-best-lecture-available-and-then-discuss-in-groups is definitely a good model. [at least one data point]

I think the issue isn't that people aren't learning properly, it's more due to with the recognition of work and time required for no tangible reward. It's the same reason it's really hard for most people to sit down and learn coding on their own. It's not because lectures are hard to watch, or they aren't learning effectively, it's because you aren't getting anything tangible from it. Bootcamps are the same way, you get a "certificate" but it doesn't really have much value, and people know it.

Getting a degree from a good school (I go to UT Austin actually) is different. That's why people go to college.

A CS degree covers more than just programming. A bootcamp focuses on programming only.

Let's assume the CS degree has on average 1 out of 5 courses per semester that directly involves practicing programming, and not just filling up the requirements on liberal arts, and math, etc. 13 weeks * (3 hours lectures per week + 5 hours homework per week) * 2 semester per year * 4 years = 832 hours.

So a full-time developer bootcamp would need to be at least 5 months long to match.

Most are only 2.5 to 3 months long, but there are some 6 month long developer bootcamps. The short bootcamps might not have as much value, but still a good minimum start to development.

> the other aspect is the chance to socialize and talk with like-minded people.

This. Also to be seen in person, and social media that you are attending. To feel special. It is not a criticism, it is a human thing.

Interesting idea. I think the real trick would be handling Q&A, and encouraging discussions?

It's interesting how content discovery has remained ad-hoc despite the arrival of ed-tech platforms like Coursera or EdX. People still prefer to ask for best books, courses, podcasts, cheatsheets on Slack/Discord/Twitter etc.

I think this has to do with the conflict of interest. Any platform that creates its own educational content, necessarily becomes bad at content _curation_. For ex, 3Blue1Brown has amazing series on linear algebra and calculus but we don't find Coursera pointing to it.

3Blue1Brown made videos for Khan Academy for a while but ultimately decided to take his career in a different direction. He talks about it on the first episode[1] of the Numberphile podcast, which if you are a fan of 3Blue1Brown I'd highly reccomend giving it a listen.

[1]: https://www.numberphile.com/podcast/3blue1brown

I have a PhD, earned while on a prestigious national fellowship, and graduated with highest honors in undergrad. I graduated earlier this decade. I have enrolled in a single MOOC to keep my skills current in my field, and I couldn’t finish it. Maybe it’s because I work now, but I simply couldn’t find the time to complete the lab assignments when only putting in 10 hours a week. My sample size is low, but I get the impression that some of these MOOCs basically demand that you have traditional student-level time and focus to complete them. Is that wrong? Perhaps not, but 4% then becomes unsurprising for a completion rate.

The issue is with how MOOCs is used and what the creators expect from it.

I have signed for many MOOCs but not completed even 1 till date. But I got the knowledge I needed and got the task done. To me MOOCs is an organized google search or detailed stackoverflow/wikipedia.

Plus my extreme aversion towards "exams", is also a reason. Never found an incentive to get a paper that tells I know something. Plus, despite having certificates from school & university, I don't remember the stuffs I learnt. I forgot most stuff after writing the exams. Sadly, some even before writing the exam.

Maybe because the definition of completion for some of the courses includes a lot of things that aren’t valuable to the learner. I consider being able to skip lectures one of the benefits of online courses

Also because some of them don't let you really see what the content is unless you sign up. I have signed up for free online courses before just to check what they contain.

That’s because overwhelmingly people don’t care about learning, they care about certification. The completion rates for online degrees are vastly higher because people get that piece of paper that marks one as an intelligent, hard working conformist. They have to care enough to apply, and pay for a course. I don’t know how much higher the completion rate is for EdX or Coursera students that pay for verified certificates over auditing courses but if it’s less than four times as high I’d be surprised.

https://www.usnews.com/education/online-education/articles/2...

> While many believe that there is a huge difference between online and on-campus completion rates, [Russell Poulin]’s research suggests the difference is slight. Based on a survey completed by more than 200 North American school officials in 2013, Poulin found that course completion rates averaged three to five percent better for on-campus courses than for online courses.

https://www.edsurge.com/amp/news/2019-06-06-moving-from-5-to...

> Today, 2U reports completion rates of up to 88 percent for their online degree programs. Harvard Business School’s online programs claim similar success, with completion rates of 85 percent.

> At Acumen, where I design online courses, we’ve also been offering selective cohort-based programs for the past year that achieve completion rates of 85 percent. That’s a far cry from five years ago, when only 5 percent of the students were finishing the MOOCs I was designing.

> How have instructional designers collectively moved the needle so dramatically on completion rates? Unsurprisingly, some of the biggest drivers of these improved metrics include making people pay for online programs, increasing the selectivity of courses, and adding program managers and teaching assistants to follow up with learners.

I’m not sure people don’t care about learning, I’m certain I’m not the only one that’s gone through a lot of stuff on OCW on my own. I think people are more likely to jump through specific hoops if they get something out of it, that has little to do with learning though.

I didn’t say people don’t care about learning, I said that overwhelmingly they don’t. There is a tiny minority’s who value learning for its own sake. They are most of the 4% who complete MOOCs with neither entrance criteria nor certification. They are the people who read complex works of literature, in whatever genre, outside an educational context. They’re the people who don’t just occasionally go to an art gallery but can speak knowledgeably about influences, styles and periods.

They’re nerds of whatever persuasion. Normal people work on their three hours of tv a day, nerds learn about Nabokov or the Assyrians or decide that it would be better to work through a modern introduction to algebra than Euclid for fun.

2014, but I find it interesting the title focuses on the completion percentage. Instead, it seems it could have equally been that graduation at the online class was more than 10 times that of his average semester (also from the article). Negativity sells?

Georgia tech's online masters in CS is the complete opposite (and orders of magnitude closer to a normal academic environment) than MOOCs, and the reason is all obvious design decisions / incentives.

Like any degree, Gatech's program has a permanent academic record, rules for grade quality in order to not be on academic probation, etc.

If you don't have the incentives in place, then many people will find excuses to not commit. MOOCs are lacking those incentives.

>If you don't have the incentives in place, then many people will find excuses to not commit. MOOCs are lacking those incentives.

I haven't researched this, but I would guess the $6,600+ cost has more to do with the high completion rate than any other factors. Most MOOC's seem to be pretty cheap, if not free.

Not to mention, completion of the Georgia Tech online program results in a degree. MOOC's typically just offer a certificate of completion, which carries little weight.

Why “complete” a MOOC though? Most MOOCs still have the same BS hoop jumping requirements as real courses alongside the meat of the course, but unlike real world courses theres no degree at the end. As a self learner I ruthlessly apply the 80/20 principle to maximize useful learning and move on. I’ve only completed a few that are mercifully all autograder based.

I loved the "achievement unlocked" feeling of completing online courses and getting a pdf saying "you did it!", so I tried to complete the classes for that (such a PDF has no real value, especially since I did classes on random topics I like, not job related).

I can see why Coursera/edx etc stopped giving out these certificates if you don't pay for them, but it has caused me to basically stop completing free online classes.

I assume this is just free courses. I’ve been an online instructor for a university for 4 years as a side gig. I’d say the completion rate is about 85%.

> I assume this is just free courses

That is what I thought as well. People who pay a handsome sum for an online course are not going to let their money go to waste. Something about paying up simply ensures that you turn up and actually attend/finish a course.

This might be outdated on some platforms, but the last time I had time to indulge in some online courses, you had to enrol in the course just to see the syllabus.

My completion rate was not stellar.

Until Visa officials around the World wake up and smell the coffee that Degrees shouldn't matter for immigration - I will care about and be anxious about my lack of a Degree. Until MOOCs don't confer a degree - it's going to be very hard to convince me to spend time getting a Certificate from it (as opposed to watching the most interesting and important bits and skipping the rest, in the "YouTube University" style).

Unfortunately, I don't think degree requirements are going to go away anytime soon for immigration purposes. Just imagine the backlash in the US/EU if the voting public finds out that people without degrees can now move to their countries. Many will link his to a mass-movement of unskilled and unwanted labor as opposed to talented and hardworking individuals.

That's very sad, but I suppose I can take a twisted comfort from it: the fact that this public attitude is unlikely to change in my productive lifetime means, it's worth girding down and getting a degree. At least it's unlikely that 10 years later I'll regret the time I wasted chasing a piece (or two) of paper(s) ~6 years prior.

I watched the Andrew Ng videos in coursera - I honestly didn't care about any course work, I was just trying to watch the vids... did I do something wrong?

Not exactly, but the coursework is worthwhile. It's well designed so that you get to only focus on the topic at hand, not setup or scaffolding of any sort.

You really get a feel for how to turn an equation into vectorized code, which the video lessons can't give you.

I've completed the first Ng course, the later five-course specialization, and a few other courses (Odersky's Scala course among them). Still, 4% doesn't sound unreasonable: I too have enrolled in more courses than I completed.

The cost of signup for a MooC is low and the time and effort needed to complete Mooc is rather high. That could explain the lower completion rather and something is still a lot better than nothing. Also why are we comparing against normal courses were people has dedicated time versus online where there is not dedicated time?

I think it would be good to design course content for the medium. Professional adults does not have unlimited spare time, especially not if you have other duties to upkeep that require time family and friends. Thus maybe it would be wise to divide the course content in smaller chunks and reward people for completing each chunk. This would be a design for Mooc that would be very similar to game level design which already works brilliant online! Aka divide and conquer algorithm. Reward people for completing levels.

https://en.wikipedia.org/wiki/Divide-and-conquer_algorithm

If you want to complete them I really recommend pairing up with a friend. Going at it alone makes it easy to quit. While 3 friends deciding to "drop out" is less likely to happen by an order of magnitude.

I've never successfully convinced a friend to take anything.

Turns out taking a Stanford class (or anything) on the introduction to logic isn't most people's idea of fun.

In my opinion, the lack of geography-specific pricing makes the evaluation (tests and assignments) and certification inaccessible to a significant population. Of course, one can audit the course anytime but it’s hard to have the same level of sincerity and commitment when there’s no incentive (certification that can probably be showcased on LinkedIN or CV) or deadline based timeline for the course.

I appreciate that these courses have been made available to anyone who wants to learn, but my point is concerning the meagre completion rate.

As it stands, MOOCs don't have the same recognition or importance in society as traditional education does. So naturally completion rates are going to be low. Doesn't mean the courses aren't good quality or the information being shared doesn't mean people's needs. Just means that the psychological contracts that people enter when they sign up for these courses are a lot less burdensome and easier to break than within higher education, which might even be a part of their draw.

I think coursera is amazing, i've finished a few courses signed up for a lot more. It's great, some i stopped because I wasn't really interested in, others I stopped because I got what I needed. I think it's truly amazing and I think completion is the wrong metric; maybe hours of instruction divided by average per course would be better.

This is great news.

I remember a time when/where this kind of knowledge was hard to find and devoured whenever it was made available to you.

I dropped one on automata after an exercise in the first exam required making use of an undisclosed algorithm that wasn't central to the topic and not covered in any of the required reading. When asking the lecturer about it, he directed me to chapters n through n+5 in some other book. I guess I didn't want it enough.

I registered to a lot that were out of my league (say advanced physics). But also a few normal IT MOOCs, like relational DB, made me strenghten some new skills even though I didn't finish it. Mostly due to trivial time constraints making me miss deadlines.

A lot of it has to do with credentialing. MOOC credentials aren’t worth the time investment.

Plus nobody cares about credentialing at all. What they do care about is solving problems, and it seems reasonable that 96% of problems become solvable with the course material provided before reaching the very end of a course. Once the problem is solved, there is little reason to continue.

You are generalizing. I am doing a MicroMaster in data science from MITx and they grant the equivalent credits of the first semester. It's worth something if I ever apply to a similar master program.

From having taught a MooC, I can share some light on this. Most people who sign up and drop it do so to access the course ware. Once they have downloaded this, they disappear or fade away.

I'd be very curious to learn the difference between free/two clicks to enroll and a paid course.

I'd assume completion rate for something that you actually paid would be way higher.

What length of course? I can't imagine that completion rates are the same for a 50 hour course vs a 200 hour course. Can't seem to find this data.

That makes mse think that are probably a lot of easy wins in the funnel of getting people successfully through the course.

The question is, does it matter ?

I've been following the MOOC phenomenon since its early days in 2010-2011, and taken a lot of moocs in the meantime.

Sometime the mooc reveals to be low quality, and I quit because I can get better information from books and/or by actually messing with the technology (xkcd.com/519). Other times, I quit the part after the interesting part, or don't care about going through all the single details to get a worthless recognition ("a website says i've watched all the lessons").

Other times I pay for a course, go through all the lessons, do all the labs, get all the certificate of completion.

It really varies a lot, and most people ask the wrong question: it's not "how do we measure success" but "how do we measure success with respect to the original goals?"

Very little people care about an automated system somewhere recognizing my 100% completion of the course materials. But I (and possibly, possible employers) care about actual competencies. It's the same thing all over again: in real life pieces of paper don't matter, what matters is the capability of solving problems and handling situations.

Average completion rate for MOOCs Is 4%

Not surprising at all. I often follow the first few weeks of a MOOC but after a bit I am satisfied and don't feel the need to push through all 13-17 weeks of material. Isn't that a success? I certainly learned a lot along the way all the same! Much more about a topic than I would reading the wikipedia page or an introductory blog post, because MOOCs (being regular college classes) take a lot of pains to lay out the course material in a logical and structured format, which is itself alone already much more valuable than a single person's hodgepodge thoughts.

Came here to say this. It's the same with text-like books. There's 100 pages of what you bought the book for, and 400 pages of filler at the end in an attempt to add more value.

Reply on news.ycombinator.com