Hacker News Reader

Comment by tsumnia | Hacker News Reader

Not knocking your own choices, but its more the backlash to the 2013 "MOOCs are the greatest thing ever! KhanAcademy/EdX/Coursera! Wooo!" Personally, I love them and would love to work with those companies. In fact, when I started teaching, I used many of the aspects I saw MOOCs using in my traditional classrooms.

However, there is a truth to the statement. Almost no one finishes a MOOC (some never even log in to take MOOCs they register for), and predictive models can determine whether someone will complete the MOOC as early as the 2nd week (if not sooner). Some people do grab what they want and leave, but that is not the case for everyone. If we then look to justify why bother, its a little hard to warrant hosting costs, etc. if you aren't seeing anything come from it. If I ran a MOOC with a 95% fail rate, how can I label it successful.

We are unable to do meaningful research on MOOC strategy effectiveness because we are too busy trying to understand why people never log in in the first place.

View on HN

> some never even log in to take MOOCs they register for

I've done this many times. Why? On some platforms (like Coursera) you retain access to the content of a course you have registered for, but that content isn't available for people who didn't register once the class is over. If I think I might ever want to view that content, my best bet is to register (there's no cost) and then have that content available when I want it.

I've also done this on platforms like Udacity where that's not the issue, because by registering I can create a curated list of courses I'm interested in possibly looking at in the future.

In my experience, the big failings of MOOCs is that they try to copy university classes, and university classes just aren't that good a lot of the time. Many people don't find lectures to be useful at all, but the vast majority of MOOCs focus on them. After the fifth time I'm stuck waiting for the teacher to finish their long personal anecdote or humorous story I usually give up on the MOOC and go looking for a good textbook.

I don't think I've ever seen a MOOC that had written content anywhere close to the quality of Dive Into Python or Learn C the Hard Way.

Also, MOOC efforts for community building are (at least were) focused on a particular class rather than a particular subject. This doesn't lend itself well to long-term community building, especially when people are going through these courses and very different paces.

There seems to be lots of discussion about how MOOCs can be improved, but not much effort into actually trying different approaches.

> the big failings of MOOCs is that they try to copy university classes, and university classes just aren't that good a lot of the time

We are still trying to understand what makes for better "learning". If I practice the piano everyday, I will no doubt have a better ability to play, but I may not understand music theory. As a shameless plug, I am trying to add practice to be a core part of learning Computer Science and a link to my research platform is in my profile.

This is the same separate of vocational schools and coding bootcamps to graduate schools. The traditional 4-year school sits in the middle ground between application and theory and is probably why we have such difficulty deciding "success".

> to finish their long personal anecdote or humorous story I usually give up on the MOOC and go looking for a good textbook

This is ultimately student preference. Some students like a more affable teacher, some don't want a social-able instructor (so long as its not interfering). I will say that your post is confusing as you say you don't want instructor anecdotes but then MOOCs should be doing more for larger-scope community building.

I will argue larger-scope community building is difficult and MOOCs struggle with it; I would say because of its availability. It is harder to build community when the individuals come from such diverse backgrounds. Many factors can make it difficult to build bounds (social, cultural, temporal, etc.). Furthermore, you cannot control why the student wants to take a MOOC. They may not want to give that additional effort to community building for a subject they do not perceive as having higher priority to other life decisions.

I would disagree with "actually trying different approaches". This is being done and academic journals are trying to study them (the deadline for the Journal for Education Data Mining just passed). There are also non-academic approaches like Duolingo and Khan Academy, and they are trying different techniques as well.

Ultimately, there is no panacea for learning. We know engagement can lead to learning and therefore if we can maintain engagement, we might be able to teach. Community-building can be that engagement. Self motivation can be. At that point we need to research these things more to know for sure.

> This is ultimately student preference.

Sure. But the last time I checked, almost all MOOCs were lecture focused. Like I said, I've yet to find any with a text component as good as Dive Into Python or Learn C the Hard Way or other free online textbooks. Perhaps there's are a few out there, but every time I've browsed MOOCs I've found just about everyone to be lecture focused. Out of the dozens I've looked at (on multiple platforms) I can only think of one that wasn't, and that was eventually removed.

Plenty of people use the internet to learn plenty of things. I think it's time to consider how much of the "failure" of MOOCs is due to the failure of online learning, and how much of it is due to the fact that university classes aren't a great way to teach people things (at least for a large chunk of the population).

> I will say that your post is confusing as you say you don't want instructor anecdotes but then MOOCs should be doing more for larger-scope community building.

Instructor anecdotes were an example of why I don't like lectures. With a book you can scan over content that is superfluous and get to the information you need. You can re-read or take slowly the parts you have trouble with, and quickly skip over the parts you already know. All of this is much, much harder to do with lectures.

I'm confused as to why you think professor anecdotes are related to large-scope community building? They're quite different things, and if people view them as serving the same purpose then there's even more of a misunderstanding when it comes to education than I had previously believed.

As for the difficulty of community building, I think you should broaden your horizons a bit. For instance, if you're a bit late on a Coursera course the forums are pretty much dead, and you're better off discussing things or asking questions on another site. Likewise with Udacity - the course might still be active, but everyone that has completed the course has moved on and won't see your message. This isn't because of a difficulty in community building, but because of a conscious decision that most MOOCs take to segregate their forums by class.

(I will ask if you could define "community building" a little more. You did not in your original post, and it seems my assumption was incorrect. Before trying to address better, I'd like to have a more concrete definition)

While I cannot speak for all MOOCS, I have not had the same experience. MIT's CS 6.00x course on EdX was work heavy, Udacity's Web Development course was work heavy, and even Coursera's Design of Everyday Things had participation components. In the only (barely a) MOOC that was video only, it was predominantly a "follow along with me" coding process.

For instructor lectures, I give more positive responses than negative about my anecdotes. At the end of the day, I'm human and I'm trying to enjoy my job (which students can tell if you don't). Development of tutoring systems is still in research phases as we identify knowledge components for different subjects. A history course operates different than a computer science course.

I will say EdX has produced research on instructor lecture videos. Users prefer 3-5 videos and so instructors should design courses like that. Instructors that simply upload a classroom lecture are not appropriately transitioning their material for online use. This satisfies your being able to flip from topic to topic.

I will end the Dunning-Kruger effect suggests that students are not the best at assessing what they know and that a professional instructor has a better idea of what "knowing" something means. This again gets back to my discussion on vocational vs. graduate school and again, humans are flawed, imperfect beings. While an intelligent tutoring system can alleviate this issue, humans are still building them for the foreseeable future.

> Users prefer 3-5 videos and so instructors should design courses like that. Instructors that simply upload a classroom lecture are not appropriately transitioning their material for online use. This satisfies your being able to flip from topic to topic.

In my personal experience, it really doesn't. Breaking something into segments can make things a bit easier, but it still leads to a lot of wasteful time if there's a minute and a half of useful information within an 8 minutes block. You can't scan through it the way you can with a book, and it's much more difficult to review a difficult piece of information you just received (it's easy to re-read a sentence slowly, whereas rewind a video to the beginning of the last sentence is more cumbersome and you're going to be watching it at the same speed).

Text also tends to be much more succinct, where as lectures are often repetitive and meandering.

I think the assumption that multiple video segments solves the problem is instructive. Might not a large part of the problem be instructors saying "We've done X, which has solved Y issue" instead of saying "We've done X, let's look at whether or not it has solved Y issue"? I appreciate the fact that people are attempting to solve these issues, but if there's no differentiation between attempting to solve something and successfully solving something than it shouldn't be a surprise

> I will end the Dunning-Kruger effect suggests that students are not the best at assessing what they know and that a professional instructor has a better idea of what "knowing" something means.

This is a pretty big assumption, and one that I don't think is accurate (based on personal experience, and my experience talking with both students and professors). The best way I've found to test one's ability is to actually apply it to a task, where it usually becomes quickly clear to the individual where the holes in their understanding are.

I searched for The Design of Everyday Things but I could only find a course with this name on Udacity, not Coursera.

Can you check and confirm?

Ah, it was Udacity, not Coursera; my apologies

I still miss the excellent Dive into Python. Last commit was 7 years ago.

> If we then look to justify why bother, its a little hard to warrant hosting costs, etc. if you aren't seeing anything come from it

> If I ran a MOOC with a 95% fail rate, how can I label it successful.

How is the completion rate a better measure of value added than the absolute number of finishers?

Both are important. There are multiple questions to be asked.

For extremely self motivated people, MOOCS are a win, no doubt. Again, MOOCs are definitely better than no MOOCs.

But the promise was that MOOCs can just replace college to a large degree, because the content you get in college will be online. In that instance the question becomes "do MOOCs work for most people?" For what we'll call the "average" person, studies have shown that just isn't the case. Most people don't learn well from MOOCs, unfortunately.

Therefore what we're learning is that a non-trivial amount of what makes a college education successful is some combination of the following things that MOOCs don't have: external pressure, scheduled courses, due dates, a community of learners, a physical campus, etc.

As such, I view part of the next step to figure out which of those aspects MOOCs are missing that are vital to the mix in order to help average people learn the things they need to know. Traditional education just says all of them are necessary, but I'm not convinced that's the case.

Could you cite the studies that say "MOOCs don't work for the average person"? While it is certainly an intuitive statement, I find it hard to empirically verify because the population of people who take MOOCs are very different from those who attend colleges.

For example, those who take MOOCs are probably working full / part-time, trying to pick up some knowledge on the side. They will obviously have less time for learning than full-time students.

Many are behind paywalls, but... let's just say there are ways around those.

http://onlinelibrary.wiley.com/doi/10.1111/jcal.12131/full

http://onlinelibrary.wiley.com/doi/10.1002/aehe.20018/full

https://dl.acm.org/citation.cfm?id=2835842

http://www.irrodl.org/index.php/irrodl/article/view/2112

https://www.openeducationeuropa.eu/sites/default/files/legac...

http://www.educationaldatamining.org/EDM2014/uploads/procs20...

While "average person" is not the term I'd use, the question you're asking is a bit loaded. What measure is used to say a MOOC "works"? If we are saying completion, then high drop out rates are an issue. If we say high interactivity among those that don't drop out, this is where the current field of research in MOOCs is at. While I do not study it, colleagues in my lab are.

My point is that quantifying the effectiveness of a MOOC is an ill-defined domain. Instead, we acknowledge the attrition rate and do analysis on those that pursue the course. Current research looks at system interactions, the social networks of MOOC forums, etc. to identify student behaviors that show higher "gains", be it course completion or grade.

If a student leaves a MOOC, there is no way to identify why they leave. Likewise, unless asked, it is hard to identify why a student joins a MOOC. Students enrolled in a MOOC can come from a variety of culture, geographical, sociological, motivation, etc. backgrounds. As such, assuming traditional student behaviors is ill advised. However, in the process of learning, access to material is not enough to learning the material and additional effort is needed by the student to build the necessary mental models in their head. If a student drops out of a MOOC, it is hard for the runners/analysts of the MOOC to appropriately say the student learned the material.

As austenallred mentions, current education research is looking at the motivation of learning, as well as what constitutes motivation (and learning for that matter).

Do most people learn well from traditional methods though?

If dropping out of a MOOC involved a big financial hit, social humiliation, and the loss of a great deal of personal freedom, the completion rates would skyrocket. Completely different incentives.

The reasoning that traditional education is successful because it has high completion rates because of external pressures is a bit circular, isn't it?

Perhaps. The question is which of those things really move the needle; can you have some, not all? For example, our completion rates are 90%+ - better than universities, but there is no social humiliation, loss of personal freedom, or financial hit.

What is the typical demographic of your cohorts? How do you compare to college programs that have similar cohort demographics?

I think all educational outcomes should be measured with respect to the answers to these two questions.

K12 schools in extremely wealthy areas can compare themselves to like peers, but comparing themselves to high-poverty areas is a useless comparison. Neither institution would do well under the other's constraints.

For-profit higher ed is definitely the same way.

We're very spread out, general lower income, more black/latinx than white, so it's not a case of us just grabbing the rich kids.

It also works the other way. It's socially/professionally acceptable to stop working for 3/4 years to go to university, somewhat less so to stop working to take a bunch of MOOCs. Therefore, most people taking MOOCs are also working full time/studying/looking for a job, and they're also likely to be older and have more life commitments than most students. So MOOCs are naturally going to take a back seat to everything else going on in a person's life. They also don't carry the same recognition as having a degree.

MOOCs also come with a number of unique advantages compared to traditional universities:

- They widen access to education to people that aren't able to attend university

- People can take courses they know they might struggle with without fear of flunking them and harming their academic record

- Mature students don't need to make life sacrifices in order to take a MOOC. They don't have the dilemma of whether to up roots and move to attend a prestigious university

>some never even log in to take MOOCs they register fo

So what? How is this at all relevant?

If we look at how many people have walked into a bookstore or library, have browsed through a book but never bothered to actually buy and read it, would we say books are a failure? Would you say books have a 95% fail rate? This is such a pointless train of thought.

Why did you decide that MOOCs should be held to the same standard as libraries or bookstores?

MOOCs are marketed as courses, and sometimes as an alternative form of higher education. Furthermore, I doubt the people who go to the trouble of designing curricula for MOOCs expect students to drop out partway through. Comparing MOOCs to college courses seems like a better match to the image the MOOC companies themselves have promoted. And if we make that comparison, completion definitely matters.

The fundamental difference is that MOOCs are recorded rather than performed live. This makes both the distribution and access patterns more similar to books than to college courses.

Of course any particular MOOC can be made more or less like a normal class by putting the content behind a hefty paywall, enforcing strict due dates, and adding perverse performance incentives.

> Some people do grab what they want and leave, but that is not the case for everyone.

Are there metrics that have measured this? What percentage of users find a MOOC useful even if it wasn't completed?

What I was trying to get at is that by looking at completion rates we might not be looking at the whole picture. Especially if the question we're revolving around is whether the course is worth the creation/hosting costs.

> some never even log in to take MOOCs they register for

[...]

> we are too busy trying to understand why people never log in in the first place

So you have at least 4 classes of 'user':

- did not register

- registered but never logged in again

- logged in at least once

- completed course

It seems to me that there's a different way to categorize the users in measuring course effectiveness:

- did not find the course useful/helpful

- did find the course useful/helpful

So if you are able to craft a course that has magically high engagement, but people don't find it particularly useful - you've still failed.

Maybe the value that course-makers hope to deliver through completion of the course simply isn't high enough?

If a user completes 70% of a course but doesn't see the need for the rest of it, then stopping at that point is the logical move. What sets college/university apart in this is degree which has a very high social impact. The fact remains that MOOC achievements are still widely looked down upon as somehow lesser than their college equivalent.

This whole discourse has encouraged me to revisit those courses I found most effective and complete them. At the very least I can help those courses stats and provide some further narrative of their utility.

A major issue with using "useful/helpful-ness" as our metric for success is quantifying the term. A self-report Likert scale is susceptible to users just rating 1 or 7, or saying its useful even when it was not (https://en.wikipedia.org/wiki/Self-report_study#Disadvantage..., not the greatest citation, but I am short for time).

The issue then looks at what is "success" in a MOOC. If the goal is to just have videos online, I'll just watch YouTube (as I have my own lecture series there). However, observing is considered one of the lower level of learning and things like Bloom's taxonomy point out there needs to be some type of interaction for better learning gains. These interactions require the student to have a more active role in the learning process. If they are not interacting with the system (logging in, watching videos, completing exercises, etc.), then they are not taking this needed active role. This is where "drop out" begins to be quantified and where then we can measure what worked, what didn't.

To address you example, if the user stops at 70%, researchers will ask "why?" From there, analysis of student behaviors, effectiveness of interface/instruction/material, etc. will arise. If the ability to study these things is confounded by the fact that the vast majority quit before completion, it makes it harder to answer the "Why" and "How do we fix it" questions.

Again, what we quantify as "successful" is still up for debate; but if a student drops out, it becomes harder to tell if they learned from the course and if it was student or material that drove that decision.

I suspect something like Udacity (on which classes rae not free and there are more structures 'degree' programs) has far higher completion rates, since it's not reasonable to just impulsively sign up and then stop. That is certainly true for something like Georgia Tech's OMCS. I still think the hype was justified in that stuff like OMCS is legitimately the future of education.

Given the nearly non-existent barrier to entry, I'm not sure that completion/engagement percentages mean much. MOOC content can be useful for help learning certain types of things. But so can books, YouTube, etc. Useful resources for (mostly) continuing professional education has to be seen as a very large step back from changing the future of education for underserved populations.

Why would you complete a MOOC "cover to cover" when they offer no accreditation? People just pick up what they need and move on. If they were offering a complete program and degrees, millions would complete them. MOOCs are amazing, but a simple VOD system can't replace the education system on it's own.

> some never even log in to take MOOCs they register for

> If we then look to justify why bother, its a little hard to warrant hosting costs, etc.

Presumably the people who never log in aren't imposing any load on the system, no? Just rescale to 2% of the size your user-metrics say you need.

Reply on news.ycombinator.com