Why Aren’t SMS Interventions Designed to Boost College Success Working at Scale?

> Text campaigns are most likely to be effective when they:

    Are sent by a school and even staff member who students know and trust
    Add meaningful and novel value above and beyond existing outreach and support efforts like giving students real-time information on the status of their financial aid applications
    Provide content that is personalized to the student and where they are in the college application and financial aid process

Scaling it would be convincing more school staff and schools to make a similar effort.

The positive effect was not caused by "getting a message", it was caused by "getting personal advice from someone you know". They scaled the wrong thing.

The software sent me an email asking me to do something, ostensibly from someone in our advising center, signed by my assistant chair instead, but obviously corporate-sounding and autogenerated. It reeked of fakery.

Is this sort of thing supposed to accomplish something?

Because, and I am not exaggerating in the slightest, this is the worst idea I have ever seen directly related to my career. This concept is an idea that I am fighting all the time.

Teachers and students alike around me sort of blindly "get used to" these notifications flying around you and even begin to expect them even when they don't work.

Essentially, they become the equivalent of spam in your email that you ignore, except they're harder to ignore and people still have this wildly misguided faith that they can think for you.

As a first measure -- I'd love to remove all "notification" type services from higher-ed; after a bit of a learning curve, as in -- yes, you must visit this website or call this person or read the syllabus to get things done.

You develop your own system of remembering things, maybe it's paper, maybe you email yourself, whatever works for you

Georgia State University is a great example of successfully implementing the goal of "college success" and continuing to innovate. They started by solving the problem of students accepting their offer letter, but never choosing classes during that crucial first summer.

First tool they built used a combination of AI chat box and counsellors to answer common questions, help students fill out forms, and/or get financial aid. At each step of product development they asked "what's next?". For example choosing a degree is "hard" for first generation college students. So they built salary, and career earnings during the UX where you pick your courses.

Another non-SMS solution they provide is automatically depositing money into student's accounts who are low and struggling. This especially helps 4th year students who may have burnt through their financial aid.

https://www.youtube.com/watch?v=Rh20W2uXHCc&ab_channel=BillG...

I think nudge theory is a subset of all the elements that make up gamification, but still, the idea that something is okay to do because "we're doing it to help people" seems a bit odd.

I think schools know there has to be a human at some point in the interaction chain.

A frustrating experience I had with this is a school I was at (expensive uni) tried to farm out this responsibility to other students.

In a mentorship program designed to do 1:1 student:student pairings and used “SMS intervention” this program asked students to take up to *10* student mentees, with the idea that that the real mentor load is pretty low (quick convos).

To me, this felt like an outrageous ask, as if they were trying to sub in students paying to be there with the tasks paid advisors are responsible for.

This was at grad school, and I think indicated something funding challenges behind the scenes. I imagine it’s similar/worse at high school levels. Maybe not the same exact problem, but it makes sense why this specific program linked has trouble.

> Implementing a texting campaign with hundreds of thousands of students may require organizations to forego advising and have texts provide just one-way content or to have large caseloads that limit how much engagement advisors can have with individual students.

> Another trade-off is that students may have less trust in a state agency or national nonprofit sending the messages.

> Finally, state- and national-level campaigns have typically sent content that is uniform across all students, whereas earlier, local campaigns often personalized information to students’ college plans or FAFSA (the United States’ government financial aid application) filing status.

> Even local text campaigns may face obstacles to efficacy that similar campaigns did not encounter to the same degree a decade ago. [...] As message volume through text increases, we should reasonably expect that any one message won’t get the same attention—and eventually, that people will migrate to other communications channels.

Using Future Authoring to Improve Student Outcomes - https://heqco.ca/wp-content/uploads/2020/03/HEQCO-Formatted_... - https://www.selfauthoring.com/research

1) Over saturation/overused: Especially as High Schools begin using these, students get too many, too frequently, and it just becomes a sort of background noise. Back when students still had email as their prime mode of digital communication, we had the exact same issue. Studies on their effectiveness were based on relatively short time spans where nudges were new & novel: The novelty has worn off.

2) A simple nudge can't help a complex problem. A college may nudge "You haven't registered yet! Don't forget to register for the next semester". That's an extremely frustrating message to receive when you're poor and can't pay next semester's bill, or some other roadblock.

3) It has been used to justify not scaling human resources for students, so any benefit is offset by lesser real human contact with advisors. People complain about administrative bloat at universities, but that doesn't apply to advising: I've seen caseloads rise from ~300/advisor to ~500/advisor. This is especially problematic for overcoming #2 above: Advisors can often help with complex issues that student's believe are insurmountable.

My observations on how to do it better:

Only use nudges for low-stakes issues, and even then choose carefully so students don't become numb to them. High-stakes issues need human outreach. With limited bandwidth by advisors to do this work, it needs triage: Even if it's high stakes, don't waste time reaching out personally to low-risk students. Determining risk requires a two-pronged approach:

1) A statistical model that uses historical data to identify students at risk of poor performance or attrition. A competent data analyst can build a reasonable model without having to go too far into advanced data mining. "Deep Learning" tends to be fairly useless here: There's not enough data. There's also a few good vendors out there that will perform more advanced risk modeling, partitioning a school's population into a dozen or more groups and building models for each of them. These aren't cheap, but I've seen it pay off when correctly paired with one-to-one human outreach initiatives. In a single year I saw it reduce attrition by 1%, not to mention academic performance increases, and that more than paid for the cost. I've also seen schools pay the invoice and expect things to magically change without aligning their resources to the risk analysis output and waste a lot of money.

2) A rule-base system of identifying students at risk. Things like a late payment; a good student getting their first D or F in a class; delaying registration for the next semester; requesting transcripts to be sent to another school; and many other factors.

Using the high risk population identified in #1 and #2, that is where you focus your one-to-one human outreach efforts.