Canvas online again as ShinyHunters threatens to leak schools’ data
theverge.com
theverge.com
We got our first email (from Academic Affairs) notifying us that it was down at 5:17pm EDT this afternoon, with little info; followup emails were sent at 6:24 and 6:57 with more info, but mostly about how we would be compensating for it and not about what actually was going on (other than, "nationwide shutdown" and "cybersecurity attacks", no further detail). I don't get a sense that they know much more than that, not that I would expect them to.
A perhaps telling detail: they're instructing us to have students email us directly with any work that had been submitted via Canvas. That suggests that they have no particular confidence that it will come back up soon.
I personally am only slightly affected; as a CS professor a lot of my students' work is done on department machines, and submitted that way, and I do the actual exams on paper. More importantly, I've never liked or trusted Canvas's gradebook, and so although I do upload grades to Canvas so students can see them, my primary gradebook is always a spreadsheet I maintain locally.
But I have a lot of colleagues for whom this is catastrophic at a level of "the whole building burnt down with all my exams and gradebooks in it"---even many of those that teach 100% in person have shifted much or all of their assessment into Canvas (using the Canvas "quiz" feature for everything up to and including final exams), and use the Canvas gradebook as their source-of-truth record. We've been encouraged to do so by our administration ("it makes submitting grades easier"). For faculty in that situation, they have few or zero artifacts that the students have produced, the students themselves don't have the artifacts to resubmit via email because they were done in Canvas in the first place, and they have no record of student grades or even attendance (because they managed that all inside Canvas). I guess they have access to the advisory midterm grades from March, if they submitted them (most do, some don't), but that might be it.
My gut feeling on this is that this is either resolved in hours (they have airgapped backups and can be working as soon as they can spin up new servers), or weeks (they don't). Very little in-between. And if that's true and we wake up tomorrow with this unresolved, I really have no idea what a lot of professors at my university and across the country are going to do to submit grades that are fair and reasonable. In the extreme case, they may have to revert to something we did in the pandemic semester (and before that, at my school, in the semester that two major academic buildings actually did burn to the ground a week before finals): let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do?
(Well, one thing you can do is not put your eggs all in one basket, and not trust "the cloud" quite so much, but that ship's already sailed. I do wonder if in the longer term, anybody learns any lessons from this....)
UPDATE: As of 11:45pm EDT, my university's canvas instance is up and running! Here's hoping it stays (but I'll be downloading some stuff just in case...)
As a parent of kids who are impacted by this, I’m not super concerned about the data being held for ransom, but I sure as fuck am concerned about how much it’s going to cost the district to move to another provider.
Does Canvas have cybersecurity insurance?
Also, ransomware gangs often exfil the data and threaten to release it if the ransom is not paid--blackmail, of a sort. It depends on the company and the data set whether this is effective as a tactic. But when it is, backups don't help.
It’s so simple to send an e-mail to the student with relevant records on completion of a quiz or whatnot. They don’t do it, because they want to control the data. (And universities don’t insist on it for who knows what reason.)
Extremely non-tech savvy, hates computers, and is gonna grumble "What the hell is a PGP? Better not be another one of those phone code things." as you try to pitch this highly-technological solution to a largely niche problem domain.
Hell just getting people to do secure passwords is a whole thing.
This would undermine Canvas's lock-in.
ed tech is the WORST performing VC sector
the ONLY game in that town is vendor lock-in! are people joking?
c'mon, canvas is a huge piece of shit. the SaaSpocalypse is coming for them - it seems it is simply that LLMs will be used to exploit it first, rather than universities writing an open alternative they share with each other for free.
In reality, universities always spin off anything that looks like it could generate revenue. It is very telling that you can't even get your college transcript from your college. You have to go to (and pay) some third party to get it. Some universities even outsource their "classes" like elderhostel to cruise lines and travel companies.
That already exists [0], and is actually reasonably popular.
> the SaaSpocalypse is coming for them - it seems it is simply that LLMs will be used to exploit it first
I doubt it, because enterprise sales has nothing to do with how good your product is, how expensive it is, how easy it is to administer, how secure it is, etc.; it only depends on how good you are at enterprise sales. I mean, my university is Oracle-based, and I'm pretty sure that you could get 3 random undergraduates to write something better, so I don't think that LLMs writing better/cheaper software will make any difference here.
[0]: https://moodle.org/
Canvas is used by Harvard, MIT, Stanford, Carnegie Mellon, CalTech, etc. If they each paid 10 FTE, they could set up a foundation that could govern the development of a top-tier LMS. Every tier-1 state institution could contribute 5 FTE. Even little JuCos could chip in an employee here and there. You'd pick up hundreds of capable employees at a fraction of what those schools currently pay to Instructure.
IT department will then build the feature as instructors are high-status and IT is low-status, and they aim to please. The software will collect hundreds of these over time. The institution will accumulate more developers, QA, a11y testers, PMs, instructional design consultants, and more PMs to deal with the instructors. The institution will then move to SAAS solution where the instructor is forced to join Canvas Jira and submit their feature request. A product manager at Canvas will then post to Jira and say thanks for your feature request, we will consider it. Game over.
It's better than nothing. (And good training for the real world.)
Also, most universities (and many schools now) issue academic e-mail addresses to students. In those cases, the email is definitive proof.
What seems easy on hobby projects gets way more difficult at scale. Source: experience.
> setting this up is well beyond the capabilities of most students.
Your comment stated that college doesn't add much to a person's employability. (If you had wanted to be less obfuscatory, you could simply have said "a [HS] education is already adequate qualification for many jobs; college doesn't add much").
That was your claim. (I don't think your claim is correct of many OECD countries' colleges, but it was the claim you made.)
You then replied to J-Kuhn to say that they had misunderstood your comment by (mis)paraphrasing it as "Students attend college to become qualified to work."
> Most graduates aren't really qualified to work anywhere that they couldn't have worked before going to college in the first place.
The double-negative could be read both ways: "Most HS grads aren't really qualified to work anywhere and college doesn't add much" or "a [HS] education is already adequate qualification for many jobs; college doesn't add much". So we apply Occam's razor and since the [U-5] unemployment rate for [US] HS grads is << 50%, conclude you meant the former. But the comment was still needlessly obfuscation. Even if we assume which country and education system you were talking about.
Conversely, it's a little weird how I ignored your lack of clarity and didn't point it out so starkly, then you reduce the interaction to this ping-pong. I don't intend to overflow the stack on this.
I know that I used a double negative, I meant to do that. Not everything is meant to be written and read the exact same way. That was the particular style of sentence I chose so you can just deal with that and you don't need to keep pointing it out.
Put it like this: it was an interesting discussion before that, and looks like people expected you would simply clarify your meaning (like most of us do when we're being unclear), so the discussion could proceed. Not go on all-out attack.
They are referring to MOST graduates of MOST colleges. This is a deliberate overgeneralization about the nature of post-secondary education meant to highlight how it's frequently viewed solely in terms of completion rather than with regards to any skills or knowledge gained from it.
I mean, anyone smart enough to attend university could probably figure it out if they really wanted to, but there are hundreds of other useful things that they could learn too. There are only so many hours in the day, and given that most students don't get that many emails, I can hardly blame them for not wanting to prioritize learning how to filter emails.
(I personally have over a hundred lines of Sieve filters, but I'm definitely not a typical student)
Exactly what is in their field of study, nothing more. That's a huge part of the problems created by treating academia as a degree mill mandatory to get a job able to feed yourself instead of a place only for those truly interested in actually studying a subject.
Going by a certain story 2 years ago, their concern should be that they're overqualified for Meta.
It doesn't help that gmail, which is the only serious direct competition to outlook, straight up doesn't do "folders" and instead goes with markers. So you can't really just put a filter that drags all the 100 low-priority alerts in what would count as a first degree abstraction of "place where things are sorted into". No, there are two layers of abstraction between point A and B of things, sorter and sorted things. The result? Muggles can't recognize the heck you're describing and refuse to even acknowledge the possibility.
Biology is a great example because of just how important digital record management is to experimentation in the field.
2. Yes, configuring gmail filters should be doable for anybody with a university degree. It's really not hard.
Perhaps Outlook is difficult to configure. Thunderbird is intuitive.
While true, unless I'm mistaken, markers (I assume you're referring to tags) can be nested to provide a pseudo-folder hierarchy, and with proper filters you can remove the "inbox" tag and have the mail only show up under the specific tag.
TBH I don't fully mind it, it lets you classify an email in multiple ways (eg "See Later" as well as "Work related").
Even if they didn’t hand in an assignment at all, without any reason provided, I’m required by regulation to offer them a second chance to pass that assignment.
The students’ rights are quite strong here (Northern Europe), which I generally support, but it has some downsides.
Yes. And most of the general population. They can do it once they know it exists, most people just are not aware it is a thing at all.
>What are they learning?
Here, their "major" as you say in the US. Someone in econ, biology or even CS is not going to learn Outlook rules. Maybe IT or business will have a sentence on it.
>Where will they be qualified to work?
Any office job. Any job really.
Are you suggesting that outlook wrangling be explicitly taught at the college level?
Delete
Delete and Report Spam
It reminds me of an old joke my father used to say about jobs with virtually no interview (fast food, etc). He called it "The Mirror Test", as in if you hold a mirror up to the person, does it fog up? If yes, you are hired!
Have you met the average community college student who doesn't even own a laptop but does all of their work on their phone? Gmail doesn't even allow you to create or manage filters from their phone app or mobile web interface.
The party line is probably something about "a lack of data security" with email, which would almost be funny given the current situation if it wasn't so stressful for those impacted...
This is to do with FERPA which requires that student grades be kept private. There is a small but still a significant legal risk that someone else such as a parent or roommate could have access to a student's email. And so to avoid even the possibility of a court case, schools prefer to play it safe and display grades only to a user they can authenticate directly.
This doesn't have anything to do with common sense, it's simply about legal risk. And it's not about security in a broader sense, it's specifically about privacy FERPA legislation.
There is no more risk of access to email than there is to Canvas. They are usually secured by the same SSO, too.
However, congratulations for finding the exact dodge around implementing a useful feature. Back when I worked at a university, it was apparent we had a “toolbox” of reasons to deny requests we didn’t want to do: HIPAA, FERPA, ERISA, PCI, GLBA, Title IX, ADA.
“We can’t do that integration with student health services due to HIPAA concerns.”
“We can’t implement that sign up form due to FERPA.”
“We can’t update that site because we’d have to do so and be ADA compliant and that would cost too much.”
“Due to Dining Services’ server being in scope for PCI, we can’t run reports off of it.”
“Adding that ability to Student Affairs’ portfolio app would raise Title IX concerns.”
It was great. You had endless excuses to say why you can’t email a student their grade.
It's about edge cases like someone set up your email to forward all your emails to their account without you knowing. Or other additional situations you could imagine.
There is no benefit to not emailing grades directly, from the perspective of Instructure. There is no ulterior motive here. But universities are genuinely risk-averse and their lawyers tell them that not including the grade in the email simply shuts down one more avenue for some potential lawsuit. Which costs money to defend even if a university wins it.
This isn't some kind of "dodge". This is literally just Instructure doing what university lawyers demand.
I agree with you that the email address is generally always also controlled by the school and has the same login authentication. It doesn't matter. I told you this isn't about common sense. This is about lawyers saying that it could reduce legal risk. And that is a true thing that is coming from real lawyers. Even if you disagree with those lawyers.
And Instructure isn't going to try to disagree with lawyers for its own potential customers. It's going to give the schools what they want, which is not revealing grades via email.
It's not a "dodge."
It is a dodge. Society should not just say "oh those silly lawyers". These people are not being responsible. They are not doing their jobs.
You would be surprised at the number of frivolous lawsuits and seemingly "zero risk" decisions that wind up turning into actual legal risk and legal fees.
The legal world is a lot more complicated than you think. I've been in some of these conversations. Quite frankly, you don't know what you're talking about.
Refusing to give a student their own data because of a privacy law that's meant to give the student control over their data is them failing. Full stop. There's no room for excuses for government funded entities to act in the exact opposite way that they are supposed to to avoid their fear of government imposed penalties from a deliberate misinterpretation of what the entire thing is about. That's incompetence by everyone involved. It is people going out of their way to make the world a worse place to act important. Absolutely unacceptable.
It's like if teachers aren't teaching the kids to read or add, the details about all the compliance stuff they need to worry about and how the school "can't" remove disruptive kids from a class or whatever is missing the point; the schools can't sacrifice actually doing their job at the alter of compliance, or we should just shut them down since all they do is waste resources. The compliance people should be figuring out how to shield the actual workers/create plausible deniability if the law is supposedly that stupid.
Blaming lawyers or Instructure for "failing to contribute to society" is both incredibly immature and factually wrong. It's not the 1980's where jokes about "kill all the lawyers" get laughs.
I'm going to be blunt: you seem to have a kind of black-and-white, adolescent understanding of the world where it's split up into good actors and bad actors, and good actors should do what's right (regardless of the law) and bad outcomes are the result of bad actors. But that's not how the world works. Everybody involved can be intelligent and trying to do their best, and we get suboptimal outcomes because this stuff is hard. Writing laws that protect student data while maximizing student convenience are probably never going to get it perfectly right in every situation. But insulting the lawyers or the schools or Instructure as "failing to contribute to society" or insulting the law as "supposedly that stupid" is to deeply misunderstand everything.
Frankly it's a perspective that I've only developed as I got older and realized that such excuses are poor, and that the real world has quite a few people in it who don't really care about the outcomes of what they're doing, or even understand why they're there. To me it feels adjacent to the adolescent view I often see on this site/reddit around "why is the company laying people off when they're making lots of money?" It's because those people aren't needed for anything, and those jobs aren't a form of charity. They exist for a purpose. If they no longer have a purpose, why would you keep paying that person?
If people are going to exist as obstructions to the purpose of the institution we're trying to serve, then they are useless. It's like a computer security worker saying the best way to be secure is to unplug everything, and push for policies that no one shall use computers for anything. Completely missing the point.
Finding ways to follow the law in the most risk-free way to the detriment of everyone is exactly missing their purpose in the world, and everyone should rightly call such a person incompetent and useless. It's casual acceptance of this kind of incompetence culture that slowly leads to societal decline. It's the same kind of thing as when Berkeley took down their lectures because of the ADA. How about the same state that ignores federal immigration and drug law say that actually they're going to keep giving away their free educational materials because they want universal education, and giving those lectures away is strictly better than not doing that, and if the feds want it made accessible, they can fund a project to do so?
If you don't see how extreme that is, and how much society would break down if everyone started thinking laws were optional and ought to be ignored when they prevent you from accomplishing your "mission", I just don't know what to tell you.
E-mailing a student their grade is not "breaking the law".
Not e-mailing a student their grade is not "being careful about following the law". It is just sheer laziness.
A university may develop a policy of "we don't e-mail grades" for another reason, but FERPA is not a valid reason.
It's not "sheer laziness". I can almost guarantee you that Instructure would prefer to e-mail the grade itself, and probably had the code working somewhere before feedback from universities told them to remove it.
There are absolutely cases where sending an e-mail to the wrong person is a violation of FERPA. Can you guarantee that your software will never be configured to accidentally e-mail someone besides the student? That no administrator will ever accidentally set up the wrong e-mail address? Because you're not sure if you can make that guarantee, it's legally safer to restrict it to the actual LMS login.
It’s rather simple to restrict sending email to @student.uni.edu and then further force their email to match the username and email address that is synced from the SIS.
How much FERPA compliant software have you written?
You are right that if you are creating a custom tool you can create that restriction easily.
But if you are creating a learning management system where administrators can configure it a million different ways and the university lawyers want to make sure that administrators don't set it up the wrong way, it makes sense to have that safeguard.
You are looking at the wrong level here. This isn't a software coding issue around technology. This is a policy compliance issue around people. When you create tools you have to consider the possibility of those tools being misused by an employee and mitigate those risks when possible.
Like the entire AI industry could only work by completely ignoring copyright law. Basically no software could be written if developers were concientious enough to check for and avoid patents first. Tradesmen ignore safety policies. Doctors ignore limits on hours. People do work on their homes with no permits.
Part of being an adult is exactly knowing which rules are important and which you ignore.
Corporations, universities, etc. are very different. They create policies which are documented and which their employees are required to follow. They engage in risk analysis.
"Part of being an adult" has nothing whatsoever to do with the laws and regulations that apply to organizations. You're making a severe category error.
At IBM, vim was specifically banned by legal because of reputational risk because the license asks you to consider donating to children in Africa, and IBM didn't want to be called out for not doing so. Guess which editor pretty much everyone in my org used? We also weren't allowed to move furniture because of some union agreement, but guess how many people cared when furniture mysteriously moved from an empty office room into ours? None.
At the startup, people in our satellite office in Arizona openly mocked the California HR harassment training over lunch. It was also an open secret that one of the managers started dating a report. As far as I know many years later they've both moved on to other jobs and they're still together. Nothing bad happened.
Breaking some policies will absolutely get you fired, but that's mostly around doing things you shouldn't be doing, and even then usually only matters if someone else that has some power might themselves get in trouble/have more work/lose something because of what you did. Others no one will care about. Again, part of growing up is figuring out which policies have a purpose and which came from some busybody.
I also already gave the entire AI industry as an example. We know for a fact that Meta trained on pirated material, and it's pretty obvious that everyone else does too. It's blatant industry wide flouting the law. The realpolitik answer here is everyone knows that enforcing the law here would be the final nail in the coffin for China superceding the West, so it's not going to happen.
An old lawyer joke: What do you call 100 lawyers drowning in the ocean? A good start!
(Told to me by my dad, a former attorney till he retired.)
Actual, real lawyers who work for or at real universities often do contribute quite a bit of valuable work. I enjoyed the one I worked with and think she did a great job of putting the brakes on over-regulating or using legal compliance as an excuse for just not doing work.
Of course I presume it's also not a strawman because it's not in any way some unique thing to lawyers.
Lots of fun if a department had been stonewalling for “legal reasons” and she was summoned to a meeting.
The same kind of broken thinking exists across MANY other aspects of life. Turn up the radio in your car to avoid hearing the problem that would be 'easy' to fix now and instead ignore it until it turns into a considerably more expensive repair.
I find it's often the same people that make these kinds of misguided decisions that crow the loudest about "kill all the lawyers". Perhaps because of their fear of being called out for their ignorance.
The law is a lot like an app: It has to take into account a gazillion edge cases and corner cases — not to mention that people can be ignorant and/or malicious. It really is complicated, as you say above.
Well done on not hurling insults at @ndriscoll, BTW. Personal attacks don't persuade the target, and they can turn off onlookers who might be undecided. (Competent lawyers learn early that judges and jurors don't like personal attacks and can be less inclined to believe the attacker.)
It looks very weird and is hard to understand from the outside, and unfortunately all technology vendors are on the outside.
Basically every technology has an impedance mismatch when brought into the university environment. And when you combine them together it keeps getting worse.
That's why you see things in this thread like CS professors who operate their class using pen and paper and maybe a spreadsheet.
One thing I really appreciated that she did was refuse to put e-mail disclaimers in the bottom of e-mails, because she said they had zero legal weight and actually were negative from a legal perspective, since it means people might think they have legal weight (when they don't).
Overzealous e-mail admins would periodically want to do it because it's what everyone else does, not to mention vendors of frankly B.S. software whose only value prop was adding a disclaimer to all the email that went out of Exchange or Google Workspace.
What you are saying about e-mail is simply not factual. Student e-mail is inside the FERPA environment, and is considered private to the student. It was designed to be that way. If a student sets up forwarding to go to someone else, that's their problem. The student e-mail uses the same SSO as the LMS, so it's nonsense to act like someone else could have access to e-mail.
Ironically, this incident shows they don’t have control of anything.
Everything we know has come from reddit threads / hackernews threads. There has been 0 official communication today indicating this was an attack, yet the login page was defaced by ShinyHunters.
And then wish for the death of saas and a day where you can deploy your own software you can control and modify as you need.
Does a future employer look at pass/fail vs the grade? do they care? Are there even jobs that matter enough to care out there for them?
This seems like, solving the problem but without actually seeing the broader goal or trajectory education is supposed to follow.
I don't know for a fact how pass/fail is treated by employers, but there are indeed some that look at your college GPA even 10+ years after you graduated. I suspect they don't care about the specifics of how your overall GPA was derived though, so pass/fail likely doesn't matter (unless you did really well and expected the grade to boost your GPA, and then pass/fail essentially does nothing to the GPA, thus kinda eliminating the GPA boost).
I got asked for my undergrad GPA (I graduated ~10 years ago) more than once over the last year by some finance/quant firms.
As for whether "do those jobs even matter enough," I guess it is more of a personal subjective take. I found the work that the people at those companies did (and the problems they solved) to be very interesting and challenging, I found the people working there to be extremely sharp, smart, and genuinely nice to interact with (which is an ideal work environment for me), and I found the total comp to be great. Honestly, I cannot think of much more to ask from an employer.
Canvas is mostly FOSS
"Courses were taught in a range of subjects, including Latin, chemistry, education, music, Esperanto, and primary mathematics. The system included a number of features useful for pedagogy, including text overlaying graphics, contextual assessment of free-text answers, depending on the inclusion of keywords, and feedback designed to respond to alternative answers."
"PLATO III allowed "anyone" to design new lesson modules using their TUTOR programming language, conceived in 1967 by biology graduate student Paul Tenczar."
"The largest PLATO installation in South Africa during the early 1980s was at the University of the Western Cape ... For many of the Madadeni students, most of whom came from very rural areas, the PLATO terminal was the first time they encountered any kind of electronic technology. Many of the first-year students had never seen a flush toilet before. There initially was skepticism that these technologically illiterate students could effectively use PLATO, but those concerns were not borne out. Within an hour or less most students were using the system proficiently, mostly to learn math and science skills, although a lesson that taught keyboarding skills was one of the most popular. A few students even used on-line resources to learn TUTOR, the PLATO programming language, and a few wrote lessons on the system in the Zulu language."
The full PLATO system included grade books, attendance tracking, and class scheduling, as I recall. Perhaps a University of Illinois alum can say more.
I would really like to know how much more useful the current systems are over, say, PLATO in 1992, when evaluated for pedagogy and course management benefits.
I had a lot to learn about actually developing software after I finished my CS degree.
... and assuming they have a documented, tested, and trusted restore process.
Some data was permanently lost, and then officers told reporters that multi-regional backup was not yet built because it was too hard at such a massive scale... of 858 TB.
There are probably many S3 buckets in existence that are bigger than that.
Not saying that they should've used S3, but it's definitely possible configure multi-regional backup (and a government can afford it).
Schedule a single exam and that's your grade for that subject? That's how it should work anyway, credits for work during semester (or worse attendance) are not needed to evaluate if someone learned the material, give them an exam and done.
At my school, tomorrow is the last day of exams. Most of the students have left campus. There's no time or mechanism to schedule an(other) exam.
Which to me seems the best way, you still have to learn throughout the year. Especially to avoid cheating this works nice. And as an aside, most people I know that did a year abroad in the US got 1-2 grades higher, as it was quite easy to just farm extra credits.
Using attendance is a carrot to get students to show up, which leads to better learning outcomes overall - which should be the goal.
And from the hacker's message itself, it's clear they want money in exchange for not releasing private info, not for the data itself.
Do we live in a fear based culture? Why the panic? Even if everything was hosted on Instructure's infrastructure, it's all AWS. I'd be VERY surprised if there aren't multiple way to go back to a previous state.
Most of the work and delay is to make sure they figure out where the breach occurred.
Backups can be sabotaged (turned off or schedules manipulated) or compromised (say, by lateral movement).
> Even if everything was hosted on Instructure's infrastructure, it's all AWS.
AWS Backup isn't foolproof. Get your hands on administrator credentials as an attacker and suddenly the only thing between everything being gone for good and unrecoverable even for AWS is remembering to have put a permanent deletion protection on all resources in AWS Backup.
Here in the Netherlands a data center's power source (not even the machines) burnt down, data center is offline and University of Utrecht, one of the biggest universities here, is closed. Access passes don't work, work from home environment doesn't work, student information system is down, system for grading doesn't work. No failover for any of them (or maybe it was in the same DC?)
https://nos.nl/artikel/2613485-storingen-in-hele-land-door-b...
Either way, they were under no obligation to adopt this garbage technology regardless of whether it’s available, so this is 110% on them.
Edit: No idea why this was down voted so much. I'm not defending Canvas, just wondering what the alternative would be.
But you do then have to have a sysadmin capable of managing an enterprise grade LAMP stack.
I worked at a university which did exactly this, in the UK.
It was a bespoke platform which integrated incredibly well with the rest of the systems the university used because it was designed from the ground-up to meet the institution's needs, there were regular user groups involving academics to understand what features needed to be built/worked on etc. At one point it was all OSS on GitHub too, in case other universities could've found it useful. It handled plagiarism detection (integrating with Turnitin), marking, exam grids, coursework submissions and feedback, seminar allocations, personalised timetables & mitigating circumstances.
The in-house dev team was vastly cheaper than anything SaaS would've cost, as well. It also maintained software for on-campus parcel deliveries, online exams, opinion surveys, a mobile app for students/staff, the SSO system, the course catalogue, car parking permits, a content management system and more.
My (also UK-based) university has been working on a new student records management project for years that's been incredibly ill-fated. It's destined to replace all their current systems and the first module module was meant to launch last year, except it thoroughly failed testing and nobody has heard anything about it since.
No idea how long it'll take to pull through. I don't believe it's an in-house effort.
End users who report problems:
* are ok with IT level 1 telling them IT level 3 is working on it with velocity appropriate to keep their jobs,
* are ok with IT level 1 telling them ${vendor_of_well-known_solution} is working on it with velocity appropriate for many customers, but
* are not ok with IT level 1 telling them ${vendor_of_bespoke_solutions} is working on it with velocity appropriate for one customer (if they even still exist).
https://github.com/instructure/canvas-lms/wiki/Production-St...
Or maybe consider not following the herd, and use a much simpler but sufficient system that can be self hosted, if available.
You are aware that you are posting on Hacker News, a forum for people who make their living selling software and the expertise to host it?
Does anyone have a list of affected schools?
What good is having airgapped backups and spinning them up, if they are instantly vulnerable to the same attack again?
It does depend on what the attack is, but how do people approach that scenario?
I have an idea for the midterm (pun intended): Maybe don't jump feet first into the deep end of a single point of failure going forward.
Canvas does provide a lot of value (all courses, teachers', students', and parents' contact information, all learning plans, schedules, room numbers, all grades, a lot of tests and assignments themselves, all upcoming assignments and deadlines, a lot of other coursework is in there, as are the final grades) but it shows that with external SaaS you might be one attack away from not only losing all that convenience but also in a world of hurt 'cause you lost all the data and now have to figure out how to proceed without the data and the system.
US high schools are in the middle of the finals, and seniors are getting ready for college (the transcripts to be finalized and sent out in a few weeks) so that was a scary timing.
That makes you one better than me. :( One thing's for sure--I'm never trusting it again.
I already had almost all my materials outside of Canvas and just used their API to upload it. So at least that's safe. But the grades... dang. Luckily we're only halfway through our quarter and it's not finals week.
Our instance is still down, but your update gives me hope.
Incidentally I've always hated Canvas and probably every other LMS provider, but what is particularly amusing about this current outage is that it is occurring at exactly the time when universities are demanding that all professors put all of their materials on Canvas, without exception, due to ADA compliance regulations. It is explicitly forbidden for professors to, e.g., refer to pdfs posted on a personal website.
Other commentators here seem not to understand that many faculty also do not enjoy being forced to use Canvas.
And of course the other serious concern I have with Canvas is that they are likely using all the materials faculty upload to train their AI replacements. Many of my colleagues engage in dark humor about this but I haven't noticed much action.
I'm not sure where your stereotype even comes from, because Canvas is not trivial software. You can see for yourself as it's AGPL and I assume you looked at the code before criticizing it because any good engineer would do that.
A bright undergrad could build a superior replacement in a few months, even without AI.
> A bright undergrad could build a superior replacement in a few months, even without AI.
Is quite naive. Canvas is not at all just a crud app. You can view the code yourself as it's AGPL
But it’s rarely the case in practice.
In a sibling comment right here for example someone bemoaned the difficulty in Canvas of having two TAs simultaneously grade separate parts of the same assignment. That sounds like something that goes beyond CRUD.
But more importantly any workflow system, which an LMS will be full of, has to handle the always tricky problem of how changes to workflows affect the things that are currently in the workflow. Assignments posted in course X need to be approved by person Y; some assignments are submitted for approval; person Y goes on leave and now the approval needs to be person Z. Not a simple CRUD problem.
These are things that occur to me with only a moment’s consideration of what an LMS system might need to deal with. The actual domain probably has considerable more complexity that I can’t even imagine.
In reality, Canvas does not have workflow and does not prevent race conditions in grading. I can certainly imagine an LMS that does these things, but Canvas does not.
It would probably help if you had actually used Canvas before trying to convince us that it is non-CRUD.
It's a simple question. Since you claim to be an expert on Canvas, I'm sure that you can point me to the relevant features much faster than I can sort through thousands of lines of code, looking for the one line that says "def not_crud_function()". CRUD or not-CRUD is a judgement about the purpose of a program, not its implementation.
And If you can't be bothered to take 2 minutes to click through some pages on GitHub, I don't believe you'd take the time to even read that report. So no, I'm not doing your research for you.
Edit: I will do this for you though. Here's Gemini's opinion[1]. It's quite accurate as well, and goes into reasonable high-level detail (though doesn't get into specific modules). I especially loved this quote:
> At its absolute lowest level, almost all web software boils down to pushing state to and from a database. But calling Canvas LMS "just a CRUD app" is a bit like calling a commercial airliner "just a metal tube with wings."
https://github.com/instructure/canvas-lms/pull/2630/changes/...
It has to be simple enough for the average person to use (both on the learner side and the instruction side) and have enough complexity to allow for a lot of flexibility in setup because every organization is slightly different. They have to support 50 million file formats and everything has to be backwards compatible until the end of time and everything has to load properly and quickly on 50 million different device/OS/browser combinations. Yes, there's SCORM as a standard, but even that is rickety, and an LMS that doesn't support non SCORM files is dead in the water anyway.
They're simple(ish) in code, and a nightmare in requirements.
Canvas is decidedly, not fast, fails to display even trivial files (such as source code) as well as more complex files that should just be handled by the browser (such as video), and it has a non-intuitive, verbose, and tiresome interface that would have felt old-fashioned 20 years ago.
LMSes frankly run like shit. I don't work with Canvas right now, but every one I've used has run like shit.
However, there are reasons that the complex files aren't handled by the browser: tracking and persistence. It isn't enough to make a video file watchable, it then needs to be tracked in the same system as every other training/educational material and in the same way. If you don't care whether the students actually watch the video, then yeah, throw them a YouTube link or embed a video on a personal site or just have the LMS serve a basic embed. But being able to track video, make it mandatory, make it so that it can't be fast forwarded/people can't skip to the end etc. all matter when LMSes are used for topics that are required for compliance and regulatory purposes.
I don't disagree on the interface(s). Ours is a farce and I hate it.
It's likely that they're so bad precisely because of the simple tech and complex requirements. Simple tech doesn't mean 'easy' or 'not time consuming'. But it means you're looking for developers who have a decent level of technical proficiency (to handle the numerous edge cases and flexibility the systems demand: it's not hard but things like the data structures need to be well thought out and every single piece of the system is integrated with one another in most LMSes so you can't silo work as easily) and who want to work on problems that aren't hard and require dealing with a lot of unreasonable people (in the form of their requirements). You have to allow/design for a lot of stupid things because otherwise people will throw tantrums about it.
Then on top of that, you're developing something that doesn't directly generate profit, so nobody is going to pay for it or appreciate the work you put in.
Then on top of THAT, they're fairly insulated from the actual end users.
It's just a recipe for shitty software.
> You can see for yourself as it's AGPL and I assume you looked at the code
Can you look at any codebase and tell me it's written by some of the best engineers and it's not trivial?
I completely agree that it is not trivial software in the worst sense, it tries to do too much, while not being particularly good at any one of those things, and is way too rigid for how diverse the needs of different courses might be even inside a single faculty. And saying "It's AGPL, just self host and add your requirements to it" is not really useful, that would mean way more money and effort than what a university's overworked IT dept. is capable of.
> it is not trivial software in the worst sense, it tries to do too much, while not being particularly good at any one of those things, and is way too rigid
What I meant is they aren't capable of building AI capable of replacing professors. I still consider it a reasonable assumption, as it has nothing to do with how well engineered canvas is. It's a different competency than instructure would have, and I've heard from insiders instructure has been spinning their wheels on way more trivial AI challenges. I also understand well how hard it would be to create AI that replaces professors and how the current best AI from Google, Anthropic, OpenAI is orders of magnitude away from being able to do that.
An engineering culture can change a lot in 10 years, and a company's engineers' ability to do stuff depends both on the individual engineers abilities as well as the company systems and culture.
Instructure (Canvas's developer) partnered with OpenAI last year [1], about a year after KKR and Dragoneer (PE firms) acquired it [2].
[1] https://www.forbes.com/sites/rayravaglia/2025/07/23/instruct...
[2] https://www.pehub.com/kkr-and-dragoneer-complete-4-8bn-take-...
That calculus is about to shift.
I would guess these plugins are chosen so a majority of user won't want to live without them.
It also seems these plugins "link" to canvas-lms, so keeping the proprietary would be a GPL violation if anyone except Instructure holds part of the copyright to Canvas.
The administration has so far opened with one “Canvas said” and then an hour later one “Canvas is down indefinitely” email noting that they’re aware it’s serious.
(Canvas is a glorified wiki for teaching students, with quizzes and such, for those unaware.)
That's my biggest fear.
If my peers are any indication, a whole lot of TikTok, Reels, Twitter, Discord, and other such mind-numbing platforms.
The types of platforms I would consider 'substantive' (or, at least, more substantive than those platforms) are definitely on the way out.
The few times friends have seen me browsing Hacker News or a certain Mongolian basket weaving form, the first thing they comment on is how confusing the interface is, and how old the site looks.
I truly don't understand the mentality, but if your site doesn't take three seconds to buffer a simple text drop down menu, and have JavaScript elements load in mid-scroll that bump elements around the page making you just barely miss that button you were trying to click, then your site is seen as 'inferior' or 'sketchy'.
Perhaps I've just had a bad sample, but I've experienced a variety of different environments by this point, and by and large, I've seen more people in my generation act in that manner than not.
It's true that HN looks old - it looked old before you were born, probably - but (a) I have no idea how to change it, and (b) HN is a long bet on plain text. If the smartest young people lose interest in reading, I'm ok with HN dying for that reason. I just don't want it to die for any cheaper reason.
I do find that my peers that now read HN used to be judicial about curating a Reddit feed and mostly otherwise limited on other sources. Short-form content is addictive and as nearly as unavoidable as sugar, but many of my brighter peers work on reducing that intake. Long-form YouTube is also something I find to be a marker of someone who is seeking knowledge. Many of my peers do scroll Twitter and TikTok all day, but I find that those who are easiest to chat with are those who have already scrolled HN today and want to discuss a particular article they know I would have seen. I've had conversations that start with "Did you see that TikTok?" and conversations that start with "Did you see that article on HN?" and the latter is always more engaging.
> Long-form YouTube is also something
Yes, we hear that often too. I didn't mention it above because it's not text, but in terms of how people spend time and where they go to learn things, it's a huge alternative.
I wonder sometimes how HN might interface with the videoverse. I can't imagine having video on the site but I can imagine making videos based on HN threads or articles that have appeared here. I just can't imagine me making them!
That said, it's a commercial closed-source single point of failure.
(and btw, they do say "twitter")
I don't think that any of them do, but I'm a Canadian math/physics major, which is slightly outside the target audience for HN.
> of the ones who don't, what do they read instead?
For the social aspect: mostly medium-sized Discord servers. For the news aspect: nothing at all. Both of these do have some advantages, but it's still a bit of a shame, because the Discord servers aren't indexed by Google, so they're hard for outsiders to find, and not reading the news means that they're missing out on some of the cool new tech advances.
Same question for you as https://news.ycombinator.com/item?id=48065589, btw: what do your friends read besides HN?
Note that little of this really helps the students that it is supposed to help, because as you wisely point out, raw HTML is almost by definition extremely accessible. I work in a field that uses Latex and the source code of Latex should also be considered more accessible than the compiled pdf. But for university administrators the only important thing is that the accessibility metric that appears (or used to appear, before today!) on Canvas shows 100% accessible.
Nobody has infinite energy, and disabled people don't have infinite social capital. It's a shame when energy from that shared pool gets spent on things that don't really impact meeting people's access needs.
And the other thing is that everyone's access needs are different. It can certainly be useful to try to set a baseline or propagate common guidance. But the most important thing, especially in a university setting, is for instructors to be flexible and responsive and for classes (and non-teaching workloads) to be structured in a way (e.g., small enough) that supports that.
I think metrics like "100% accessible" might even be dangerous. It makes it easy for able-bodied people who aren't in direct contact with disabled stakeholders to pat themselves on the back without actually knowing what's going on.
Bleh. Good luck doing right by your disabled students and disabled colleagues, and good luck resisting the bullshit.
That said there is certainly a lot more work that needs to be done in this area. Hopefully these regulations over time bring out practical positive change. Time will tell.
I'm a prof. When I have a student with special needs in my class, the administration tells me ahead of time. I make the necessary allowances - and those differ from case to case, anyway: whether it's extra time in exams, or someone who is deaf, or someone who is blind, or whatever.
When it happens, I make the necessary allowances. When I don't, then...I don't.
The obsession that everything has to be 100% accessible, for every kind of disability, all of the time? That's just nuts, not to mention a complete waste of resources.
It's been long enough that I can't claim to be in touch with the current generation of teaching faculty. But it might be an element of that, combined with the desire to provide accessibility for the handful of students who do in fact need the accommodation.
The MS services have not improved teaching at all. What they do, is fragment communications, and add ever more places people have to look, in hopes of finding things.
But the administration loves them. "The bureaucracy is expanding, to meet the expanding needs of the bureaucracy."
Thankfully, I store my teaching materials on my personal non-uni webpage, and the student's marks in my office's computer (apart from the MS-based Uni system).
Whenever something happens with MS, chaos ensues throughout the whose Uni and the students end up paying the consequences.
No this will not stop this and companies need to be held accountable for their lack of security investment. Every attack should be investigate if the company met an agreed industry standards best practices and staffing, etc. The penalties for not meeting the requirements should be punitive.
Kids from the local uni having a lark, stalkers, vindictive ex employees, local gangs, criminals who understand their victims because they hail from the same community. These are your local hackers. Sift them from the nation states and international crime groups, then deal with the International as a matter of diplomacy. Because we do this so poorly locally, we have little ammunition to when it comes to diplomacy. "reduce attacks by your crime groups and we buy your natural gas, seel you wheat etc"
Want more motivation?- 75% of the local attacks by volume send funds back to terrorist or separatist organizations.
It is not an in-soluble problem. Sentences are a fraction of the answer, effective and receptive reporting processes are more important, then government backing for investigation and enforcement, then policy around home-team activities (ie don't do the bad things yourselves Mr Gov). Deterrence comes after all that.
There are already significant penalties for doing anything like this. The guy involved is in prison for a very long time. I don’t recall the exact number of years but I do remember it was so long that he wasn’t going to see his kids grow up.
I don’t think anyone who puts a little thought into a crime like this doesn’t understand that the penalties are already very huge. You don’t get a slap on the wrist for extorting a company (or person, for that matter)
Really, though, if you want someone to blame, Instructure is not a particularly compelling target. Let's review:
1. Iran is intentionally targeting infrastructure due to a war started by the current administration.
2. China is actively seeking corporate secrets to steal and commercialize for themselves, spurred by extreme protectionism and retaliatory tariffs.
3. North Korea is doing anything they can -- including just taking a remote job by proxy -- in order to extract any money.
4. And Russia is working with and aiding all of them, after everything else going on has forced the embargo to break.
5. All of this while completely alienating every single one of the United States' allies.
6. Meanwhile, the American DHS is currently shut down.
7. And this is after Trump cut funding and personnel for CISA severely enough they've had to end the contract with MS-ISAC, meaning all state and local entities can only remain in the organization if they foot the bill for it directly and CISA and other agencies responsible for cybersecurity are more thinly staffed than they have been in decades.
In short, the current administration systematically disassembled all the protections we have built over the last 100 years, and then placed infrastructure -- schools, in this case, but also power companies, water treatment facilities, communications companies, local governments, hospitals, food producers -- directly on the front lines of the modern geopolitical conflict.
That vast ocean that has kept us safe historically is a poor moat in the modern era.
An investigative body, the same kind that determines the who, the why, and the how when an airliner crashes or a bridge collapses. Obviously a lot of work needs to be done to get from point A to point B, and it won't happen overnight, but software development is currently a deeply unserious profession and at some point a genuine software engineering practice needs to be developed.
I am, perhaps naively, slightly hopeful that the LLM bullshit plaguing our industry will be the gust of wind needed for the house of cards to collapse and governments to realise that allowing the entire world to be vibe coded is not sustainable.
Aviation’s safety record is not coincidental.
As someone else here said, software, for the most part, is a deeply unserious industry. The stakes are so comparatively low and the consequences less obvious that it’s a lot easier for companies like intuit to maintain their supremacy simply by being entrenched, having strong sales teams, and the hearts & minds of non-technical managers.
In recent times it seems Boeing has been flirting with enshitification and half-assery but critics are not quiet and not falling on deaf ears
You may not be aware, but there are thousands of non fatal incidents reported per year that just don't make the news.
There is a strong culture of self reporting instilled right from basic flight training, even when there is no damage or injuries, and even when the incident would have never been noticed by the authorities. You are almost guaranteed not to face consequences if you are open and honest about an incident. The FAA openly says that they would much rather educate than punish, and they tend to do that with pilots who own their mistakes. As long as there is no intent behind the fuckup, pilots are unlikely to lose their job, let alone their license.
This just in: Anthropic, Harvard and Jimmy Kimmel have been investigated and found guilty of not securing their infrastructure.
ShinyHackers, obviously.
It should be illegal to host insecure services, especially when you're dealing with PII. Breaches keep happening and nobody gives a fuck, because the worst that'll happen is you might lose a handful of customers and buy some "credit monitoring".
Incidents like this should be followed by an audit and charges being laid. Send corp officers to jail for negligent security failures. If you can go to jail for accounting fraud, you should be able to go to jail for cybersecurity-promises-fraud.
They claim to be compliant with a number of security standards [1]. I would love to see a postmortem audit of how much of this they actually implemented.
[1] https://www.instructure.com/en-au/trust-center/compliance
I do agree with the audit and punishments for clear failure to adhere to established standards.
I'm not sure that's a fair analogy.
My analogy would be: of course buildings have to be built to withstand gravity. That’s a natural part of the world that cannot be eliminated.
Buildings are built to stand up to natural forces. But not to, for example, the threat of a malicious actor crashing a plane into them. That isn’t typically considered a reasonable thing to architect civilian infrastructure for.
When you built IT infrastructure likewise you should build it to handle the natural forces it will be exposed to. But are you as accountable for securing it against the acts of malicious parties as a structural engineer is for securing a building against gravity, or as accountable for securing against those acts as the structural engineer is for securing that building against terrorists?
If someone threatens you with a knife and gets you to hand over your wallet, your bank doesn’t get to say ‘you should have hired better security’ when the mugger uses your credit card.
The problem here is the mugger, and that’s who the state goes after. Even if the victim walked into a bad area. Even if the victim could have defended themselves.
Same with ransomware attackers. They are the problem. We might encourage potential victims to behave in ways that make it less likely for them to be targeted. But if they are targeted, we should still focus our societal disdain on the criminal not the victim.
If the perpetrators of this hack were caught and in a developed country, they would certainly be prosecuted for their crimes and not get off light (especially if any data is actually leaked).
But I do think it should be much more states’ responsibility to make their domestic network safe for citizens and businesses and institutions to operate.
That standard is likely to lock people into buying some pretty bad software, but it does little to ensure that they're running reasonably secure systems.
Even if you leave your door unlocked, if someone walks in and steals your stuff, it's a crime. The state has an interest in prosecuting crimes even if the victim didn't do everything they could to prevent it.
A better version of your analogy would be if your landlord failed to repair your front door in a reasonable period of time and as a result soneone walked in and stole your stuff. Yes the theif is the primary responsible party, but the landlords negligence in maintaining the property probably also exposes them to some liability.
P.s. This is neither here nor there, but restitution is a part of criminal law.
But the post I was responding to said it should be a crime to have unsecured systems.
That is equivalent to saying it should be a crime to leave your door unlocked.
Restitution and retribution are the components of justice [1] entirely about "making things alright for the victim."
[1] https://www.unodc.org/e4j/en/crime-prevention-criminal-justi...
Just take a look at the recent Epic vs. Health Gorilla lawsuit to see how nonexistent the protection is around exchanging your medical records, one of the most sensitive types of PII.
Instead, there should be standard civil penalties for leaking various degrees of PII paid as restitution to the affected individual. Importantly, this must be applied REGARDLESS of "certification" or whether any security practices were "incorrect" or "insufficient". Even if there's a zero-day exploit and you did everything right, you pay. That's the cost of storing people's secrets.
This would make operating services whose whole "thing" is storing a bunch of information about individuals (like Canvas) much more expensive. Good! It's far to cheap to stockpile a ticking time bomb of private info and then walk away paying no damages just because you complied with some out-of-date list of rules or got the stamp of approval from a certification org that's incentivized to give out stamps of approval.
For most individuals impacted by these hacks, appropriate restitution would be $0. Anything more than that would go beyond making them whole.
Which is what the comment above was referring to. "Most people". Not "all people".
What? Why? Who died? This whole thing is perfectly dealt with through civil process.
Here’s an example. https://hacks.mozilla.org/2026/05/behind-the-scenes-hardenin...
These problems will continue as long as it is legal to operate in an unsafe way.
We've learned this in every other industry, but we can't seem to accept it in software. One of my hopes for AI is that it reduces the cost to behave responsibly to a level where this absurd resistance to acting responsibly erodes.
Every service that is online will be hacked eventually, it's only a matter of time.
Time is the most powerful force in the universe.
Like is that your actual model? I’m curious
There are no shortage of coins and no shortage of sketchy exchanges. The platforms do work with LEOs, when asked, but my understanding is that unless the perp was a serious nonce, chasing the transfers themselves is a fools errand.
It's very easy to play with lives that aren't yours.
It's a familiar example of the perennial "[THING] could be solved overnight if [PERSON_OR_GROUP] would just start taking [THING] seriously" trope.
The best response to a cyberattack on critical systems is to take security seriously. Document the offense, avoid the same mistakes and invest in penetration testing. Of course, nobody is incentivized to do that until they're attacked, so the cycle perpetuates itself.
When appropriate. I.e. never.
I think in principle, its sound. Im also just baffled hearing anecdotes from friends that are in big corp world and hearing the type of incidents they have, and how they respond to it.. It makes me wonder if there is enough capable talent to go around for the "boring corp" crowd.
Hint: I don't think there is nearly enough talent to go round, but for these companies, its either that they think they have solid experts (and didn't), OR its not a real priority until you get hit.
2. The payout to the hackers should form part, but not all of the penalties. Pay those guys for their great service to humanity they earned it.
If you're going to get the chair you might as well murder some witnesses or destroy some systems to hide the fact you got hacked. "Hack? What hack? Our servers all burned down in an arson attack".
Apart from the 4% of the total worldwide annual turnover fine that theoretically could be levied under GDPR, but has never been imposed in full.
One way to weaken any group that works on trust would be to make them less trustworthy. That way victims wouldn't be as confident paying the criminals and thereby making the effort by the criminals less attractive.
That makes as much sense as illegal to give your wallet to a mugger.
I.e. no sense.
a loved one, gun to the head: "please pay the ransom, i don't want to die!"
what's your play now? save loved one, and go to prison? or worse, bank blocks transfer, and they die?
go ahead and tax ransom payments (0 tax if human life at risk, 10x otherwise) if you have to, but making it illegal feels disconnected from the messiness of the real world. then, go after the attackers.
We're talking about vulnerabilities that have existed 10+ years but nobody noticed until AI.
You seem to think "if it's illegal it won't happen". Instead you need to think about unintended consequences and what would actually happen if this were law. People would hesitate to contact the police for help before they've decided, or not do it at all. And not report it.
But it turns out that MIT used to have their own homegrown system, and recently switched to Canvas. Bet they're regretting that now.
The build vs. buy decision seems to have swung very hard toward buy in the last decade, and I think that's a shame. Yes, orgs need to focus on their core competency, and sometimes that means outsourcing things that aren't core competencies to third parties. But there are always downsides.
IT staff who are ambitious and talented don’t last long in education. The pay is very low compared to industry. Where I worked, you could retire with a comfortable pension after a number of service years, so the IT staff outsourced as much as possible so they needed to take zero risks to their nest egg. Blame all the problems on the consultants and do as little as possible.
It’s literally where dreams go to die.
MIT is known for the brilliant professors and students but at the end of the day, running a university is pretty standard stuff. They don’t need a genius rockstar to admin the courseware servers.
My highschool, for a while, had a website, which was eventually replaces by a large corporate CMS. Was the website as complicated or complex as the CMS? No, you would have needed to know HTML to publish to it. The CMS was no doubt "more user friendly", I suppose.
But … the original site had a soul. It was unique to the school. There was a student directory! All lost, because the CMS meant utter standardization between all the schools using it (their pages were all identical, except for each got like a different picture of the school as the banner at the top) and the CMS did not do directory anything.
Of course, the directory largely didn't matter in the end. (This was when you needed people's landlines! Quite laughable nowadays…) But it was still sad to see it lost, and several of us students worked on it, which provided us with some early real-world experience.
A large number of my college professors published their own sites, too, where they'd put their lecture notes, homework, etc. I loved those far more than I loved "Canvas" or whatever the ugly LMS we used was.
it's MIT.
As a system provider you take responsibility not only for maintenance, but also to tank the blame once something invariably goes wrong
And it's pretty easy to customize which is nice.
Throw it in an auto-scale ECS cluster and you have something that goes from 100 students to 20k easy.
A lot can change in 10 years, sure. Maybe Moodle is better now (I doubt it). I'm all for self-hosting a LMS. But, can we at least self-host a good one?
(I don't have experience in hosting either software so I can't really comment beyond that)
I believe the same applied to the professors themselves, although that was hardly enforced.
The one they had before Canvas was very very inadequate.
edit: also some of the more popular cs classes have custom websites and don’t really use canvas, but that isn’t the centralized IT department’s doing.
It made it so much easier if you wanted to know more about a course before taking it, so much easier to find extra practise materials (last years assignments and exam etc).
Now its all in Canvas and only reachable for those who are enrolled at the course.
This would be like TurboTax "scheduling maintenance" on April 14th in the US.
There is a lot of people who likely are unaware the latest outage is because they were compromised again.
Them marking the incident as 'Under Maintenance' means the status page isn't reporting this as an outage and adding to downtime%.
The incident yesterday was technically from April 28th, with most communications coming out on the 2nd and 3rd, with it being "Resolved" yesterday.
This incident is the second attack, because they failed to secure their infra again. Everything being reported is a bit delayed, which makes it seem like this is a single attack, not technically two instances.
It's not unreasonable that non-technical people would expect paid cloud services to be good custodians of the data entrusted to them.
These services also do everything they can to encourage you to work within the online platform rather then working offline and then uploading.
For example, there's no easy way to author a quiz, set up the answers offline and then later upload it.
Last month it was a presentation. She had to make a poster that would be displayed on the big electronic "whiteboard" running Windows of some sort. The page layout software was so terrible that she repeatedly deleted the entire thing on accident moving text around.
This month, it was a short paper she had to write in Word, but through Teams. Literally, the Word icon is in the Teams sidebar, and she also had all kinds of trouble with it freezing or misbehaving.
In both cases, I advised her to write all the content in Notes in macOS and when she had it all ready to go we'd paste it into the crappy software so she didn't have to worry about losing any more work.
Long story short, she's non-technical and she's learned a very valuable lesson about these systems and how much trust to place in them.
One thing to target coroporations but leave the students alone....
Heard you loud and clear sheesh
I personally love em-dashes and hate that they're a sign of AI writing now.
That doesn't excuse any of their other messaging though.
Also looks pretty bad their whole platform was compromised by the same hacker group again.
I also ran the entire DOMPurify sample XSS and managed to find one way to download custom content onto someone's computer.
I'm actually much more interested if there is any financial liability for Instructure here? It's interesting that it's the universities being ransomed, while the technical failure was Instructure's. We're used to uptime SLA's -- what about security breach SLA's?
Don't ransom all your eggs in one basket
My guess would be they get likelihood of getting paid when blackmailing 9,000 schools (at least a few would pay up) than blackmailing Canvas/Instructure.
I don't think any SLA/terms would change who gets to feel the pain.
Also yeah there is value in being able to blame another party, and also being down when everyone else is down.
I dont think a competent CS department requires their being a homegrown or on-prem system for use in the university. That could happen, but if resources could be better spent by purchasing rather than building, then that should be the correct choice.
Universities which do have large agriculture/farming related departments often operate their own small scale test/development/experimental farm.
do you mean equivalent ?.
Instructure, "the developer and publisher of Canvas," was founded in 2008 [1].
Edit: https://status.parchment.com/ says "While Canvas, Canvas Beta and Canvas test are currently unavailable, we are simultaneously monitoring all of our other product environments, including Parchment. We continue to see no reason to believe any Parchment resources have been impacted."
We received communication that Canvas is down for "Under Maintenance" although it seems ShineyHunters have compromised Canvas again with that message you posted.
We do not see that message anymore, although all instrucuture.com URLs are down. The list of schools in the ShinyHunters publication can be found here: https://web.archive.org/web/20260507042014/http://91.215.85....
Original now shows 404.
I totally understand why a university wouldn’t want to bake their own learning portals but just feels like such a single point of risk to use third party solutions for something like this.
Back in my day… all we had was a school email via on-premise services. I guess we registered for classes in a web portal but that’s about it. The idea of online class was entirely foreign at the time. Ain’t nobody hacking a blue book.
Well not with that attitude
Canvas quizzes cannot beat the anticheat of having another screen out of view. Lock down the browser all you want, it is pure security theater.
Best anticheat is still taking the quiz on paper in the classroom. Best way to engage with students is to still speak with them and ask that they might speak up.
As a faculty member at a large university…I have a deep respect for the impossible job of university IT departments.
We originally rolled our on LMS decades ago. When we switched to canvas we kept the home brew running for five years past its expiration date because faculty refused to remove their files. Finally each one was manually moved by IT for the recalcitrant old faculty.
They are large databases yes but they do a lot of small and large things that that analogy glosses over
I used to work in academia and am now an LMS admin (in private industry). I've interviewed for LMS admin positions at educational institutions and each time I've ended up walking away. The questions I was asked at the last interview revealed what a ridiculously unplanned, spiraling mess their system was and that I would have no agency over it. No, thanks. And it was clear the reason for this was faculty recalcitrance and an inability to tell them no. Each one wanted a special plugin/special way of doing things, causing a giant mess of insecure bloat, and a fair amount of interview questions always amount to 'how do you wheedle faculty into doing things/placate their egos to keep things running?'
I'm not a rockstar candidate either: I'm a disabled, geographically-constrained, self-taught(ish) sort-of techie. The disability means I have substantial holes in my resume/work history, etc. I don't have a CS degree or any kind of formal IT education. If people at my level of knowledge are looking at these jobs and passing because they're not worth it, I can't imagine the actual pool of people who get hired is great.
LMS admins in particular are going to be harder to find/retain because we tend to have options we can jump to that would be less onerous than doing LMS admin for a dumpster fire. I could go straight IT or full Instructional Design, for example.
In private industry, I can tell people to kick rocks if they want to do something that the system doesn't support/is a really bad idea. And if I can't, I'm not held responsible for the consequences.
They used to, in the pre-cloud/SaaS era; and they were much simpler and better UX than the slop that they're renting today, because the actual users were not far from the developers.
The amount of corner cases and performance requirements during rush times (semester start) made it really infeasible for a university to roll their own.
* German universities have this funny system where 51% of such boards are controlled by the professors and the rest is made up of other employees/staff and students. They call it academic participation.
There is a saying in the software security industry that (I'm paraphrasing from rusty memories) a system is secure if the cost of hacking it is higher than the value it protects.
Each system being completely distinct from another means that the cost of hacking the average student goes up by 9000 (from the article, Canvas is used by 9000 schools).
Still not saying that rolling out your own is the preferred solution, but the idea is not as ludicrous as it would seem, and should definitely be entertained and discussed, at least.
But also, the cost is much, much higher to the institutions, which is the salient point. You're going to spend years developing a system, deploying it, training staff and students, supporting it. I see mentions here of in-house systems being developed much more cheaply and I don't believe it. The economies of scale are at work.
I worked at a university for many years and I can't recall anyone I'd consider to be a competent software architect working for the IT department. Hell, we had students writing major webapps that kinda sorta worked well enough.
Brought up a question I've had every time I read about these leaks... what kind of pipes do these shadowy groups have that they can grab all this data? I've spent days waiting just downloading a few 100 of GB from OneDrive. How do they grab all this data, are they just slowly gathering it for months via a compromised desktop somewhere, or if not, are the companies not monitoring for unexpected massive amounts of outbound traffic from their database or file servers?
QUOTE
To our Instructure Community,
We know that for many of our customers, concerns about the potential publication of data related to this incident remain top of mind. We want to acknowledge those concerns directly – we understand how unsettling situations like this can be, and protecting our community is also a top priority for us.
With that responsibility in mind, we reached an agreement with the unauthorized actor involved in this incident. As part of that agreement, the data was returned to us, we received assurances that it will not be further shared on the dark web or elsewhere, and we received proof that any copies of that data were deleted. Further, we have been informed that no Instructure customers will be extorted as a result of this incident, publicly or otherwise. While there is never complete certainty when dealing with cyber criminals, we believe it was important to take every step within our control to give our customers additional peace of mind, to the extent possible.
We are sharing this update in the continued interest of transparency and so that our customers know that we have addressed this element of the incident directly. To reiterate, the agreement covered all of our customers, and there is no need for individual customers to attempt to engage with the unauthorized actor.
We appreciate your patience and trust as we continue to respond to this incident thoughtfully and comprehensively. We remain committed to providing meaningful updates as our work progresses.
Regards,
Steve Daly, CEO, Instructure"
END QUOTE
doesn't seem that scheduled to me
That's just the quickest page/status update to throw up; it was a one-liner to push it live back when I was on the deploy rotation.
I'd hazard a guess they have more important things to worry about right now than exact status page messaging ;)
Funny how a lie is always quicker than the truth...
In the 99% of cases where this status page is used, it is in fact the truth: we'd throw it up when we had e.g. data migrations to do as part of a rollout that wouldn't allow for our normal zero-downtime deploys.
So no, lies are not always quicker than the truth.
There's plenty to criticize Instructure about; let's not go reaching for straw men in the process.
Figures. Else people would not fall for the lie.
edit: here's the list of impacted universities (unsure if they all have their canvas instances offline, but i'd be surprised if not): http://91.215.85.103/pay_or_leak/instructure_affected_school...
dig canvas.ucdavis.edu
[...]
;; ANSWER SECTION:
canvas.ucdavis.edu. 1974 IN CNAME ucdavis-vanity.instructure.com.
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.125
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.103
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.15
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.18
;; ANSWER SECTION:
canvas.duke.edu. 300 IN CNAME duke-vanity.instructure.com.
duke-vanity.instructure.com. 60 IN A 18.173.121.125
duke-vanity.instructure.com. 60 IN A 18.173.121.18
duke-vanity.instructure.com. 60 IN A 18.173.121.103
duke-vanity.instructure.com. 60 IN A 18.173.121.15Back when I worked for Instructure ~10 years ago, Canvas was effectively a single, giant, monolithic multitenant app with one instance backed by several thousand app servers and ~100 separate Postgres database clusters that any app server could talk to.
Schools were grouped onto pools of app severs and Postgres database clusters more or less according to locality and cluster availability. I want to say a handful of the largest schools got their own clusters, but I'm not certain, and at any rate their clusters could certainly all talk to each other.
It was actually kind of neat from a technical perspective: any Rails model across the entire Canvas world could have a "foreign key" pointing to any other Rails model anywhere else. Among other things, this allowed for users who could administer multiple Canvas organizations, even if those organizations resided on different Postgres clusters. https://github.com/instructure/switchman is their gem that made that all work. (I put "foreign key" in quotes because the whole thing was implemented in software, not with actual database FKs, for obvious reasons.)
---
Of course, the massive downside to that sort of thing is that if you manage to pop one Canvas app server, you have the keys to the kingdom. I wonder if they'll sharpen the edges between clusters in response to this...
---
(Disclaimer: I left Instructure back in 2017; much could have changed since then, and my memory could be faulty about the specifics. Caveat emptor.)
Someone dumped the content into a google doc on reddit[1] if anyone's interested.
[1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT...
> Someone dumped the content into a google doc on reddit[1] if anyone's interested.
> [1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT...
Thanks for linking this. Ended up finding my kids school district on the list unfortunately.
This suggests a bad actor at any institution could do the same thing done here. No?
I believe FERPA's PII provisions apply to Canvas and contractors handing PII in general (at least as interpreted by the Department of Education). Now, will Canvas be held accountable by ED in this administration? Hah – DOGE probably ran that through the shredder as well.
What we don't have access to includes:
* Already graded work
* Ungraded work
* overall adn assignment grades
* lists of students and student emails from the course
* messages from students that are often sent through gradescope
Just...complete implosion.
Let's not side with the parasites.
What did Canvas PR do except do a poor job? Doing a poor job of PR is a whole, whole lot less worse than actively destroying people's lives for profit.
Is this accurate? Or is this still an ongoing issue?
https://github.com/instructure/canvas-lms/wiki/Quick-Start
> It is recommended that you have at least 150GB of available hard drive space, 8GB of RAM, and a quad-core CPU to use this script.
As far as I can tell, this is not for running a production environment with assets. This is just the development environment.
Our whole testing center is down. This is inconvenient, but mainly it's amusing. I swear strangers are talking to each other more. I'm noticing people just sitting in the sun and relaxing. Nature is healing.
(Of course, plenty of people have also just finished their exams, so it's hard to know the cause.)
Any idea what data Instructure-and-also-now-ShinyHunters even purport to have beyond names, profile photos, pronouns, homework assignments, school communications, phone numbers, and email addresses?
i.e. What makes this threat so different from what any old data brokers have already scraped?
What leverage besides aura farming do the ShinyHunters really have?
All I can think of that's really valuable is passwords. And private communications in Canvas DMs. But if you're being at all intimate over your school email, that's kinda on you.
Anyway surely Instructure only stores user public keys or something?
Alternate history question: If they just sold the data, never revealed the hack, and didn't make a scene, from a customer perspective, how different would this be from business as usual?
I'm a software dev who was affected by the outage. I was working on an app that connects to the Canvas SAML endpoints. One minute I was able to run my code, the next I couldn't. This was a little after 17:00 EST.
We already bond over how awful the Canvas UX is (and she has a bunch of Chrome extensions to improve it.) Now we’ve got something else to gripe over together.
haha i went to go check and they haven't merged a PR since 2017
Everything from middle school up to grad school.
It's a particularly interesting time to have this happen too -- many finals going on now.
I lost access when I left as it was tied to my work email. I downloaded a lot, but there was still some useful stuff on the boards.
I wonder what the havkers found out about me. Perhaps the class notes will be lifted to train AI, higher quality than a lot thats on the internet anyway.
https://www.abc.net.au/news/2026-05-08/students-lose-access-...
I'm friends with a professor who complained to me a couple times about how sometimes he will need to scroll through pages and pages of courses he taught in the past. He also mentioned that profs aren't able to delete their own course shells either.
[1] https://digital-strategy.ec.europa.eu/en/policies/cyber-resi... [2] https://ec.europa.eu/commission/presscorner/detail/en/ip_22_...
Consider surgery instead of software development. There are general best practices, but the difference between a good surgeon and a poor one is a small number of deaths. Malpractice insurance is high. Litigation is constant. And patients still die on the operating table. It's unclear what all the malpractice tort law actually gets you in the end.
In most of these cases, the companies involved did NOT follow standard security practices.
I am pretty sure that is what people mean when they say "held responsible", they mean "held responsible for failing to follow standard security practices", not for the actual act of getting hacked.
I like this analogy, but deaths shouldn't be the leading indicator just an indicator. Family member had a surgery with well known procedures, say removing a gall bladder. Unfortunately, this surgeon skipped a step in lieu of setting a record for fastest procedure. Because steps were skipped, the gall bladder was not scooped into a net to avoid spilled gall stones which resulted stones spilling into the abdominal cavity requiring numerous follow up surgeries to remove the spilled stones as they made themselves known. So clearly not following accepted procedures should be a clear win in a malpractice case, yeah? Wrong. No doctor would testify against the surgeon and the case was dismissed. I feel like this is exactly how it would work in software security incidents as well.
That was the foundational premise of Dr. Atul Gawande's book The Checklist Manifesto, an expansion of his article The Checklist in The New Yorker [0]
[0] https://www.newyorker.com/magazine/2007/12/10/the-checklist
But even if they do everything right, is it really fair to let the companies just shrug their shoulders and say "it happens"? While their users are the ones who really get hurt.
Is that really the analogy you want to use the bolster your argument? Licensing was forced on the medical profession because of rampant quackery causing a large number of deaths. Some of the horrors that went on before enforced medical licensing are well-nigh unbelievable, e.g. https://en.wikipedia.org/wiki/John_R._Brinkley
Just like 100 years from now, many of today's medical practices will also be seen as barbaric.
If we increase the penalties for a company being hacked, you create even MORE incentive for hackers to try to break in, because if they succeed, they have a pretty big stick to threaten companies with when demanding a random payment - not only will the company have the negative effect of the data being leaked and the PR that accompanies it, they now know that if they don't pay and the attack becomes public knowledge, they face a big fine or other punishment.
A company is much more likely to pay a big ransom if they know they are just going to end up paying that much or more in fines if they refuse the ransom and report the hack instead.
If you take this route, and increase punishment for being hacked, you are making a pretty big bet that the main reason companies are hacked is because of poor security practices. I am not sure if that is true or not.
I'm under the impression files are getting released 12th May. I don't see any reporting on 800GB?
They moved away from Teams because it objectively sucked, but I haven't heard of widespread compromises like this in Microsoft's systems so...
Of course if you can't complete your exams because of this, that's more of an issue!
It's always been as stupid as requiring that your chalkboard, chalk, chairs, bluebooks, pens, paper, gradebook etc etc all come from the same company.
I, for one, am very much looking forward to my IT Gov council meeting tomorrow.
I'm honestly surprised more people aren't talking about this.
I mean, maybe it changed in the last 10 years. But I was a TA grading CS majors for a while. Their C capstone or what have you.
Some were decent but naively coded. Most were pile of shit half hazardly put together so it output what is needed to get passing grade.
But I agree with you in spirit!
The timing probably isn't a coincidence. Great time to stress out students and staff alike. Hopefully it doesn't affect them too much in the end, but I imagine it will.
this is really, really, REALLY bad. it's not great that names/emails/etc will potentially be leaked, but also private messages between students and instructors. and since many of the campus systems rely on canvas integration, things have pretty much ground to a halt a week before finals.
after they were breached on the 1st of this month, instructure had an announcement yesterday that "everything is great! we're good! hackers are gone! we've rotated our keys!".
no. nothing is great. we are not good.
Is that a Pokemon reference?
This will keep happening, more and more, and never stop, until we create a software building code and legally require it for all online businesses.
Universities, Parents: ya'll actually have the political and economic power to get a software building code passed. This incident isn't the last.
Shame on your existence basically.
...what does that DDB DNS issue have to do with anything?
And GitHub doesn't provide a way to record grades that remain private per student last I checked, much less sync them to the university, or 99% of other things Canvas does.
I don't love Canvas, but it's far, far preferable to a world without it.
last I checked it appears grades remain private per planet or so ...
Or is it an entirely different class of beast?
Canvas generally is the 'easiest' to use, and the 'cleanest' looking one although D2L Brightspace is pretty good too. Moodle out of the box is pretty confusing and ugly, but I've seen some heavily customized instances that look a lot better. Blackboard is the worst of the bunch IMO.
Looking into the payload they sent me this is how they hijacked the screen. Everything in the payload is unchanged except for one line of code:
<link rel="stylesheet" href="https://instructure-uploads.s3.amazonaws.com/account_9363000..." media="all"/>
This links to the following styling sheet:
@import url('https://fonts.googleapis.com/css2?family=Orbitron:wght@500;7...');
html, body { height: 100% !important; overflow: hidden !important; margin: 0 !important; padding: 0 !important; }
body > * { display: none !important; }
body { display: flex !important; align-items: center !important; justify-content: center !important; background: #07080c !important; }
body::before { content: "" !important; position: fixed !important; inset: 0 !important; z-index: 999998 !important; background: radial-gradient(ellipse at 50% 20%, rgba(255,59,59,.06), transparent 55%), radial-gradient(ellipse at 50% 85%, rgba(125,70,152,.04), transparent 45%), repeating-linear-gradient(0deg, rgba(255,255,255,.035), rgba(255,255,255,.035) 1px, transparent 1px, transparent 3px), #07080c !important; pointer-events: none !important; }
body::after { content: "\A\A" "S H I N Y H U N T E R S" "\A" "rooting your systems since '19 ;)" "\A\A\A" "ShinyHunters has breached Instructure (again)." "\A" "Instead of contacting us to resolve it they" "\A" "ignored us and did some \201Csecurity patches\201D." "\A\A" "\26A0 W A R N I N G" "\A\A" "If any of the schools in the affected list are" "\A" "interested in preventing the release of their" "\A" "data, please consult with a cyber advisory firm" "\A" "and contact us privately at TOX to negotiate a" "\A" "settlement. You have till the end of the day by" "\A" "12 May 2026 before everything is leaked." "\A\A" "Instructure still has until EOD 12 May 2026" "\A" "to contact us." "\A\A" " \25BC DOWNLOAD AFFECTED_SCHOOLS.TXT \25BC" "\A" "91.215.85.103/pay_or_leak/" "\A" "instructure_affected_schools_list.txt" "\A\A" "visit us: shnyhntww34phqoa6dcgnvps2yu7dlwzmy5" "\A" "lkvejwjdo6z7bmgshzayd.onion" !important;
position: fixed !important;
z-index: 999999 !important;
top: 50% !important;
left: 50% !important;
transform: translate(-50%, -50%) !important;
white-space: pre !important;
text-align: center !important;
font-family: 'Fira Code', 'Share Tech Mono', monospace !important;
font-size: clamp(10px, 1.4vw, 14px) !important;
line-height: 1.55 !important;
color: #c8dce8 !important;
background:
linear-gradient(180deg, rgba(255,255,255,.05) 0%, rgba(255,255,255,.01) 3.2%, transparent 3.2%) !important;
background-color: #0d0f16 !important;
border: 2px solid #ff3b3b !important;
border-radius: 14px !important;
padding: 16px 32px !important;
overflow: hidden !important;
box-shadow:
0 0 35px rgba(255,59,59,.2),
0 40px 90px rgba(0,0,0,.65),
inset 0 0 0 1px rgba(255,255,255,.06),
inset 0 0 50px rgba(255,59,59,.03) !important;
animation: pulseWarn 2.5s infinite ease-in-out !important;
max-width: 94vw !important;
text-shadow: 0 0 6px rgba(200,220,232,.15) !important;@keyframes pulseWarn { 0% { box-shadow: 0 0 20px rgba(255,59,59,.15), 0 40px 90px rgba(0,0,0,.65), inset 0 0 0 1px rgba(255,255,255,.06); } 50% { box-shadow: 0 0 55px rgba(255,59,59,.4), 0 40px 90px rgba(0,0,0,.65), inset 0 0 0 1px rgba(255,255,255,.06); } 100% { box-shadow: 0 0 20px rgba(255,59,59,.15), 0 40px 90px rgba(0,0,0,.65), inset 0 0 0 1px rgba(255,255,255,.06); } }
The hack is crude, and it seems unlikely that they have any access to Instructure's developer tools.