How I Hacked My University's Registration System with Python and Twilio
twilio.com
twilio.com
The immigration website of Saskatchewan province opens up randomly to apply for immigration. I missed it many times because there is no indication other than the "Apply" button being enabled and a small text in their homepage which says "Applications are now 'open' ". They will close the application intake when they have reached X number of applicants. So timing is very important.
So I hacked up a script which diffs their home page every 10 minutes for "open" regex. When there is an "open" keyword in the diff, the python script calls twilio API to make a phone call to me along with an SMS.
So this script was running in AWS for many weeks and one day I got the call. Logged in to Saskatchewan's immigration homepage and applied. Now I am in Canada as a permanent resident. Thanks to Twilio.
edit: add H1B to make clear which type of immigration is broken IMO.
I would argue that the fact that you find SMS/phone calls a more urgent alert is a fault with your phone/communication setup - and twilio is an interesting hack around that.
But there really should be an easy way to make just as much ruckus from a simple email, based on topic/sender filtering... (hm, maybe there's a nice side project for an app in there...).
I found a house for myself and a friend in a similar way, reverse engineer mobile app from MLS, scrape all listings once a day to get new stuff as it hits the market and you can now be far more selective than the crappy search on the MLS site allows.
https://github.com/WilliamFCipriano/FreeSMS/blob/master/data...
https://martinfitzpatrick.name/list-of-email-to-sms-gateways...
Doesn't the asynchronous nature of an IM make it less urgent than a synchronous phone call?
The funny thing is, you go on to acknowledge that a phone call is psychologically more urgent than an email, and ponder how email might be extended to cover that use case...or maybe, the person was not at fault and was actually using the right tool for the job!
Until the FCC develops redevelops any sort of interest in dealing with scammers, my phone is basically useless for most of the world to speak to me.
If you're trying to reach me and are not on my list of approved callers, you can email me, or decide you didn't actually want to talk to me. (No, I don't use FB Messenger or related spyware, either.)
On the other hand, I apparently pay much more attention to email than you - I generally respond within about five minutes, assuming there's some reason.
IRC exists for a reason.
Except when you are asleep.
And when it comes to immigration, you are happy to be woken up in the middle of the night by a phone call when it means the difference between failure and success.
Given the synchronous nature of a phone call, it seems like the perfect medium for this kind of alert. Similarly, SMS piggybacks off of the same network and therefore inherits some of the same properties.
I believe OP did this by running a cron or so on an always-on server.
Canada does seem more attractive than US. However, there are definitely a lot more jobs in technology, and a lot more software engineers here in the US.
After years of hard work I successfully got in via Provincial Nomination.
I was surprised to see the number of applicants increase without the button having ever become available (no SMS received and I had logging to confirm). It was fun, nonetheless.
It's made me want to make something like this to diff arbitrary things and send notifications to people who're watching them, but I didn't sit down to do it, and sitting down to do it is all that matters.
$seats="$(curl $URL | sed \"140qd\" | sed -nE \"s/<TD CLASS=\\\"$CLASS\\\">|<\\/TD>//gp\")"
if [ $seats -gt 0 ]; then
echo "Go register for class $URL" | msmtp -a "default" $EMAIL
fiObviously everywhere is different.
I think you could also opt out of "CS102" if you desired. It was on the student to determine how much they knew. The school just understood that a number of CS students had been doing this for some time on their own, and had mastered some of the basics.
I skipped the "CS101" course, which was mostly intro to programming, control flow, etc. I decided against skipping the "CS102" course; the first half wasn't that exciting (I knew how linked lists, vectors, hash tables worked) but trees got me (and some of the more exotic stuff at the end of the course), so I'm glad I took it.
This past semester, the active users was between 1/3-1/2 of the undergrad students in the school. As you can imagine, popular classes had dozens of people "subscribed" to receive notifications when it opened, so it became a race to sign up once the push went out. On the plus side, this gave us a treasure trove of data on the most popular courses, and we've been in communication with the school to see if they would be interested in this data.
A class was sniped from me one time while I was still on the confirmation screen. So unlike OP I fully automated the process.
EDIT: A notification & response is good for solving a captcha if they have one, if you don't want to outsource that to mechanical turk (or trust their timing / accuracy).
For many universities though, it won't be this simple. Many hide the open seats behind a login, which means you need to be a student there to see them. Many also use student information systems that are a real pain to scrape like this. Once you get to the point where interacting with the student information system is done via AJAX requests mediated by a mess of JavaScript, it starts to push the limits of what you can do with this technique.
When it gets to this point you either have to carefully dissect everything on the Network tab of your developer tools, or use a real browser. I've used PhantomJS (basically a headless WebKit) and Selenium WebDriver with great success on sites that are not amenable to scraping. The neat thing is that you really only need to use that for the interaction to get to where the information is. Once you've navigated there you can just have it dump the rendered page as HTML and parse it using the same techniques shown here.
0: We used https://www.charlesproxy.com/
It's really too bad that universities don't foster creativity like this.
Though at that point you'd have a lot more luck with selenium or some other web driver.
You have to love the education system...
So in general, do be sure to not violate a site's terms, but I don't see what that point has to do with your final comment...
If you just want to automate registration to ensure you get a seat in a course you should be looking into your school's network topology to minimize latency.
echo 'Text message text' | mail -s 'Subject will display in braces in the message' -a 'From: Myserver@Mydomain.com' 'number@vtext.com'
This ended up working faster than Twilio for me. (Less than 10 seconds from enter key to text on phone).
College administrators who are considering Oracle for your registration needs, please consider anyone else instead, for everyone's sake.
Using this example, couldn't someone (University IT) write an SQL script to dump out the current seat count/capacity for every "current" class and throw it on a web server somewhere? Assuming that the underlying database schema isn't completely insane, that would probably be about a half-day of effort. The student writing the scraper could then point it at the output of this script.
With a half-day of effort, you've reduced the load on the application, you've provided a service that the students probably want, and best of all you've got someone else to do most of the hard work.
Students rank their course preferences in an order and submit it. For each course, free seats are first allocated to the students who ranked it as their first choice, then what is left to the students who ranked it as a second choice, and so on until the round n and there are less seats available than people who ranked the course as their nth choice: the remaining seats are allocated by a lot to them. (For a very popular course, they would have to start with the random selection right away on the round n=1).
However, instead of enrolling the lucky students right away, everyone gets notified about which classes they have been tentatively admitted, and they must confirm their attendance before they can take their seat. If they don't confirm in a timely manner, they lose the opportunity and the free seats are propagated to students left in the waiting list until someone accepts (again using random lot).
The one problem I can think of: the process (especially the propagation of rejected offers in the waiting list) may be too long-winded to be practical. So maybe skip the confirmation part: there's a set maximum number of courses you can take, M, and slots are filled until you're enrolled to maximum of M courses.
But in any case, if the overbooking is enough of a problem so that you are looking at methods like these, the real problem is that have too many students compared to the teaching resources. All other attempts can only mitigate the symptoms of the root cause.
So maybe staff should choose limit the amount of students eligible to take some very popular courses by an entrance exam, or imposing certain mandatory minimal GPA requirement in the prerequisite courses. This option should be especially considered if the attrition rate is high (there's a problem of students taking much-coveted seats in the class but who fail to show up or otherwise don't put in enough effort to pass). Usually that would have been done on the admission level already, though.
"All students are given an equal amount of points per semester to bid for modules... The allocation of modules is based on the lowest successful bid points against the last available quota for the module at the end of each bidding round. If supply (module quota) exceeds demand (number of bidders) for a module for any bidding round, the lowest successful bid will be 1 bid point. If there is a tie in the lowest successful bid points, the outcome will be based on first-come-first-served. Unsuccessful bidders will be fully refunded. Any unused bid points after each round will be carried over to the next bidding round or to the next semester at the end of the registration exercise."
EDIT: to those asking for the source, it was just a bash script that I have probably lost. The GitHub linked in this subthread looks a lot better than what I had.
This reminds me of the story of beartracks, and how it was shut down by the University of Alberta to make everyone register by phone .. and eventually was brought back by the university.
Tried to find a link I had read about the story, but can't seem to find it.
I used successfully with Eventbrite for hackthons sign up.
I remember getting SMS's in the middle of the night alerting me that a seat opened up!
I'm very surprised registration systems aren't (generally) some kind of phased auction. There's got be some research, experiments somewhere exploring different strategies.
I worked on student facing software in higher ed. Admins were obsessed with fairness. Our registration systems had all sorts of arcane rules. Which were hard to explain, validate, troubleshoot. And probably neither fair or effective.
Some kind of auction, where blocks of seats were made available to different populations, progressively over time, would greatly simplify the implementation and understanding.
For (a greatly simplified) example, GRADUATION REQUIREMENT ABC has 200 seats. 50 seats are released every hour. First hour, seniors have first shot. Second hour, opens up for juniors and program enrollees. Third hour, sophomores. Fourth hour, freshmen.
Of course, there are other factors, like multiple sections (same class offered at different times).
--
We also talked about predicting demand (capacity planning). A novel solution there might moot the entire registration stampede. Perhaps a "buying club" type solution, where students state which classes they'll "buy" and roughly when. Then registrars form up sections to satisify the largest number of students. This could reduce the twin connundrums of waitlists and over capacity.
Anywho. It's an interesting problem, ripe for innovative optimization, matchmaking, auction/market solutions.
http://www.thecrimson.com/admissions/article/2013/9/10/cours...
For airfare, I think there's already a few players in the "this flight got cheaper" space (Kayak and Hopper come to mind), but they don't have data on some airlines (because of the airlines' ToS IIRC).
It sounds like exactly the sort of service they should be providing.
Some other schools in our system are really reactionary, though, and consider any automation a ToS violation and will freak out.
e: And if you know the URL pattern / platform of your Uni's registration system, there's probably already a couple of examples on github of a registration bot.