Better bus predictions (a lot better)
medium.com
medium.com
It's working pretty well, the key thing is the point 4 in the article - the timetables show "in X min" if they know how long it will take for the bus to get here, or just "at XX:XX" if they don't know for some reason (transmitter or bus broke, whatever) so they just show the scheduled arrival.
So you can be pretty sure that if it says it's coming in 4 minutes you won't be waiting 30 minutes.
Clearly marking how sure you are of your prediction is IMHO more important than making very good predictions. I don't care if I have to wait 3 minutes or 6 minutes, I do care if I skip another bus waiting for a more convenient one in 3 minutes that never arrives.
I know I have to leave my condo when Transit says 3 minutes and it has the "this bus is reporting" logo.
There are far too many large, bloated consultancies that specialize not in delivering quality products and services, but rather in "surviving" the government procurement process.
Props as well to the Swiftly team - I had a change to take a peak at some of the APIs they expose to their customers and it's quite valuable. In particular, they roll up stop-pair segment performance on routes by time of day, which allows someone to query for bus performance by discrete route-schedule-segments.
Gathering this type of data is quite labor intensive and a significant technical lift (I was once part of a project doing this with GTFS-RT data from the NYC MTA). This type of information, and the broader ecosystem of performance related API services they provide to their users (based off the limited amount I have seen), can enable operators to extract highly articulated performance statistics about their fleet, on their own.
We used a couple of metrics to assess the quality of the predictions. The first one is sMAPE (Scaled Mean Absolute Precision Error), which tells us quickly where and how our predictions fail. We plot this with the precision on the y-axis and the minutes till arrival on x. Also plotted is bias, which is important, especially since you want to be slightly biased towards being too early. Similar axes.
Other metrics we used are MAE (Mean Absolute Error), RSE (Root Square Error) and 2D kernel density plots.
In the end, for a contract, I would take into account the density of use of these predictions. It's nice that you can predict perfectly during the middle of the night, but if that line is not used, it is next to useless. So something like sMAPE * passengers or sth like that.
Also, even though I cannot relay this one-to-one to the passengers, a confidence interval on predictions is gold.
The article suggests people will use these predictions to spend more time at home before leaving their bus stop. If that is the case, what happens if the prediction says their bus will be, say, 10 minutes later, but then that bus catches some lucky breaks with lights, heavy traffic unexpectedly clears up, etc., and the bus makes up 5 minutes of that?
People who relied on the prediction could miss their bus.
Is there a mechanism to address this?
The only one that I can think of offhand that would cover almost all cases would be if the predicted arrival times are also sent to the buses, and if the buses arrive ahead of the prediction they wait until the predicted time before leaving.
You wouldn't want to do that for all predictions, though. Maybe only predictions made within 10 minutes of the original scheduled arrival time are binding, or maybe only predictions made when the bus is within three stops of the stop in questions, or something like that.
Timing when you leave the house/office to catch the bus matters a lot more when weather is miserable and/or when buses arrive infrequently than when they arrive frequently and the weather is nice.
The bus I take most often is scheduled to run every 3 minutes on average during my morning commute. That means that even with severe bunching I rarely have to wait more than 10 minutes. Unless it's under 10F outside or is raining buckets, I just leave the house when I'm ready to leave the house, and use the bus predictions mostly as a way to manage frustration levels.
It takes me about 90 seconds to get from home to the bus stop if I make the light, BUT there's a really long traffic light in between. If the bus is listed as 4 minutes away, I have a 50% chance of catching it (depending on whether it makes the previous light). Less than that and there's a 90% chance I'll miss it.
Also, the arrival predictions seem to be based on distance and stop count rather than typical local traffic conditions. It also doesn't take into account times when the bus is empty and skips most stops or is full and not only stops everywhere but boarding and debarking take forever. Which means that while it's eerily accurate for certain stops and times of day, at others I can count on the bus to arrive in half the time predicted and at yet others in twice the time. Unless there's a crash or construction along the route, in which case all bets are off.
During my evening commute, the buses are scheduled much further apart (and are a much longer walk from the office). A 10 or 15 min scheduled headway means leaving whenever you're ready works well if the buses are reasonably spaced (regardless of whether they're actually on schedule). But bunching sometimes causes a 20-25 minute wait (and occasionally more). What I find the bus arrival predictions best for is to be informed on those occasions when the bus is half an hour away and your best bet is to say "screw it" and walk or take a cab.
On another route I frequent, I get on at the first stop on the line. The next bus is always parked around the corner (unless it's super late completing the incoming leg of the route). The app I use lists it as due in 1 or 0 minutes. The actual arrival time will in fact be the scheduled time.
What it comes down to is mostly about knowing your route and adjusting your perception of the predictions to take that into account.
Why are we "predicting" at all?
Why doesn't every bus have a tracker that tells you exactly where it is at all times?
This is 2018. GPS with refinement isn't rocket science anymore.
Tell me where the bus is and I'll do my own prediction thanks.
The problem is that they should have spent 0% of their time on prediction and 100% of their time on actual tracking.
Once you have tracking, prediction pretty much comes along for the ride.
Same with the train - I use a website that shows the exact locations of all the MBTA trains, and I am able to accurately predict how long the train will sit outside of the Alewife station (which is infuriating BTW) based on the number of trains currently at the platform. A prediction would not be able to take into account this domain knowledge and would be completely worthless.
This x100.
I do this all the time with Lyft drivers. "No, please don't listen to Google Maps for this. The route it is trying to take you on goes from totally open to totally clogged over the span of 6:45AM to 7:00AM. Trust me, the route I am having you take me on will be faster by the time we get there."
People are way more clever than you think.
I have found One Bus Away to be completely inferior, to the point that I usually don't bother to check it.
- Time. Time has the advantage of brevity ("15 min" takes up a grand total of five characters) and indisputable meaning. It's also the hardest to predict, and people get pretty upset about incorrect predictions.
- Intersection. On a signboard, intersection names can get quite long. You would also need familiarity with where exactly the intersection is located, how the bus gets from there to your stop, etc. Boston is famously non-gridded, and has no real consistent rhyme or reason to its street layout, so that would be very confusing.
- Miles/stops away. New York's BusTime uses this. This is a pretty indisputable metric, and if you have GPS or a working odometer is trivial to confirm. But this runs into the same problem as intersections; you need context. The bus is two stops away, but how long does it take to make those two stops? A mile passing through Times Square is very different from a mile through the leafy, quiet suburbs of Queens.
You can pick between meaning and certainty, but it's very hard to have both.
And that is basically what is happening early next year.
I have worked in support for a well known GPS navigation software and the GPS accuracy is terrible for most consumer devices. You have to spend a lot of money in a GPS receiver to get past this accuracy issue. What Lothian buses (the company I have mentioned above) have done is using something lower tech to replace the need for such an expansive device. And overall it does work surprisingly well even in heavy traffic occurrence (road work, rush hours, etc ...).
1: https://www.sfchronicle.com/bayarea/article/Muni-looks-to-re...
There are pages on the main site about this project, for example: https://www.mbta.com/projects/better-bus-project
I think the Medium blog was set up for posts intended for a more tech-savvy audience than the broader audience that visits MBTA.com to check schedules, plan trips, view alerts, etc. Maybe at some point it will be merged in.
The website was relaunched ~year ago as an Elixir / Phoenix application and there are still quite a few features and content being added.
Anyway, I really would like to see a histogram of the predicted distribution of arrival times. Sometimes I 1-sigma need to make the bus and sometimes I 3-sigma need to make the bus...
There is a post on the first of every month called "Who is Hiring." You might do better planning to participate in that.
Most recent: