Amazon Has to Disclose How Its Algorithms Judge Workers Per a New California Law
interestingengineering.com
interestingengineering.com
But basically... let's assume tasks come to the warehouse on some sort of random poisson distribution, each with a value, and a time they must be completed by. Everything must be completed.
If these parameters are highly differentiated, a really smart system might be able to prioritize tasks such that high value things get done quicker than lower value things. But this isn't something that the worker should be figuring out.
If the parameters aren't highly differentiated, all you really want to do is move stuff through the queue faster. There's just one master metric in the "perfect" system which is, "how long did it take you to move this item through the queue vs. the expected time to move it through the queue". But to be a FAIR system, you need to be able to parse out what portion of the time to move distribution is based on random chance vs. individual agency. Delays are likely to be long skewed right. That is, most items will pass through with minimal delays but some items will have big logistical problems that completely slow down the process to no fault of the worker.
Measuring this is likely to be incredibly difficult and mostly picking up on noise. It could likely catch people who are truly slacking off. But... so could a human manager with basic observational skills.
Yes, it's true that they're not paid piecemeal per box so working "faster" seems pointless but some workers still hustle because they want to keep their jobs.
A worker having a defiant attitude of "I don't get paid to work any faster" is fine -- up until the point they are replaced by another worker who is faster -- which means they're unemployed and make $0.
> It would get a clear 'no' from me
Seems kind of entitled to assume this view generalizes. Sure, I too have a cushy job and would prefer more time / less stress instead of more money. But that's a privileged place to be. It's a view derived from circumstances, not wisdom, even if a lack of wisdom may permit you to fail to realize this.
All too true but I think sharing is caring.
I am aware of what a worker needs to achieve to be competitive in a first world nation. This has nothing to do with that, on the contrary, it just worsens working conditions for no discernible reason and surveillance software is unproductive baggage, but most importantly you imply that this productivity tuning would exempt the wage of workers being subject to market forces, which is quite a huge assumption and unproven I would say.
The success of Amazon certainly didn't rely on that.
Oh, we don't already? There's a graph on my companies VCS that displays my own velocity, along with a subgraph that differentiates reviews, code writing, etc...
Velocity is literally the speed at which we move items through the queue.
We were the guinea pigs, they just took the illusion of choice (priority) away from Amazonians.
Like adding lines of useless code when measured by the LOC you will surely start to add useless items to the queue if you have any control over that?
Delays are actually accounted for in these systems - generally in multiple ways:
* In the case for instance that there is an out of stock, or you need to go to another picking location, the system re-calculates the required distance/time considering the out of stock and this is accounted for in your score.
* If there is, for instance, a mechanical issue or something stopping you from being productive, you move onto an 'indirect task' where your time is no longer tracked as productive time.
> It could likely catch people who are truly slacking off. But... so could a human manager with basic observational skills.
We are talking about observing staff in a 1 million sqft warehouse across 5 floors of mezzanine, in a maze of aisles, and also the manager and associates shift patterns could be different - it's just impossible. I was a department manager (not for Amazon) and it's just impossible to eyeball picker performance in any sort of fair way, you have to look at data.
Additionally, dependent on the laws in your region, disciplining/firing someone because 'a manager observed them being slow' likely isn't good enough if it comes to a tribunal.
I've got direct experience with how these performance management systems are configured and work, so if anyone has any questions let me know!
There have been reports from workers to the contrary.
I don't mean to refute your arguments. I'm just suggesting workers don't (or haven't) believe this to be the case.
The misconception comes from being given a target-time which is based on a calculation assuming the task can be completed successfully with no delays, however afterwards the target-times are recalculated to account for the additional movement.
A common assumption is that performance is (Time it took / Initial target time) - especially because the users see the target time - however it is really (Time it took / Recalculated target time accounting for additional movements).
I'm not aware of Amazon's exact implementation - but all performance management suites I'm aware of will re-calculate afterwards based on 'actual' task rather than 'forecast' task.
So maybe the system has gives you "bonus time" if an item is out of stock so you can fetch it at a different shelf... except for a special item that needs to be stored in special conditions. So if you need to fetch the item and it's out of stock, you need to wait five minutes for a colleague to fetch more, and the system classifies it as wasted time.
From what I've gleaned in articles and the like, it's the main problem with these systems: if they encounter a problem that was accounted for, things go smoothly. If they encounter a new problem, they default to assuming the employee / contractor is at fault.
For instance to take your example at this scale the system will always tell you to go do something rather than wait at the pickface for a replenishment (there are lots of options for how this situation can be handled, but the job of the WMS is to capture and handle these exceptions). Less refined warehouse processes might wait, but at the level of system sophistication that these tracking systems need this is likely already engineered out.
IMO in terms of being a fair measure of individual effort/performance - these systems are about as good as you can get in a warehouse and are pretty sophisticated and good at calculating the exceptions. If people disagree with using them as a management tool that’s fair enough, but IMO the calculation is much more thorough and fair than in other places where people get performance managed.
I dunno. Maybe the Amazon's warehouse system is more polished, or maybe the complaints from the drivers were anecdotes not representative of the industry. I haven't really seen recent articles/studies about the working conditions in Amazon facilities, so I don't know how up to date the old complaints are.
You can have a really poor performance management calculation and great working conditions and visa-versa.
Maybe this is different? Hard to say. But if I were to guess, I would imagine the differences in individual performance that are truly due to employee agency are not that high.
> We are talking about observing staff in a 1 million sqft warehouse across 5 floors of mezzanine, in a maze of aisles, and also the manager and associates shift patterns could be different - it's just impossible. I was a department manager (not for Amazon) and it's just impossible to eyeball picker performance in any sort of fair way, you have to look at data.
And I tend to imagine (perhaps uninformed) that these systems don't catch any small details. Presuming these people do in fact have managers, the big picture assessment of whether they're totally slacking off vs. working normally is likely to be apparent without data. The amount of actual agency people have in a task like this, I'd imagine, is heavily skewed left. That is, very few people are noticeably better than the median worker. But many people just don't work at all. Absenteeism is a problem in a lot of jobs like these, including pre-COVID. A large part of it is just getting people to show up.
The systems basically direct user work, so you have a pretty good level of detail, particularly as users will also often scan to tell the system when they are doing other stuff for instance breaks:
* Go to location X (User scans barcode) * Take 2 of item and confirm (User scans barcode on each of the two items) * Now go to location Y (User scans barcode) * Take 1 of item and confirm (User presses not-found) * Now go to location Z (User scans lunch break barcode) * Please scan lunch break finished when complete (User scans lunch break complete) * Now go to location Z (User scans location) * Take 1 of item and confirm (and so on)
Each prompt and response is recorded as a transaction in the WMS, and then the total time estimate is calculated afterwards based on those transactions.
> That is, very few people are noticeably better than the median worker. But many people just don't work at all
Absenteeism is a huge problem - you are absolutely right! Yeah, it can be easy to hide in a giant DC. Regardless of how good of a manager you are and how much you walk around, if an associate decided to spend their full shift hidden behind some pallets reading a book, it would be really hard to notice their absence without seeing that they picked nothing on the system (particularly when you have a team of 30!).
Good note on the scoping of the tasks - it depends totally on the implementation! Ours was pretty good, but the difficulty was keeping it up to date as you have to change the walk routes and distances any time you move physical equipment on site, or change site rules/physical processes.
I've never gotten to work with an activity level system, just overall daily or larger outcomes.
A lot of the conversations were bonus related too - “Hey, FYI you are scraping the bonus threshold so if you really smash it across the next 3 days you can pocket an extra $200 or so”.
It’s a shame that CDs and DVDs went out of style and they went out of business. It was a fun place to work.
Instead of looking at a per package performance, you simply look at the average over time.
If the average target is move 100 packages per hour, you are looking at the average of 16,000 samples for one month of work.
Look at standardized testing in the USA (SAT, ACT) - there is tons of coaching and preparation specific to the test that isn't training anyone on the material. Instead, we know how much time to spend per question, how to quickly eliminate possible answers on a multiple choice to improve odds when guessing, what patterns in the language a question exist, etc.
I think this is so so similar to customer support and that issue big corporations (and especially us, as their customers, face) which is that quality evaluation is not something that can be boiled down to quantitative measurement. Sure, it can weed out many common problems, but it's just not worth it.
Judging workers solely based on some rubric will create a class of people who min-max the system and will lower the overall quality of the service provided, while disenfranchising the best employees.
On the other hand, maybe this will help move these opaque and (almost certainly) biased processes towards some better baseline.
But this misses the point entirely. Because these metrics and criteria are not transparent, the people affected by them have no recourse. There is no specific thing they can point to that is unreasonable about the standards by which they are judged, because there is nothing specific accessible to them at all. With no transparency, the "algorithm" could be some stupid nonsense, or something highly illegal, or something boring and human like managers playing favorites by up/downvoting employees on a secret app. The intention of this regulation is to shine a light on these practices, so that both the people affected and regulators can see how stupid and unethical they probably are - instead of speculating - and do specific things about it
There are a lot of things in society where you can gain an edge and/or profit by doing crime: either straight up stealing, or extorting/assault/bullying, or subtler things like playing into long-standing biases for your own benefit through hiring (say, for a sales force?) only the demographic that people most expect to see in positions of power, and refusing anyone who wouldn't fit that image on the behalf of your most prejudiced customers. They're all variations on crimes to the extent that they're coded into current law.
If you plausibly build these crimes into the algorithms in any way, you benefit from committing them while being able to pretend you're not doing it. Stands to reason that the law-makers are interested.
In the absence of regulation it's a race to see who can commit the most profitable crimes the most enthusiastically, with much hiring competition for whoever is the best at doing it. With regulation, it's more about competition for whoever is the best at HIDING it :)
If it wasn't somewhat (or very) adversarial by nature, we wouldn't need to have regulators in the first place… they're a natural part of a system that inherently tries to grow by any means necessary and can't regulate itself in any meaningful sense. Sounds like things are proceeding as they have to proceed.
I would take my chances with someone playing favorites compared to a shithead developing KPI analysis for workspeed metrics. And it doesn't even come close.
It absolutely can. For the majority of workers inside an Amazon FC, the job has no decision making in it. Decisions take time, take thinking, and can be made wrong. All of the apps driving work do not offer branches, choices. They direct the associate: do this thing.
The entire job is "do the thing as many times as you can per hour, without doing it wrong". It's almost always the same thing, over and over. The metric they are judged on is their overall rate of doing that job. And mistakes from doing it wrong are found, identified, and tracked back to who did it.
> Judging workers solely based on some rubric will create a class of people who min-max the system and will lower the overall quality of the service provided, while disenfranchising the best employees.
The cases where people have 'gamed' this system usually make the news: people peeing in bottles rather than walking to the bathroom, so that they can keep their rates high. Apart from that kind of insanity, there aren't really any ways to improve the metric except to go faster. And either way. it's a net win for Amazon.
It can be, but the question then is should it be? And is that a safe and humane relationship between employers and employees?
If Amazon could get away with it, they would probably do worse; but the role of regulation in this regard is to represent the employees. In the absence of a strong union, the employees rely on the government to push back against the unreasonable demands of employers on their employees. The entire use of an algo is a larger move in sidestepping previous regulations.
I wonder what's the value of all that coaching. How many extra points can someone score thanks to coaching?
Also, the SAT is extremely well documented, especially when test prep books are available for almost free.
As soon it comes in contact with a human, you don't get to wipe your hands of the outcome if you dislike it. You have to own the successes and the failures.
When you disclose the numbers by which you "track" people, people will find ways to maximize those numbers without necessarily maximizing output.
But then, is there a good alternative? Big tech asked for this by being extremely non-transparent and mistreating people.
While I agree with the intent, I don't think simple solutions like these are necessarily the best ones to solve the problem.
I work in logistics consulting (including WMS config) and used to work in a company that had very in depth performance management calculations - and it did account for things like the size of items to be stowed. This is standard functionality now in tier-1 warehouse management systems.
Basically there are two parts - goal setting where the system works out how long a task should take (based on distance, acceleration for the equipment type being used, product characteristics, level of picker experience, fatigue etc) and then there is a performance calculation based on a re-calculated target time based on what actually happened and how long it actually took (usually a %). You have to recalculate the target to account for the fact that, for instance, the object could have been stowed at a different location than the system estimated if put away is system directed.
While an individual worker might get a %, a more typical department kpi will be items stowed per hour for instance. This is because these calculated KPIs are terrible for management of overall site performance because they only measure how ‘hard’ you are working, not how smart you are working (eg you might be generating lots of unnecessary travel but still scoring high in the %).
And some sort of half-version of this has been going for decades. In my first warehouse job we used to have a legacy warehouse system but had worked out performance targets which varied by the make-up of work you got (e.g. a combination of total weight picked, types of order you had picked, orderlines picked and units picked).
IMO the only difference is that these new algorithms are generally 'fairer' (the old issue was that rogue department managers could to give preferential treatment to some employees, or put them on the 'best' jobs) but they are also more complex and opaque, which leads to employee mistrust.
Where I worked a union actually audited the calculation so it is possible to do this in a somewhat 'open' way, although the problem with opening the algorithm up to everyone is that it is complex and can be easily misunderstood/misinterpreted, particularly if you do not have training in how it all works (which then also requires WMS training!). There were always misconceptions with how the calculation works, because all users saw was a rolled-up number and it's quite a complex thing to understand (even department managers would describe how it works wrongly, and the union would also describe it wrongly, which would confuse people further which obviously isn't a good place to be).
Sure, there could be a case where the chemistry between line worker and manager doesn't work out. I think these cases are rare and people must find alternative arrangements. We must be productive since we produce in a first world nation on an international market and line workers are low wage workers, no illusions about that.
We can calculate needed material and expected output. We plan with that. If that is reached, we have succeeded. People help each other when things go wrong. Far better than a mediocre developer fantasizing about KPI and making stop watches for people taking a shit.
Obviously creating a condition where workers are competitors might be a hindrance here. No wonder your factory sucks on productivity, everyone is a adversary and if others are slower I will look better on performance sheets. Genius management approach.
I'm not going to debate if it is / isn't a good thing, because that's individual opinion, but in a distribution setting these systems absolutely help improve performance/output when put into a DC operating at Amazon-scale (at smaller warehouses it's pointless).
> We have that too in our production, although it isn't down to the individual
Fair, I mean specifically individual user performance tracking has been happening for at least 30 years - because it used to be in a legacy system I used to use. And the time & motion based performance management a-la-Amazon has been around at least 15.
And yes, we always tried to take the carts full of SD cards and memory sticks, rather than cauldrons and sinks!
Are you serious? Defining goals and think about consequences for your workers when there are acute problems. Otherwise you just put pressure on your workers for no reason at all.
Are the algorithms necessarily explainable? It seems to me that over time, the all-seeing eye will have a logic that no human really groks.
How is this different from surveillance and grading in any other company or profession? As the cost of spying on your employees drops to zero, I would expect this everywhere. Physical injuries are only one aspect.
Wondering how long before something like Amazon is basically just a DAO?
After, of course, killing everyone at the firm.
This implies corporations should "put people over profit", which is welfare.
You need to be specific. Injuries? Stress? Responsibility? Overtime? General drudgery? People are paid a salary to do something they wouldn't do otherwise. There is always a downside. Laws exist to limit certain downsides, eg undue physical stress, sexual exploitation or too high a risk of injuries. If that was the actual policy goal, you would not say “We cannot allow corporations to put profit over people”, you would be mentioning a specific downside. Doing otherwise would be dishonest sloganeering.
You seem to think the labor market is some sort of textbook-perfect market, where employees have roughly equal power and information as employer. That simply isn't true. Thus we need worker protections. And every time an employer finds a new way to profit off bad behavior, we need to add more protections.
Fired for no reason=perfectly fine.
Severely injured on the job=can't sue the company in most cases, and have to rely on suppar Workman's Comp.
(I probally left out a lot of Corporate Welfare. It's early. here, and I need to go back to sleep.)