Hacker News Reader

Comment by st-keller | Hacker News Reader

Why do i read such stories at all? Now i can‘t tell, if it’s the algorithm or the implementation or both and i’m left with more questions than before!

Edit: typo

View on HN

Seemingly it is implementation:

>Why did such errors occur?

>When the algorithm finds an ideal candidate for a position, it does not reset the list of remaining candidates before commencing the search to fill the next vacancy. Thus, those candidates who missed out on the first role that matched their preferences are definitively discarded from the pool of available teachers, with no possibility of employment. The algorithm classes those discarded teachers as “drop-outs”, ignoring the possibility of matching them with new vacancies.

And (possibly) also incorrect data:

>Often teachers input the wrong data on the system because of an interface that is not very transparent, complex and difficult to access. The scores can therefore already be distorted upstream, without taking into account human errors by the school offices.” Aspiring teachers often find themselves alone in facing complex procedures and error is often inevitable and sometimes irreparable.

Yes that's the explanation according to the article. But that seems sketchy. With such an algorithm the pool of potential candidates would quickly converge to 0. So I doubt the algorithm exactly does this.

> With such an algorithm the pool of potential candidates would quickly converge to 0.

From the sound of things, that may have been happening:

> Moreover, recruitment is not keeping pace with schools' need for teachers, thus leaving hundreds of classroom positions vacant which is affecting the educational progress of students all over Italy.

That "drop out" concept just seems entirely wrong. Not only because its erroneously constricting the applicant pool, but because it strongly biases the earlier roles in the queue.

For example, consider two roles and two applicants, with fit scores as below:

               Role 1    Role 2
  -----        -----     -----
  Applicant A    96%       95%
  Applicant B    95%       50%

Ignoring the "drop out" bug, under the algorithm described the system would evaluate all candidates for Role 1, determine Applicant A is the best, then move on. At that point, Applicant B is the best candidate for Role 2... even though they're not a very good one. Overall, not a great outcome (73% avg.).

You'd think the algorithm would want to maximize outcomes across all roles: the more optimal "best fit" solution would be Applicant B in Role 1 and Applicant A in Role 2 (95% avg).

(I'm assuming the reality here is that Role B isn't available at time of evaluation, so there's no way to evaluate the universe without waiting, which may be sub-optimal.)

at first glance the algorithm seems to reward compliance ("take whatever is offered") and severely penalize any teacher who insists on some placement (by refusing the first placement you are knocked out of the applicants, maybe for a long time)

It could serve as a parable you'd tell a junior engineer interested writing algorithms that get applied to people's livelihood. Beware your own hubris, demonstrate or simulate results and show them to all stakeholders like the union, do controlled rollouts (why not start with 1 region in the country?), verify inputs that could be untrustworthy. Basically all the usual lessons one learns the first time they write an algo for people and it blows up in their face embarrassingly.

>demonstrate or simulate results and show them to all stakeholders

This is a very important point. Especially for any algorithm/policy change that affects people's pay, time off, scheduling, etc., it may seem like a reasonable change to you and to management but a lot of people will have a visceral reaction anyway. Not everyone may be happy at the end but it's important they're at least part of the process.

But leetcode hiring implies we're looking for juniors to roll their own algos and call them recursively as part of the job. People hired this way could be walking time-bombs.

Define your expected optimization outcomes in plain human language, and verify them extensively in trials.

I can write some useful formal properties right here:

   - there are n openings and m qualified applicants. At the end of assignment, there should be (m-n) applicants without a role.

   - if (m>n), all roles should be filled

   - if there are n identical average candidates and m identical ideal candidates, if there are p roles where (p>m), all ideal candidates should get a role

   - every candidate not selected should be worse than every candidate selected

   - every candidate not selected should have participated in n trials where n is the number of openings

   - etc

Absolutely, assert anything and everything! Especially the obvious conditions that would never be violated when it's working.

Reply on news.ycombinator.com