Our recent server issues | Hacker News Reader

Our recent server issues

lichess.org

I expect that what they've pointed to as the cause is only part of the problem. We'll never know the full picture unless they share it.

Being upfront, my experience of the people in charge of the organization there doesn't have much goodwill left. Nothing against Thibault personally, I think he's done some great things but seems to be busy with whatever he's interested in and management isn't it, and he has some unprofessional people with access that work for the project/charity.

I had volunteered my services years ago as a System Administrator (no charge) with suggestions, but with limited modes of communication, multiple issues going stale, no response with an auto issue closed. They have issues that don't get addressed, and their process doesn't appare to be aimed to cultivate qualified volunteers or improve the bus factor of the project.

To make things worse, when I expressed disagreement with constructive feedback regarding one of their process decisions, one of the other dev's with access apparently took offense and the next day I found my lichess account had been edited by an admin without notice or notification. I could not log in (wrong password), the password and email for password recovery were changed, and trying to access the profile URI directly showed the account as banned.

It seemed this was done out of spite, and definitely without any kind of due process. Appeals by email went unanswered within the 90 day cutoff I gave them. As a result I submitted a complaint to the french charities regulatory body and moved on since the group wasn't worth wasting any more of my time. I haven't heard back so who knows if anything came of what I reported.

In my opinion, they've got more internal problems than they let on, and to me this is just spillover.

Its unfortunate because any failure like this impacts so many people, but I don't find it surprising given my limited experience of the people there.

Most anyone who has built a high-demand service has encountered issues exactly like this. Unusual activity places stress on a small piece of poorly optimized code that was never tested for it, and never considered because it’s never been an issue before. A fleet of hardware goes idle while the single lock spins.

Whatever personal issues you may have with the admins, it's extremely plausible that this issue was sufficient to cause an outage.

And yes these things happen at big tech companies and mature software products even when they have teams and testing frameworks specifically looking for them. A smaller project like this, I cut them a break.

Would you consider reading the excellent book "Working in Public: The Making and Maintenance of Open Source Software" by Nadia Eghbal?

The data behind what "Open Source" projects look like differs from the popular culture narratives and assumptions about what they should look.

I doubt there was justification for disabling your player account. and I'm sorry that happened to you.

But it sounds like your expectations about code contributions, and onboarding new volunteers may have been far from that project's reality.

Thank you for the suggestion. I had not heard about this book and based on the amazon description this looks interesting.

I've worked with a few OSS projects in the past that were much more open and transparent. My expectations were framed by the contribution documentation Lichess provided on their site and it did differ significantly from reality.

I'll definitely be picking a copy up in the near future for reading, currently I have to first finish up with the holiday gift exchange books from colleagues.

I'm not a management person but I'm currently making my way through HBR 10 Essential reads (book) which was given; I'd never have bought it myself but I've found some useful insight reading through it (unexpected).

Just would like add a +1 for Working in Public. It's a really excellent look into a number of different community dynamics, including but frankly not limited to open source projects exclusively.

What did you report them for?

At the time, I was looking for more detailed information on the charity in the hopes of finding a more direct contact than a non-responsive email box.

In most countries, operating as a charity has certain annual reporting requirements. These reports often contain a direct contact (i.e. Agent of Service type contacts).

I was able to find the charity registration info from sirene.fr by their address (there were 14 listings out of that address when I just rechecked). Siret# 83037871700029, Siren# is the first 9 digits.

There was no phone number listed so I moved onto JOAFE and found that there were no annual reports attached to that Siren #. Backing up I went to RCS, the company & trade register which is required to obtain a kbis-extract (needed to open a bank account as a company in france) and there were no listings on the Lichess or Lichess.org trade names. RCS doesn't allow a search by Siren # and that's where my search died.

The complaint was that they were operating, and had not filed the required reports.

I don't speak french, so the complaint was in English via a webform and since it was difficult to pursue the matter further at the time, I chose to move on to better things.

Edit: I've confirmed that the annual reporting is still a problem on JOAFE. Also, back when this originally happened I had noticed the budget they linked to on googledocs for what they spend on infrastructure was dated, not sure if that is still the case.

If anyone with a greater knowledge of OSINT or a command of the french language wants to look into this further there are a number of administrative inconsistencies that might be worth following up on.

Personally, I don't see how you can be registered as a french charity and accept donations/other assistance without the appropriate bank account or filing annual reports.

Though I admit, I don't have much first-hand knowledge of doing business in France. Either way, its not worth my time taking this any further.

> Eventually there was no way of keeping up with the queue

A chess congestion pile up... sounds like an event stream rook-ie mistake. :-)

Seriously congrats to Lichess for growing. It's an amazing site. Donate if you can.

I'm not sure I completely understand. They say that the only thing that was affected was the tournament because its events needed to be processed synchronously, but I remember the entire site being unavailable for people. Was that unrelated?

On a side note, huge props to Lichess, the fact that they can compete with chess.com which has so many resources behind it is very impressive. Everyone who plays chess should consider becoming a patron.

During the first crash immediately after the initial start of the tournament, the entire site did indeed go offline for a few minutes. Even address resolution failed. Then things went smoothly for about an hour after which the 2nd crash came. This one just seemed to affect the particular tournament (participants couldn't get fresh pairings) while the rest of the site was still working, as the article mentions.

Ah I see. That makes sense.

Would be interesting to know what caused the entire site to go down in the beginning. Wonder if it was just too much traffic.

Hikaru was live streaming the event so you can see the series of failures and how they affected the tournament here: https://youtu.be/YKfvNl8UoxA

You can also see agadmator's stream here: https://www.youtube.com/watch?v=w0efZdnPnHQ

And a game review with some commentary about the issue: https://www.youtube.com/watch?v=lh_NeBZhqPM

Anyone take a look into the code and see why it can't be parallelized? The bottom of the FAQ mentions that but I'd think at least certain aspects should be parallelizable or at least be prioritizable (like maybe forgoing leaderboard updates to focus on more important events?)

Good idea. Tho in the code itself (from a cursory glance) in L141 Sequencing(...) I see a bunch of nested maps - I don't know Scala but I think this may be a performance issue? Rather than hyperfocusing on parallelism or event systems etc since that stuff is comparitively hard to solve maybe refactoring this function/algo at the core of the pairing would have more bang for buck

https://github.com/ornicar/lila/blob/98691c8901cc0e7d0f338f4...

If you think about it, you can't really generate pairings in parallel because each thread would need write access to the entire pool to ensure no one is paired twice and that you have a consistent view of all the results to that point before creating a new pair. You could maybe create a lock for each participant but that might actually be slower, and would definitely be more difficult to reason about and can lead towards bugs just as serious under load.

I'm not sure exactly what the algorithm is for choosing pairs, but naively it seems like it should be possible to bucket the players in some way, at which point the pairings within each bucket can be generated in parallel, with buckets getting merged if they get too small. The tricky part is coming up with a way to bucket them that's both efficient and will lead to enough pairings, and I'm not sure what the criteria is. If it's completely random, then ideally any bucketing should work fine. If it's based on ratings or existing records, it seems like starting by grouping players within certain ranges of that stat should work (assuming the boundaries can be picked based on the distribution of the players). I don't know enough to know for sure that bucket grouping would work here, but it doesn't seem like knowing the entire set of results beforehand should be necessary (unless there's some standard algorithm for choosing pairings that's very prescriptive).

If they're doing tournament pairing the traditional way then you can't do that kind of bucketing because it's always possible that someone will "spill" down if they can't be matched in their bucket, and that would completely change the pairing in each lower bucket (if you add one more person at the top of a bucket you would zig-zag down all the pairings below them, except the new pairings have nothing in common with the previous pairings and so you have to redo things like whether they've played each other before).

The pairing is based on ratings and existing records, and happily isn't dependent on a standard prescriptive algorithm, unlike traditional Swiss tournaments. It would be fine to accept some reordering of game-finishing events.

I'm certain the process can be parallelized - it just hadn't been considered necessary before.

>During this tournament, for the first time, players were inputting events faster than Lichess could process them. The queue grew larger, causing delays in pairings.

Mostly interested in what these other events were and how they interact with pairing. Don't really know what pairing involves but seems that should go fairly fast even on a single CPU core (unless it's doing some kind of lookup or calculation but that part should be parallelizable)

For some things there is simply no way to parallelise events when things need to be processed in order, and the limitation is simply the processing capacity. In this instance I'm surprised Lichess didn't have surge capacity to automatically scale up for a big tournament like this. Sure, it might be expensive, but that's when optimisation should come in (to save costs). Having to cancel a major event like this due to poor optimisation is a deployment failure moreso than a coding failure.

It's not going to go faster if you add more workers to a queue that has to be processed one by one

It's not just a case of adding more workers, but adding more processing power and removing bottlenecks so that each event can leave the queue faster.

I doubt it's as simple as adding processing power. That only helps in a linear fashion and it sounds like it was a bigger problem. Probably was IO bound.

Also removing bottlenecks is not related to surge capacity but rather code level optimizations that they didn't realize were needed until this happened so it's easy to say that after the fact...

You're basically talking mainframes looking at vertically scaling for high throughput on something serial

You could maybe paralyze the matchmaking process by making buckets in which players of all the elos are ranked. Nobody in the top 100 will be paired with someone who is in the last 2000 anyways. So why not split it up on every 1000th player and treat them separately.

The leaderboard then only needs to modified (read can be done without race conditions) by the top 100 or whatever.

The "easy" solution is to put a cap on tournament size.

Apart from when Agadmator wants his fans to be in the same tournament as Magnus Carlsen etc., there is basically no need to hold chess tournaments with over 1000 players.

The lichess marathon tournaments usually are a lot more than 1000 players.

For example, a recent 24-hour marathon had 24,000 players. Though not all playing at once, surely there were periods of time in which several thousand were playing/pairing at once.

https://lichess.org/tournament/autumn21

It would be very interesting to hear about the technical details!

Further down in the article, there's a more technical explanation under the heading "Can you elaborate on the technical issue?"

Sounds like they need backpressure. Isn't that the usual solution to a queue growing without bound?

You mean you want to pause the games that are in progress? The events are created by people finishing their games and queuing for the next pair. You could limit total participants but only if you know the limit ahead of time. Nothing else makes sense.

Is there any requirement that all games start at the same time? If not I guess you could limit the amount of games that can be outstanding at any one time.

No, after the first game they don't start at the same time but the point of an arena is you get a new game as soon as the one you are playing finishes. The number of points you can earn is limited if you can't start another game right away so players would hate that system. It would be much better to limit total participants because at any given time nearly everyone should be playing.

I see, but in case I think this is less about finding an optimal solution and more about the general best practice of not having any unbounded queues in your application. Having to wait would have been annoying, definitely, but it would have surely been preferrable to the outage.

Load sheddding might also be a good solution

How would that work here?

"Load" is players finishing games and requiring new match pairings / rating updates for the tournament to continue. You can tell them to wait, sure. But as long as the rate of games finishing exceeds the rate at which they can process them I don't see how that would improve the situation, the tournament would be stuck anyway.

One option is to eg. limit the total number of players in the tournament up front but they explicitly said they didn't want to do that.

Depends upon of every event is strictly necessary for the proper functioning of the system, or if some are just nice-to-have.

This is where it helps to have domain knowledge before pontificating on someone else's architecture. It is an arena tournament. If you want people to join the tournament and get paired with games after finishing their current game you'll need those events. There are no "nice to haves".

> There are no 'nice to haves'

That's an assumption, there's a lot of assumptions flying around right now. The simple answer is we don't know.

Often there are a lot of design tradeoffs, bcrosby was likely questioning whether all events going into that specific queue were strictly necessary for matchmaking. We don't even know that this was actually the problem.

So far, no professional post-mortem, or root cause analysis has been released.

There are certainly ways one can setup these type of services so they dynamically scale correctly, but since the architecture they are using isn't fully disclosed its difficult to say anything with certainty and this type of guesswork is entirely unproductive.

lol are you all just repeating scaling buzzwords? back-pressure, load shedding, whats next horizontal scaling or maybe blockchain?

Web scale sharding.

Web 3 POINT OH blockchain though none of that 2.0 nonsense for me

This particular tournament was being played for bitcoin valued at almost 50k. It would be nice for everyone to have a fair shot at it without the system getting in the way.

Reply on news.ycombinator.com