Hacker News Reader

Comment by ffggvv | Hacker News Reader

that’s not a ddos. it’s just called not being able to handle a spike in traffic.

The application has a retry mechanism that will keep trying until a connection succeeds (with an exponential backoff), but it doesn't handle a case where the server is basically rejecting the messages on purpose (due to capacity issues). Suddenly all applications will start retrying the connection at once and there is no way to turn if off. So it will make the problem even much worse than it already is. They have added a server feature flag yesterday to change the maximum backoff time and they also added the handling of HTTP 508 error response.

Commits from yesterday:

Feature flag for maximum retry backoff time: https://github.com/signalapp/Signal-Android/commit/93e9dd642...

Add jitter to backoff time: https://github.com/signalapp/Signal-Android/commit/8f7fe5c3e...

Handle ServerRejectedException (HTTP 508): https://github.com/signalapp/Signal-Android/commit/c95f0fce6...

Feature flag for automatic session reset: https://github.com/signalapp/Signal-Android/commit/a3c7e7e55...

Can somebody clarify if I'm correct in believing that feature flags are client features that can be activated server-side (as opposed to an app update)?

These can be set by the server, you don't need to update the app.

"A location for flags that can be set locally and remotely. These flags can guard features that are not yet ready to be activated."

Here is the file with javadoc on top:

https://github.com/signalapp/Signal-Android/blob/2c1c6fab356...

That is correct. Typically used to enable or modify a feature in the client on your own time table vs. App Store approval.

Problem: whatever

Solution: TRY-CATCH

WHILE TRY CATCH

The technical term for this is “thundering herd”

I'm wondering if another way to handle that would be the server accepting the request, but not actually servicing it. Basically let it timeout so that the client doesn't retry (at least, does not retry before timeout)

USDOS - Unexpected Success Denial Of Service.

I woke up one Monday morning to crashed servers and angry phone calls once in that situation. Unfortunately it wasn't success on my part but what I had done the Friday before was so crap it couldn't handle a hundred users. But we blamed it loudly on unexpected success, quickly patched the mistake and everyone was happy. YMMV :)

Lesson learned: never ship on Friday

Which is basically what a DDoS will do with its botnet. Nthis case it was an involuntary one without a C&C.

Reply on news.ycombinator.com