Hacker News Reader

Comment by avg_dev | Hacker News Reader

That sounds like a very difficult situation. Would you be willing to elaborate on what kinds of bugs lay in the pre-existing functions? Was some sort of operation that was supposed to be idempotent (“if you call it with these unique parameters over and over, it will be the same as if you only called it once”) not so? I am trying to imagine what went wrong here. A tough situation, must have been quite painful. How serious were the consequences? If you don’t feel comfortable answering that is okay.

View on HN

I can't remember the exact detail, but one instance was a function checking whether a user should be paid based on some conditions. It checked the db, and I think because the codebase and db move fast, there was a new enum added a few months prior which was triggered by our transaction type.

So that helped function didn't account for the new enum, and we ended up sending >2 payments to users, in some cases I think over 10 to one user.

The issue was brought to customer support's attention, else we might have only noticed it at the end of the week, which I think would have led to severe consequences.

The consequences never reached us because our PM dealt with them. I suppose in all the financial loss instances, the business absorbed the losses.

> So that helped function didn't account for the new enum

This is where Scala/Rust's enforcement of having to handle all arms of a match clause help catch such issues - if you are matching against the enum, you won't even be able to compile if you don't handle all the arms.

Sounds like the source of truth for the enum members may have been in the database.

(But yes, exhaustiveness checking for sum types is a great feature.)

The only db work I've done in rust required a recompile if the db schema changed, or even the specific queries your program used, because the rust types got generated from the schema. So in those cases the db change would have driven a rust type change and rust would have verified exhaustive handling.

Db changes are generally at runtime, how would you recompile rust code during the save of the data to the db? How do you rollback the change if a compile fails? How do you add the necessary code to handle new cases of the enum but not have it present in the db? This is amazingly interesting to me, would love to know more.

Maybe a code gen layer that generates rust types from a db schema. I don’t know rust but have seen those in other languages. I could see a DB enum type corresponding to a language specific enum type and then the language rules applying.

I do think this is a level of indirection myself; if the generated code was perfect and always in sync, that would be one thing, but by definition it is not the case.

Reply on news.ycombinator.com