The $500M Dollar Typo | Hacker News Reader

The $500M Dollar Typo

cockroachlabs.com

114 pointsmakaimc30 comments

This is interesting, but the telling is fairly vague. It cites a congressional report, which would be interesting to read, but there's no reference that helps me track down such a report.

This is curious but saying that “single typo cost the USPS 500 million dollars” is misleading since they got it all back in the end.

The redundant "Dollar" here bothers me more than it should. I can't help but read it as "The 500 million dollar dollar typo"

The post writer is a former journalist who should probably know better.

In a company a worked for many years ago I was in charge of delivering a weekly report to the board with What went wrong this week, Why it went wrong, How we fixed it, How we are making sure that it doesn't break again.

I'd really like to know which checks the USPS added to prevent for something else to cost them $500 M again. OK, there is the usual read the report check which worked on Saturday that time, but it didn't happened on time. So, maybe no automation of actually doing the payment at midnight on Friday?

This seems like the most obvious fix. Create the Payroll Job in the "WAITING FOR APPROVAL" state instead of the "APPROVED" state.

Perhaps the OG JCL cutely encoded the job state into a single bit and no one has dared touch it since the Ford Administration.

Begs a loosely related question: what VCS do you suspect the JCL is stored in? I'm going with the "incremental .BAK system", if any.

People who live paycheck-to-paycheck not getting paid at all is potentially a much bigger issue, particularly when you have laws and unions involved. The conditions for that issue are also far less stringent, you don't need an extremely unusual operator error and an unplanned federal holiday happening at the same time, all you need is a single person forgetting to click "approve".

> If a federal employee is accidentally overpaid, the government can only recoup 16.67% of the amount of the overpayment per paycheck pay period.

Very curious to hear the story of that rule

I can wager a guess. In the military, it was pretty common for finance to mess something up, especially around TDYs and moves between assignments. You might get overpaid by a few $k and only 6 months or a year later would someone notice. Of course that is a lot of money to most people, especially military folks who might make $30-50k/year and wouldn’t immediately have a few $k to give back or subsist on if a paycheck was docked.

TL;DR - it probably cost the Post Office several houndred thousand dollars in internal time to fix this, rather than $500M.

They did have to borrow A good chunk of that $500M from the US Treasury for several months, though.

If Cockroach Labs think it remarkable that someone wouldn't be fired for a single typo, what does that say about working there?

Guess it was too difficult to just adjust the pay amount (with a deduction) for the next pay period?

From the article:

The second problem is, they have to get all that money back. If a federal employee is accidentally overpaid, the government can only recoup 16.67% of the amount of the overpayment per paycheck pay period. You can’t take it all back at once. So then they had to write a special payroll job to pull all this money back. It took ‘em like four or five months to get it all, but finally it was all accounted for.

I read that part, the language of the article makes it seem like they had to actively debit everyone that was overpaid to get the money back. Underpaying them on the next pay period wouldn't require any debits to the bank accounts where the money ended up ("take it all back"). Article doesn't have really have specifics to the solution. Rather what I mean is it's not clear what "take it all back" means: is that referring to direct debits to where the money ended up in, or is that referring to adjusting what goes out in the next pay periods?

I don't think even the USPS can order your bank to transfer money to them, you would have to order it. As a result, I guess that the 'take it all back' refers to adjusting pay.

I'm pretty sure that the agreement I signed to enable direct deposit allowed for errors and overpayments to be removed from my account.

I can't speak to what specifically you signed, but there is a process guide available here: https://about.usps.com/manuals/elm/html/elmc4_027.htm

It suggests that even if technically and/or legally the option exists to recover the money directly from you, it's not part of the procedures that govern this type of thing (...as of today. It may have been in the past. Indeed, these procedures may well have been drawn up pursuant to the event discussed in TFA).

This was 20 years ago and a lot of people were paid by check. So the only way to reclaim the money from them was to underpay folks. They probably just handled it that way for everyone.

Even now, at my Federal government job when someone is overpaid we issue them a "debt letter" and then reduce their pay until the account is balanced. I assume it's because of the same reg/statute that fixes the clawback to 16.7%.

I viscerally hate seeing the name of this company. It's gross. I have a strong sense memory, and it immediately triggers bad images and smells. My brain just does that.

Why on earth would they choose this name? Don't they know some nontrivial subset of their customers can't handle it?

I'm a CEO. I won't question what my team wants or needs if they make the case, but I'd really have a tough time stomaching a choice for this database vendor. I'd probably let them use it, but good god...

Why? Just because cockroaches are gross?

Yeah. My brain just starts thinking about mold and rot and disease and I can't divert it.

It's weird, because I can watch nature documentaries of lions eating prey without flinching.

Think about them fried

With butter

Have you considered the possibility that you might be strange?

Not the commenter, but for me this is it. I'm don't mind any other insect or rodent. But for some reason roaches are absolutely disgusting.

I know how silly it sounds but I would hesitate to work on any project that worked with this tech just because it would make me uncomfortable to hear the word so often.

>nontrivial

Is that based on established data, or based on you having such a strong visceral response that you assume there're plenty of other people that agree?

Maybe this is a good opportunity to explore the experience of being a minority! It feels so weird that they either didn't even think about the possibility of people like you existing, or that they did and decided your money and good will were fine to discard.

I mean this without any facetiousness, this could be a very valuable experience for a CEO, in a position of power and authority, to carry forward. Even if you're a minority in some other way, maybe this will help you understand the experience of having dietary restrictions or allergies, or having frustrating sensory experiences with like overstimulation.

I think it's more to do with the resilience of the cockroach.

Cockroaches are the classic can-survive-a-nuke animal. Maybe that's what they were going for?

You're getting downvoted, but I kind of agree. That's a terrible name for a company that has nothing to do with cockroaches. They might as well call themselves "Garbage Labs" or "Refuse Labs."

Reply on news.ycombinator.com