Hacker News Reader

Comment by CobrastanJorji | Hacker News Reader

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.

View on HN

> Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.

If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).

You certainly don't want your service caring about a test currency.

Nobody is pricing their services in test currency

They are either pricing in $ or in "units" and then converting upstream

This is an even bigger foot gun than what's happening here

AWS supports billing in multiple currencies: https://repost.aws/knowledge-center/supported-aws-currencies

So how is it a "foot gun" to add XTS as an additional currency, for internal use only?

I presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate - the services themselves don't know about UAE dirham, but the billing system does. So XTS just becomes another exchange rate. You could even fix it as 1 XTS = 1 USD, although choosing a different exchange rate than parity is likely to surface more bugs.

> AWS supports billing in multiple currencies

Of course. I'm not saying that it doesn't.

Read it again

I'm saying, the conversion to (any) currency needs to be done downstream of the service (to a general billing service). The service needs to bill "credits".

> presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate

Yes that's what I wrote in the first post

> The service needs to bill "credits"

When talking about e2e tests, it matters where the e's are. If they are at the public interface of "the service", then, indeed, you'd test against "credits".

But ideally, there'd be e2e tests that test the consumers' experience, where the 'e's are the web interface, emails, CLI and other things consumers click on, read, enter etc. This is where a "test currency" makes sense.

Yup, this makes sense for e2e.

You can have your billing service billing in the test currency for e2e (and probably with a value that does not match the USD) so you can test "ok this should cost 1 credit so in the end I should have 18 $currency"

Thanks, I wasn’t familiar with XTS.

Now I’m picturing the 3am end of quarter fire drill in finance when they discover the company has accounts receivable of fourteen billion $XTS and it’s appearing in the quarterlies.

But these aren't the right services where the test should be, right?

There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X".

It's the third service that multiplies these things that failed. Where are the tests for that?

To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough eyes on it to notice.

I’m totally guessing though.

> human-or-LLM-driven error.

In a computer system, dont those categories cover pretty much everything except a meteor strike?

Occasionally you have faulty hardware. Most commonly RAM.

What I meant to say is an error from manual account management input— not a human error via code.

Not everything is an LLM boogeyman. Sometimes a bug is just a bug.

> Not everything is an LLM boogeyman.

Indeed.

>> To me this sounds like a human-or-LLM-driven error.

In fact, half of what’s implicated in that very sentence is not an LLM.

> well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime

You don't have a non-production environment where you can have billing that doesn't involve real money?

the cynical take is of course that the tests were written by Claude and not validated.

Unfortunately "automatic generation" of unit tests have made a big part of the value of unit tests ("I am telling you what I expect here") disappear.

Nah, this is not uncommon in AWS. I came across a similar bug 10 years ago when playing with Kinesis and nearly had a heart attack at a $2m billing alert! Turns out it was just 5c

I love how AI means that all human-written code is now perfect, and any problems are onbviously a symptom of vibe coding. Another few years of hearing this and I may begin to believe I only wrote perfect code, when my current memory suggests otherwise.

Reply on news.ycombinator.com