Hacker News Reader

My experience has been that there are typically two types of engineers: those who have worked on billing systems before and those who haven't.

I think a lot of the issues arise from the difference between payments and billing [0]. When just starting out and signing up your first customers, you primarily care about collecting a few (recurring) payments - and it's really easy to set that up with Stripe (or even just manually invoicing your first customers).

However, over time, more billing requirements gradually sneak in, such as more complex entitlements, multiple plans, grandfathering, and eventually enterprise/high touch sales customers (where the money is!) who need custom billing cycles, terms, and entitlement provisioning. Since billing is never a technical focus, numerous additions and small hacks accumulate over time, taking engineering resources away from the actual product. Eventually, this turns into an unmanageable mess that can significantly slow down the sales process or limit what you can sell.

The complexity of billing is riddled with hidden pitfalls and edge cases, and it has become even more complex now that most plans include many different limits and usage-based components and that most SaaS companies sell globally. Many later-stage companies have teams of 15+ engineers working solely on billing. I fully agree with the author that, unless it's at the core of your product, no organization should build a billing system themselves (Disclaimer: I'm the CTO of Wingback, a SaaS Billing Platform).

[0] https://www.wingback.com/blog/saas-payment-vs-saas-billing

> Under no circumstances does my engineer brain think this is a good idea. At all.

Yeah, if that's the model engineer in the "don't build your own billing system" mantra then the article had the opposite effect on me. Now I want to build such a system because this made me feel like I'm significantly more qualified than I previously thought.

Don't do it

Now I'll also do it out of spite! :)

Don't say I didn't warn you

Naw, you still don't want to, at least not in a context in which you're billing enough to be a real workout, because the non-engineering aspects will still get you. Like this: https://news.ycombinator.com/item?id=37557989 which, in short, refers to the fact that nobody in the company considers billing a system to be designed in the first place, so it gets approached with a truncated Waterfall that doesn't even have a requirements analysis phase, so you'll have no support whatsoever for your attempt to bring higher quality to billing.

Also, with all due respect to those working on billing I have seen that it is generally handed to engineers who are, shall we say, adequate for the task but no more, and unless you're 100% greenfielding you're going to be dealing with their stuff no matter what.

From another cynical point of view, it is a situation where you can fail miserably and bring great pain to the company that is traced directly to you, but there is no success; a functioning billing system is just what is expected. It's all downside and no upside for you personally.

I have a billing system (a component of the larger whole) and I'm pleased to be able to hand it off next month. I've not enjoyed my stay there. Though it was miles better than when I brushed the data lake/data analysis world. If I had to maintain this system for another few years I could and I wouldn't scream if I had to do it again someday. Dealing with the data analysis world beyond just throwing them some data is a "never again".

> ...miles better than when I brushed the data lake/data analysis world

The data pond, puddle, and lakeside cottage methodology would have spared you that pain.

More seriously, my experience with that world reinforced my view that it's all monkey motion (much effort, negative value).

Otherwise, spot on summary of IT dev work. Which now applies to "web services" too.

I've long been befuddled by the rise of "agile". It's not meant for product dev, right? Well stupid me took forever to realize that most work is actually IT (aka data processing) and not product dev.

> I've long been befuddled by the rise of "agile". It's not meant for product dev, right?

It...is?

Is "agile" meant for product dev? IMHO, no.

Very loosely, "product dev" projects have concrete deliverables, deadlines, and high cost of change (updates). So therefore are ameniable to PMI-ish critical path like methodologies. (Avoiding the evergreen unresolvable sophistry wrt "waterfall".)

The original motivation for "agile" was to protect the dev team from terrible customers (clients). Those who cannot or will not commit to requirements, actively resist any kind of formalism, have no clue what "done" even means.

Meaning most IT (data processing) efforts. And now most web stuff.

"Agile" is totally rational in those unwinnable situations. Because the dev team would always have something to show for their suffering. Lessening the likelihood of being the scapegoat come the inevitable failure.

Unfortunately, "agile" became a fad, a belief system, an identity. Curtailing any kind of introspection. Only in the last few years have we been able to have calm, frank discussions about "agile". Mostly because some of its progenators have broken ranks and done mea culpas.

> Very loosely, "product dev" projects have concrete deliverables, deadlines, and high cost of change (updates).

Why? A web based product can iterate, for example. We see the most experimentation in places like Amazon, where they're constantly deploying updates and experiments. Why should such a product have a high cost of change?

For APIs, backward compatibility, mostly.

Back when we burned CD-ROMs, distributing updates like bug fixes involved non-trivial effort.

Today, libs/packages and apps are somewhere between "agile" and PMI.

FWIW, for web stuff, I'm very interested in "test into prod" methodology. Neither agile or critical path; its something new. First popularized by the gilt.com fella (IIRC). But haven't yet found any teams (to join) that bold.

> For APIs, backward compatibility, mostly.

But you can still evolve APIs; breaking changes can occur much less than normal changes. And even then you can version them and run old and new in parallel if you want. I don't see why that would be slow.

I also don't see why making things in a way that's hard to change suddenly makes something amenable to agile.

I think maybe our wires got crossed somewhere.

I have never advocated for "agile", for any purpose. Ironically, IMHO "agile" is silly make work (ceremony) and too heavy weight. I was just trying to steelman the original justification for "agile" methods.

The PMI based processes my prior teams used had minimal communication and project mgmt overhead and very little drama. I see no reason why IT / web projects shouldn't be similarly managed. It's just that PMI is now like a lost art.

I only mentioned "test into prod" because I'm very curious about its methodology innovations. I think it would have better modeled and explained some of my prior experiences.

For instance, one of my teams would live code and then commit to prod changes while we were talking with our customers (for immediate verification and so forth). Almost zero overhead or ceremony.

We just came up with that system intuitively and couldn't really explain ourselves to others. (We had one senior VP in particular who reacted very negatively to our processes.) If we had the "test into prod" narrative, maybe things would have gone better.

> but there is no success; a functioning billing system is just what is expected. It's all downside and no upside for you personally.

I just wanted to say thank you for giving a robust voice to the frustrations I've felt after building multiple billing systems. This and your linked comment were a pleasure to read, and simply spot on.

Coincidentally, I am also pursuing a career transition toward data analysis. If I understand correctly, you are lamenting "data engineering" in contrast to "ad hoc data analysis deliverables"?

The problem is there are a lot of detail that is different just for the sake of being different and those will suck your soul. There are interesting problems in billing. However most of the issues are another government has a law that is essentially the same as every other government, but some small detail is trivially different so you need to figure out which subset of customers that government covers and then apply those details correctly to them. Nothing hard about this, just the same old tax but the details are annoyingly different.

Note that you need to get the above details 100% correct in all cases. They can put you in prison for mistakes (almost always they will just fine the company a lot of money, but they have the right to apply prison and you never know when they will want to make an example of someone)

Despite what jerf said, which I whole heartedly agree with, there are plenty of interesting billing problems for a good engineer to sink their teeth into.

But there is also a long, long tail of boring problems that you won’t have time to do properly.

To put it a different way: even after you’ve solved the “mathematical” problems of billing, you still need to solve the operational stuff. Like payments integration, GL integration, invoice formatting, tax, reports, etc etc.

I think most engineers underestimate both the difficulty of the “mathematical” problems, and the length of the long, relatively boring tail.

I’ve spent the last 12 years doing payments-and-billing-focused engineering for various companies. Most engineers, in fact, do not want to do this work. It’s like specializing in plumbing for septic tanks, most people have an innate sense of revulsion of self-protection that keeps them from doing it. You screw up the money pipes, and you have a Real Problem.

However! The problems are interesting, the challenges are endless, and it’s possible in some orgs to get recognized as the person who is keeping the money flowing.

I do have a sense of frustration that every company seems to start off with a bad design for billing, so I’m always cleaning up after negligence… but it pays the bills.

Sometimes I think I want to build an open-source billing engine. To show off what it looks like if you do it right. But would that be killing the golden goose? Currently there’s endless demand from people who don’t know.

My vision just now was to create a sand box with a DSL (that doesn't use floats) and have others write the rules into it.

   VAT.include(country, type, amount){
     if(type == "high" && country == "NL"){
       return percentageOf(price, VAT.NL.high) + amount;
     }
     throw new Error("Error in VAT.include: VAT not found."); 
   }

Another great article from Arnon on doing exactly that:

https://arnon.dk/why-you-should-separate-your-billing-from-e...

> I prefer to have the entitlement itself just store the expiration date and the details about what entitlements the subscription grants during the time period it is active. The billing system can store the subscription and sync back to the entitlement as-needed. This makes both manual billing by human operators (not to mention debugging and patching around momentary issues) and something like a Stripe integration very easy.

Having worked with SaaS companies using our (Warrant) solution for customer entitlements [0] for the past couple years, this is the approach we arrived at as well (e.g customer stores entitlements in our system and checks against them when needed, adding rules/entitlements as subscriptions are updated/deleted with their payment provider). It makes it easier for companies to work with any (or multiple) payment providers, and there's a clear separation of concerns. Someone shared another blog post by OP about separating your billing and entitlement systems [1] below, but I'll share it here since it's more relevant within the context of this comment thread.

I think the ideal entitlement system is (1) dynamic (i.e. rules stored in a database), (2) can handle one off scenarios (for enterprise customers, etc.), and also has a policy layer built on top (so it supports almost any scenario a developer can throw at it -- e.g. pro plan supports <= 5 seats, growth plan supports X feature up to N times per day, etc). I think it's also a huge benefit to have a UI where non-technical folks can make changes for customers without needing to involve engineering (which was always a drag on engineering in my prior roles as an engineer).

[0] https://warrant.dev/use-cases/pricing-tiers-and-entitlements... [1] https://arnon.dk/why-you-should-separate-your-billing-from-e...

The way that we ended up with is similar:

- your billing is just a mechanism to compute the invoices.

- you accumulate billing using a ledger of invoices sent and payments made (and adjustments, etc) - ie, receivables

- you have a set of entitlements, yes

- but in the middle you have a policy mechanism that determines when the entitlements should/should not be applied.

The trick is that entitlements are not just about billing. The policy mechanism is the “glue” between entitlements and everything else. It means you can give an entitlement to the boss of your biggest customer without it necessarily being linked to billing in any way.

> and something like a Stripe integration very easy

What I did (and would recommend others do), was a deep dive into Stripes API when designing my schema. It doesn't even cost anything to create an account and run some test charges, refunds, disputes, balance changes, etc. Of course I was modelling a more complicated setup (charges on behalf of customers), but still, useful stuff IMO.

If you have access to an accountant / bookkeeper (relative?) that can help too.

Speaking of decoupling entitlements, this article is a classic on that theme: https://arnon.dk/why-you-should-separate-your-billing-from-e...

Is that legal in germany? I don't know but I guess it isn't.

Which part?

Probably sending your customers information in plain text to Amazon.

But there is probably a way to do it, and even if there isn't, there certainly is some equivalently insane design that is legal.

S3 is encrypted in transit (like most data nowadays)

S3 can also be encrypted at rest. The encryption can be performed client-side or server-side [0]. Nowadays, AWS performs server-side encryption by default, unless you disable it.

[0]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingE...

Thereby proving that checkbox security is pointless.

What threat models does AWS server-side at-rest encryption protect against? Someone breaking into an S3 data center and walking off with a box of hard drives? AWS reselling obsolete hardware on eBay and forgetting to wipe a disk?

And yet what does it not protect against at all? A misconfigured load balancer that can access 169.254.169.254 and dump your AWS access key straight to an attacker, who can then use it to ask AWS to politely decrypt all of your data.

What does it really not protect again? The US.gov issuing a National Security Letter to Bezos threatening jail time if they don't decrypt and hand over the data.

To reach the spirit of the "data must be encrypted at rest" laws, I believe you should encrypt the data using your own keys which are maintained in an HSM that AWS cannot access. This means that you cannot run your entire stack on one cloud, which would be a huge driver of cost and complexity.

> What threat models does AWS server-side at-rest encryption protect against?

A lot of threats. A rogue employee sucking up data. Malware on AWS servers.

Not likely, but certainly possible. Server-side encryption is useful, but if you want to be _certain_ that your data is safe, then client-side encryption does exactly what you're asking for.

Client-side encryption can use a local key, or a key in KMS.

https://docs.aws.amazon.com/amazon-s3-encryption-client/late...

Is there a reason you have such a negative take?

Yeah I generally have a pretty negative take about tech regulations, especially security ones. They are often written by people who don't understand either the technology or the threats, and are often implemented by people who don't really care about the threats, just about compliance.

Client-side encryption is a lot better. If the regulations were drafted in a technology and security aware way, maybe they would include something more detailed than "data must be encrypted at rest" which would require such practices. As it stands today, you can be in compliance without actually gaining much in the way of practical security improvement.

> A lot of threats. A rogue employee sucking up data. Malware on AWS servers.

IMO server-side encryption-at-rest does not protect against either of these, although there might be some edge cases where it does. For example, malware on the S3 gateway would see the data decrypted. A rogue employee with very limited access may be stopped by this, but I would guess that most rogue employees would be abusing the support impersonation processes or would deploy evil code that can access the data after decryption.

Client-side encryption would prevent both of these threats as well as the more common ones like losing access to your AWS keys somehow. As long as the method that leaked the AWS key didn't just leak your client-side encryption key as well.

All security evaluations should start with a threat model. If you're trying to protect yourself against APTs or nation states, you have a very different challenge than if you're trying to protect against drive-by ransomware.

Even such an insane design can be made passable, by making the file in-practice-a-queue if you squint.

Remove old entries, keep a large margin of old entries. Append new records to the file. Use a counter. (Make it 64 bit if you are paranoid.)

On the "receiver" end, keep track of what you last saw. Could be as simple as keeping track of the counter.

Add only records you never saw before.

Process new records.

There are of course nowadays so many better ways, but "real" and stable systems have been built with such duct tape.