Balanced's Architecture | Hacker News Reader

Balanced's Architecture

blog.balancedpayments.com

105 pointssteveklabnik33 comments

Solid architecture. For those not familiar with payment industry terms PCI stands for Payment Card Industry. You usually know this as PCI DSS (Data Security Standard). Basically all the payment card providers got together and said that you need to follow these rules if you want to accept credit cards. The rules are pretty straightforward ( have a firewall, don't save CVVs, encrypt credit card #s at rest). You then pay an auditor to audit against PCI DSS.

If I were looking for vulnerabilities I'd probably start with any XSS. Chances are the credit card data is locked down tight and encrypted. But what if I can scoop it up as it gets transformed into a token? Also look at where you store the encryption keys to decrypt the card data. There are hardware devices you can use that are especially hardened.

The problem with bitcoin is that it necessitates an even more secure architecture because you don't have a 3rd party to run to if things hit the fan. Suppose all your credit #s got stolen in this case. You can run to Visa/Mastercard and they will invalidate the card #s in bulk. Or at least do monitoring on them. What do you do when all your bitcoins are stolen?

With the Coinbase integration, the OAuth secret is the equivalent of the card number (and stored in knox like all other sensitive data, even though OAuth tokens are outside of PCI scope), and so if they got stolen, could be invalidated by Coinbase.

With a name like balanced, I was expecting double entry book keeping[1], so: you give an account to a device, e.g. the credit card processor, and measure how much money is entering and leaving the company per device. Plus you give accounts to each customer and measure money going in and out their separate channels.

Then you check these two different modalities of tracking money conclude your the same liquidity measure for your company.

If they do not then you have sprung a leak somewhere and can halt everything.

That would be an extra layer of security. Passing loads of tokens round is still a single point of failure at the conceptual level.

[1] http://en.wikipedia.org/wiki/Double-entry_bookkeeping_system

This post wasn't talking at all about ledgering, just about high-level components.

If ledgering is interesting to you, http://blog.balancedpayments.com/the-ledger/ and http://blog.balancedpayments.com/state-machines/ are what you're going to want to read.

oh great reads. Thanks

(Former PayPal employee here.)

I first have to give you a thumbs-up for calling your fraud detection layer "precog".

Does AWS let you firewall off Knox from the open internet? PayPal's architecture has most of the machines that touch payments isolated behind hardware firewalls, with only certain front-end machines able to punch through the firewall.

Amazon has VPCs which are virtual private clouds. They let you configure networks with specific requirements, such as being closed off from the outside world.

Once closed off from the world, only your servers within the public subnets can access those in the private subnet. By default, the private subnet can't talk to the outside world. You'd typically setup a NAT instance in your public subnet that tunnels your private subnet's internet to the outside world (because the NAT is in a public subnet, it can access the outside world).

That's just an example setup. It's a very powerful tool for securing your infrastructure. For example, you should typically put your databases, and anything that isn't password protected that stores information or something (except web servers) in a private network so that only your public servers have access to them.

User -> Public Network -> Public Server -> Private Network -> Private Server -> NAT (Tunnel) -> Public Network -> Internet

VPC does take quite a bit of effort to setup, but after that, it's pretty straightforward.

I wonder: is EC2 secure enough for this type of credit card store? What if the management layer running the underlying hosts is vulnerable or a XEN zero-day vulnerability shows up?

I'm sure Amazon does a lot on securing its infrastructure, but for credit card data wouldn't a physical, fenced off server be more secure?

I suppose it comes down to the amount of investment available. Amazon can pour resources into security, monitoring and have a large staff actively keeping an eye on such things. They're signed off for PCI compliance Level 1[0] (Any service provider that stores, processes and/or transmits over 300,000 transactions annually) which helps isolate you from a lot of costs around getting your dedicated hardware audited yourself.

It's also worth noting that Amazon.com itself is hosted off AWS (since ~2010) though I'm struggling to find a good cite for that

[0] http://aws.amazon.com/compliance/pci-dss-level-1-faqs/

[1] http://www.dummies.com/how-to/content/amazoncom-runs-on-amaz...

I love that plexiglass on the wall type of whiteboard. Is this a premanufactured solution or homegrown?

We repurposed the glass tabletops from IKEA desks and threw them on the wall for white boards ;-)

Am I wrong to want to know more about how (for instance) communication between the networks and components is implemented? This post, while well written, doesn't really answer the questions I might ask about any given architecture.

You're not wrong! You can only fit so much in one post, and I wanted to keep it fairly high-level here.

I got asked this on Twitter, you might find the thread worthwhile: https://twitter.com/moritzheiber/status/442037431089786881

It seems like a lot of people are interested in hearing more details, so I'll try to get into that eventually in another post. Always more to talk about!

Thanks very much. The question about latency is certainly relevant, and I would add reliability to that too. I'd definitely be interested in a technical follow-up, when you have the time.

I would really love to use Balanced, I'm really excited for the escrow and the bitcoin features... If only they accepted customers from the Netherlands.

Yup, it's a bummer. We're working on it, but it's really hard. :/

Great post and diagrams. How do you manage/centralize logs ?

Thank you, they took forever to draw... My handwriting is terrible.

We wrote about logging here: http://blog.balancedpayments.com/status-page/

  > We already log these to a centralized server using RSYSLOG, so I already had
  > a data source to draw from. Next, I went and brewed a fresh pot of coffee and
  > bestowed it upon bninja for his prescient work in building our log parser,
  > Slurp. We wrote a quick Slurp script that read the HTTP status code from each
  > request and then fed them into Graphite buckets. Each bucket was based on
  > service name (DASHBOARD, API, JS) and then response code family (2xx, 3xx,
  > 4xx, 5xx, and a special case timeout for slow requests).

If infrastructure stuff is interesting to you, you may want to check out https://github.com/balanced/balanced-infra . If there's interest, I might blog about it in the future.

what do you do when rsyslog drops log events?

I am not an ops person, but my understanding is that rsyslog does 'store and forward', which would allow replaying of the log if something was dropped: http://www.rsyslog.com/storing-and-forwarding-remote-message...

Great way to explain on a blog. Well done!

Who removed (YC W11) from the title of this, and why?

The HN mods typically sanitize titles to match the title of the underlying article. (Except when they don't, of course.) They are generally unwilling to explain themselves, even when their edits are clearly making the title worse.

It's honestly a mystery to me why the system even lets submitters specify titles.

The problem I have is that Stripe, a competitor to Balanced, seems to keep the (YC S) emblem every time they have a story. It feels like YC is trying to hide the fact that Balanced is a YC company I guess. Pair this along with the fact that (IMHO) Balanced is killing it, but I only ever see PR about Stripe (except for a few weeks ago, when Balanced announced a partnership with CoinBase, another YC company)

But I could be paranoid.

> The problem I have is that Stripe, a competitor to Balanced, seems to keep the (YC S) emblem every time they have a story.

That doesn't seem to be the case:

https://www.hnsearch.com/search#request/submissions&q=Stripe...

I think you are paranoid. I see them both tagged and stripped equally. Just depends on the random whims of the moderators who do not explain themselves at all.

Yet another unlicensed money transmitter handling Bitcoin transactions (through another unlicensed money transmitter). What could go wrong?

Are you posting on the right thread? Balanced Payments handles credit card and ACH payments, not Bitcoin.

Balanced did start accepting Bitcoin through Coinbase http://techcrunch.com/2014/02/20/balanced-coinbase-bitcoin/

I expounded more on the technical details of that integration here: http://blog.balancedpayments.com/more-details-about-bitcoin/

Expound on the details of your legal status. You're an "open company," right?

Ah yes, the scheduled trolling of payment companies by Aaron. Wouldn't miss it for the world.

Interesting, missed that, thanks! Interesting model, although I wonder who is shouldering the risk of BTC in that relationship.

Reply on news.ycombinator.com