Hacker News Reader

Comment by throwaway_str | Hacker News Reader

From the engineering friends I know at Stripe, you'd be correct: there's a strong cultural push to not question the status quo / prevailing wisdom.

On the tech side, they continue to build on top of a broken foundation (MongoDB), resulting in millions of man hours wasted dealing with the complete lack of transactions. Mongo now has transactions, but last I heard Stripe was still running a very outdated version and spending absurd amounts of time dealing with issues that transactions would have entirely avoided. If you suggested that maybe it'd be worth changing course to something like Postgres because of the insane amount of work being wasted, you'd be shut down for not being optimistic enough.

View on HN

It is always amazing to me that people downvote actual anecdotes that highlight the arguments being made, while at the same time upvote rampant speculation that has no basis in fact.

Having worked at Stripe, these exact examples came up more than once. I can verify their authenticity.

The fact that Stripe ever used MongoDB is dubious enough, but that they've endured on it well past the fad's expiry is reason enough for me to avoid doing business with them in the future.

Handling money is serious business and if one insists on using the unproven flavor-of-the-week to do so, they'd better make sure it has at least the fundamental requirements to accomplish its task, e.g., transactions/MVCC. Stripe apparently failed to do so, and has chosen to propagate that failure for many years.

One can make excuses for the spunky startup with a handful of employees needing to save time by using what they know or whatever, but Stripe is well past that point, and serious people should've taken over by now. I'd expect "replace Mongo with a real database" to be near the top of the todo list for any serious people.

FWIW, having worked with the Stripe API for ~5 years and put all our sales through them for the lifetime of the business, I wouldn’t have guessed they used MongoDB. The API is mostly excellent, and it has always presented a consistent view of the world to our side.

The same cannot he said for a number of other services we use, including those from some companies with supposedly very good tech credentials. We’ve had to back out of integrations due to the clear evidence that they are non transactional and/or eventually consistent.

You couldn't tell what backend they use just from their API unless there's some errant exceptions leaking information. Their test API might give us some clues. It can only purge your data one record at a time, at about 1.5 records a second, and locks your API credentials for however long it takes to crash or finish iterating through each object. I don't know what storage engine would have that constraint but they've stuck with it for years and simulate a fast test API with a whole other piece of software instead of just having a fast test API.

What storage options don't have the ability to delete all your records at once quickly? If it was MongoDB there would surely be an index attaching the data to your account. Same with PostgreSQL.

You can make all those mistakes with a traditional RDBMS. Iterating over objects to delete them one at a time with database round trips is also a common mistake people make with ORMs(along with many other things that could have been done in one statement).

I’ve worked with Stripe enough to have a hunch that it was Mongo. Never really thought much about it until it was confirmed here just now. There’s a lot of immutability in their system, tons of it makes sense but some of it smells of a weak data store.

You are laying down some hefty assumptions here and I'm not sure they are fair.

You seem to indicate that Stripe has consistency issues with their payment APIs. Have you seen this? Having worked directly on these APIs (note, I said ON and not WITH, I worked at Stripe on these products), it would greatly surprise me to see non-trivial consistency issues with their data model that surrounds money movement. Stripe might be on Mongo, but assuming that they have consistency issues because of that is quite a leap. Remember, Stripe is regulated federally and in most states, as well as in their international jurisdictions, so the claim you are making would be one regulators might be interested in if there is any veracity.

You also seem to be insisting that there are no "serious people" at Stripe, and that might be true! At least, I hope no one has mistaken me for a "serious person". But what does "serious" mean here and why can only "serious people" work on financial infrastructure? Moreover, why do "serious people" not see Mongo as a "real database"? In fact, what is a "real database" to you, other than something that has transactions?

I'm very interested in your answer here; while I had my issues with Mongo in my 2.5 years at Stripe, most of the issues I encountered I would lump into the "this is a trade-off consequence" bucket, as Mongo solved a whole lot of other issues for me. The other main bucket I would have used was "distributed database problem". I know Stripe had no plans, nor should they, to move to a non-distributed data store.

No one is forcing you to use Stripe, but I would suggest that attacking Stripe for their technology usage is not very useful.

> what is a "real database" to you, other than something that has transactions?

How about "Atomicity, Consistency, Isolation, Durability"? That would be a good start for a database.

A nearby comment suggests lack of set-based operations (e.g. delete), so that would be another nice thing to add.

> The fact that Stripe ever used MongoDB is dubious enough, but that they've endured on it well past the fad's expiry is reason enough for me to avoid doing business with them in the future.

Just wait until you find out what technology actual banks use!

COBOL, Fortran, mainframes, and other 70s-era tech are much more trustworthy than new-fangled VC landgrabs like MongoDB. As long as the banks haven't transitioned everything to Cassandra or whatever, I don't think we'll have an issue there.

> Handling money is serious business

How do you say this without knowing that governments around the world feel the same way and so heavily regulate this space? Do you know any publicly available recommendation from these regulators against MongoDB?

> replace Mongo with a real database to be near the top of the todo list

This is where the businesses separate themselves from technology enthusiasts. Let's assume this is made the top priority, in your opinion, what task immediately follows? Or what former priority does it replace?

> Do you know any publicly available recommendation from these regulators against MongoDB?

If there isn't one, there should be.

>Let's assume this is made the top priority, in your opinion, what task immediately follows? Or what former priority does it replace?

Mongo jeopardizes the overall security and reliability of the system, thus exposing Stripe to serious liability, so I'd say replacing it is part of the basic expectation from any production-level computer system. Obviously I don't have a copy of "Important Stripe Person's Priority List", but I'd assume such basic functionality is part of an implicitly-assumed functional baseline, and that Stripe would expect said important people to escalate and develop a plan to handle such a fundamental flaw ASAP.

If we were just tracking people's favorite cat videos or something, it'd be one thing. Attacks targeting malicious data modification and exploitation of races wouldn't really be that much of a concern. It's not great to have that attack surface but not necessarily the end of the world if someone's favorite cat videos get added to their account 1000 times. But when we're dealing with money, this kind of thing needs to be taken seriously, and tacking on a big pile of crap in front of the datastore is not a serious way to address its fundamental shortcomings, at least not when there are many better, proven options on the market.

Just in case someone thinks I'm being an alarmist here, this has already happened; multiple bitcoin exchanges have been pwned due to their reliance on Mongo's flawed semantics. [1] I can only assume that systems like Stripe haven't been attacked similarly because it would provoke the ire of much bigger fish, and why do that when you can knock over smaller bitcoin exchanges?

[1] http://hackingdistributed.com/2014/04/06/another-one-bites-t...

Transactions aren't the only solution for that problem space, and using them can introduce other (likely even harder-to-solve) problems with distributed systems.

I'm currently dealing with precisely this exact scenario and database. Dated Mongo, homegrown ORM, a closet full of opaque hacks that are apparently all related to some MongoDB feature, etc. And I don't know if this is a cultural issue with MongoDB users, but my god the database has its hooks throughout the stack, all the way to the UI. It is amazing to see shit like this in 2019 and I've seen some heaps over the past few decades in this field.

But the optimism point resonated. Before reading your comment, I actually reflected on whether I am being pessimistic. I'll tell you why: at this point in the game, cleaning up the software would effectively require a rewrite. And the company is in no position to do this. They do not have the talent, leadership, or effective recruiting necessary to undertake a meaningful second system effort.

So I think I've come to the realization that the job is a poor fit and I should move on, but while I am here it actually wouldn't help pointing out the obvious issues with the system. There really is no point.

Wouldn't be easier to just upgrade mongodb, which is no longer broken? Unless its api is not backward compatible?

From what I understand there have been various attempts at this, as well as other data storage solutions, but they always run into issues. MongoDB is being used as a database and data warehouse and there are always complications.

Updating any non-trivial layer in your stack usually gets you into "complications" - that alone must never be an acceptable reason to be stuck on old versions, especially if there are serious problems with them.

You simply chew through those complications. Again and again. And try to optimize your application for less complications in the future, if possible.

Reply on news.ycombinator.com