Hacker News Reader

Comment by GordonS | Hacker News Reader

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet?

This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for an API workload is absolute insanity.

These engineers are building stuff for banking. Banking!! There is literally no way I'm going near Plaid with a very long bargepole after reading this.

It I was someone senior at Plaid, I'd be pulling this blog post before it harms reputation any further.

View on HN

I mean, that's always the thing, isn't it? If a company publishes about the problems it has, the question is whether other companies have the same problems and just hide it, or whether this company is actually worse. This comes up a lot with gitlab, for instance; remember the time they discovered they had no backups? At most companies, customers would never find out about that, so I'm not sure that them telling us about it usefully informs my view of their competence. Similarly, here, the only way we'd know about this if they didn't say anything would be poor performance, which would be... less than surprising, on a financial website, in my experience. So maybe they suck, or maybe they're equal with others but a little more open; I don't know how to tell.

Thanks for a reasoned response to what I realise was a very negative comment. I do agree with what you've said, and I do feel a little bad for slamming them when they're being transparent.

OTOH, I do still feel this is so bad they need to be called out on it, and it really does scare me off using them. Given they're being transparent, it boggles the mind that they're tried to justify this, rather than just owning it, admitting it was the result of letting a junior do some resumed-driven-developlemt (or however it came about).

I posted it on my engineering org's random channel. The 4,000 instances of the same service thing immediately got a laugh out of everyone. How a tech company operated like this is beyond me...

Hah, I actually did the same, and it was basically a stream of WTFs?!

To put a positive spin on this (perhaps a first for me in this thread!), I plan on doing something of an internal post-mortem with my team, where we'll look at the deficiencies of this design, try to reason about how on earth it came to fruition, and critique our in-place review processes to make sure something like this never happens to us.

Hi, Plaid engineer here (not the author, but I helped with the post).

I don't think we've tried to assert that the old system is perfect. We went into some detail in the post about why it took us this far. Certainly, the single request per container approach wouldn't scale if our unit economics were different. We didn't get into this too much in the post, but the Node service sits behind a couple of layers of Go services, so the we had more control over scaling API traffic than it might appear.

Likewise, I hope we didn't give the impression that the new system is perfect. We've explored other languages for integrations in the past (even Haskell, at one point), and are continuing to do so. A migration away from our years-old Node integrations codebase would be a massive undertaking at this point. Absent that, it doesn't seem consistent to say "you're incompetent for handling 1 request per container" and also "you're incompetent for writing this post" – if you believe the former then it makes sense to be an advocate for this project, at least until a language migration can be done.

I think the set of hoops we had to jump through in order to add concurrent requests without adding latency is a good demonstration of why we didn't do this sooner. It wasn't a massive undertaking by any means, but it wasn't trivial. At any rate, we're not really looking for a gold star here – just putting this out there and hoping this will be useful for others who are, as other commenters have put it, building their own "Frankensteins" :)

I mean, my read of this is:

1. We used a system which uses event loops to achieve great concurrency, but we turned that off because we don't trust it. 2. Instead, we spent $300k/yr rolling out one-process-per-API as though we were using Apache 1.3. 3. We used an arbitrary JSON library without knowing anything about its performance characteristics, which it turns out were inordinately bad

It's not that this wasn't a great exercise in engineering and problem-solving, or that it's not a great demonstration of how to solve scaling problems at scale, those are definitely true. It's more that "we spent $300k/yr more than we needed to so our engineers didn't need to learn how to use our technology stack properly."

I'm not meaning to be harsh, I've kludged enough garbage into production in my lifetime, but more that the fact that you got into that situation in the first place gives a poor impression of either your development team or your development processes.

I don't disagree with most of what you're saying. The Nth engineer at a startup rarely looks with admiration at decisions made by the (N/10)th engineer – but it was those decisions which helped the company grow to its current size. Likewise, I think most of us will be happy if the company 10x's again. Then some super-duper-senior engineer can look at the decisions we're making now – they're not perfect, but we're doing the best we can with our current knowledge and resources – and gripe about them. The circle of life goes on.

FWIW, I don't believe Node was chosen specifically for its concurrency – it was just the language chosen for the entire stack by the company's founding engineers, and lives on in just this one service.

Gah, I can't believe you're still trying to justify this madness!

In a parallel universe, a barely-competent engineer would have designed something more far more obvious, simpler, performant - all while using less hours, and not borderline-fraudulently wasting substantial amounts of your VC's money.

If the company 10x's again, it won't be because of poor engineering, it'll be because of marketing and VC's who don't know how you're wasting their money. If the company 0.1x's, it'll likely be because of a security breach because of appalling design.

So, in your estimation, plaid's engineering is a bunch of less-than-half-competent madmen, who might as well be committing fraud, correct?

Is that overly negative or just the right amount?

Ah, a serious question as an aside to my last (scathing) comment - does Plaid have architects? What about architecture and/or code reviews?

I'd be very interested in reading about that.

Your question feels pretty sarcastic, but I think it'd be great to write a post about our process for projects specs and reviews. We do have an old post on the blog about code reviews that you could read, but it's more focused on the cultural approach to making code reviews less intimidating for junior engineers, less about the details. While we don't have an "architect" title, a fair portion of our more recent hires have 10+ years of experience (I don't have exact numbers though), and we're working on the best way to coordinate efforts and improve our system-wide architecture.

In general, I think it'd be great if more companies communicated their engineering practices and war stories externally – I'd love to read such posts myself! Unfortunately, it takes quite a bit of time to write a post (for engineers who are already stretched thin), and it seems that being honest about shortcomings at an early-stage company is an invitation for people to be personally disrespectful. It is what it is, but I imagine that's one reason we see such posts from only a small handful of startups, and the subject matter is often cherrypicked and sugarcoated.

So anyway, thanks for engaging with our post all the same – and maybe we'll be back with a post on architecture reviews in 5 years when all the kinks are ironed out :)

If I may add, Monzo bank which utilize Golang has interesting stack.

https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac...

Reading the article, I'd fully expected a post-mortem at the end, describing how architecture and code review processes were going to be tightened up to ensure a monstrosity like this never happened again - that would have been transparent, interesting, and given me confidence in Plaid's engineering.

Instead, you've peppered this thread with comments that kind-of, sort-of justify the approach taken.

I'm sorry, but this approach cannot be justified - it's overly complex, and far from the simplest or most obvious approach. I'm truely shocked that Plaid has produced an architecture like this, and doubly so that Plaid would try to justify it. My guess here (and given the attempts at justification, this is me being really charitable) is that a junior dev was given too much leeway, and did some resume-driven-development, just so they could say they'd worked with 4k containers.

It's important to keep in mind that efficiency isn't usually particularly important for a startup. I'm sure they knew when they initially set up this system that it wasn't performant...but it was nice and quick and easy and gets the feature out the door. Why should they worry about $100k or whatever when they're funded for > $350M? Their bottleneck is engineer hours, not dollars.

Instead the rational thing to do is build something quick and dirty and optimize later, and that's exactly what they've done.

I understand that, and plenty times myself I've "done the simplest" thing - sometimes you need to ship an MVP, fast.

The difference here is that what they did wasn't even the simplest thing - it was a crazy, insanely wasteful thing that just happened to work for a while. Being honest, for me, it's an indefensible approach.

> Why should they worry about $100k or whatever when they're funded for > $350M? Their bottleneck is engineer hours, not dollars

Arg, but this rubs me up the wrong way! Any half-way competent engineer could have built something simpler and much more performant, and likely in many less hours too. Sometimes stopping, thinking and discussing for a few minutes or hours will save numerous hours. I mean, how many hours did they spend on this "diagnosis" alone?

>Their bottleneck is engineer hours, not dollars.

Their bottleneck was software being able to scale past a hard stop. I guess having a known breaking point of scalability is a good thing? But building things in a way where you either have to overhaul your development runtime or not be able to scale past a certain point is pretty terrible.

It seems like the only reason they did this was because they really felt the pain of it from the business and dev side and they were lucky enough that they had traffic spikes to raise these issues. If they had more consistent day-to-day traffic then this would have just hit a breaking point one day and they would've been fucked until it was fixed.

Frankly the reality is so surreal there, it's actually surprising it doesn't mention uploading csv files over ftp, generating excel files, having cameras pointing on monitors of legacy systems that read data (no, this is not a joke) or spawning a promise for Martha to cross check something and click ok somewhere behind two bastion hosts, three firewalls and one and a half soap integrations. I'm not defending "4k containers because even loop can be blocked" which is silly, just reminding of the context - in other words you can do shittiest automated thing there and you're a hero. Next year-or-two hero is going to be somebody shrinking it by another X orders of magnitude.

I kind of alluded to it in my reply, but I tend to agree- they spent a lot of time and hard work- looking in all the wrong places! Its hard to imagine how they missed the forest for the trees so badly here.

Worse is- they never really explain where that 30x improvement came from- or if they even understand it themselves? They talk a lot about getting their memory issues under control, but hardly at all about actual parallelism- and it seems that even then they confuse it with merely speeding up operations that are blocking.

I kind of expected this post to be "We did a whoops and had a blocking call to a DB/fs/compression call/whatever. This was all happening in the event loop and not being farmed out to the threadpool by libuv. We fixed it and now look like heroes to our CTO!"

What they talk about are issues that blocked them from parallelism per node and how they resolved the issues. I'm not sure what additional information you're expecting?

Though I'm somewhat surprised they didn't use Worker patterns per node with self monitoring for health above and beyond what they already did.

For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later.

The simplest solution is to scale to one worker per node initially if you're doing anything compute intensive... once you've done that, and/or you need better performance for any number of reasons including cost, then you can do more. Now, I'm not sure I would have gotten to 4k nodes before I started to re-evaluate parallelism or better scaling options, but the initial implementation is absolutely fine.

> For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later.

I get it, but come on - this was not a "performance optimisation" issue, but one of bad architecture; an architecture that certainly doesn't inspire confidence in the priorities you mention: accuracy, simplicity, safety.

Having a worker that does one request, processes that one request and returns a result isn't accurate, simple or safe? Scaling that simplistic interface in a 1:1 manner across many systems via ECS is pretty straight forward.

Now, in addition to probably optimizing what they've done, converting to, for example another container system, like K8s where they can scale vertically a bit better may have been another approach.

The biggest issue that I see is gRPC doesn't work that great with Node. You can't use it with cluster which means you have to self-manage threads/processes and it adds complexity there. Yeah, there's definitely issues that come into scaling in terms of performance optimization.... but where they started from isn't unreasonable imho.

Where I work compliance is job #1.

That doesn't prevent us from thinking about performance. GTFO with this nonsense.

Your comments in this thread have been breaking the site guidelines, and getting worse as they go along. Would you please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here? Note the bit about curious conversation.

They started with a simple implementation where one node handles one request at a time... end to end. They used ECS for easy-button scaling. That's a perfectly reasonable approach for starting out.

I would have probably pushed for a shift in orchestration to kubernetes along with some tweaking as an initial uplift. Others would re-write the whole thing in another language. They chose to add a bit of complexity for multiple requests per node support. They all have their pluses and minuses, but in the end it doesn't mean the initial approach was bad, or their refactor wasn't pragmatic or practical.

Dramatic rewrites to a codebase lead to instability and in practice fail as much as succeed.

> Dramatic rewrites to a codebase

Otherwise known as re-architecting?

Linus Torvalds quote: "You need to grow thick skin". I also got a knee-jerk reaction when reading the first part, but the article explained it well, given that they probably don't want to give out too much information.

So how would you have engineered it? I would just send the data uncompressed granted that the receiving server is probably in the same data-center with switches capable of handling Tbit's of data per second.

I liked the article, but would have wanted more details. I love optimizations, it's such a drug, the rush when you make something x times faster. This article doesn't give me a bad impression. Contrary I'm thinking about sending an application.

In their defense. It looks like they have over 400 employees and raised over 350 million in funding. On all things that truly matter currently they seem like a very sucessfull company.

I can guarantee you a VPE or CTO who can say they helped do that... but ran into a scaling issue from their success will have no issue with employment and no reason to be ashamed. All the more impressive if it was just a bunch of junior engineers.

This comment says more about you than it does about Plaid. Their "insane" design met business requirements successfully enough to grow them into a multi-billion dollar company.

Did you consider the likely (and more charitable) explanation that they were aware their design was "bad", but had higher priorities until now?

If I were you, I'd be pulling your comment before it harms your reputation any further. :)

> Did you consider the likely (and more charitable) explanation that they were aware their design was "bad"

I think marketing and VC valuations grew them into a multi-billion dollar company; whether they remain so, to a large part relies on how fast they burn through VC cash - so, not looking too good on that front...

No even half-way competent engineer would come up with such a complex, unperformant solution to a simple problem - I think a higher priority should be hiring engineers who actually have a clue what they're doing.

As for meeting business requirements... while this might have worked for a while, it was plainly not a good way to meet them, and given Plaid are in the banking sector, really doesn't bode well for the future (I'm having flashforwards already to security breaches, plaintext passwords etc...).

Downvoted for angering the VC class...

>multi-billion dollar company

WeWork is a "multi-billion dollar company" in the same way that Plaid is. Private funding valuations don't really mean anything anymore.

> no way I'm going near Plaid with a very long bargepole after reading this

But you'd go to a competitor who hasn't published a blog post, whose internal code you haven't audited and simply presume is just fine?

In plaid's defense, lack of performance tuning isn't necessarily a lack of security focus.

> In plaid's defense, lack of performance tuning isn't necessarily a lack of security focus

Come on, this is not about "performance tuning", where you're trying to eek out every last drop of performance - it's about a completely indefensible, complex, wasteful solution to a simple problem.

I'd say engineering insanity at this level is very worrisome for what they've done at the security side of things.

ROFL "performance tuning" this is not tuning this is architecture.

Some people think you can just write software, sell it to customers, and it's "tuning" to make it work properly.

You should be fired from whatever job you have.

My guess is that you have no job, you are fronting USD.

In which case you have absolutely no place in this conversation and you should be ashamed of yourself for speaking up.

A fool and his money are easily parted.

We've banned this account for breaking the site guidelines and ignoring our request to stop.

If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future.

https://news.ycombinator.com/newsguidelines.html

Reply on news.ycombinator.com