Hacker News Reader

Comment by dkarl | Hacker News Reader

> Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.

I wish people would follow this rule and just let stuff work. I recently encountered the most extreme version of this I've ever seen in my career: a design review where a guy proposed a Redis caching layer and a complex custom lookup scheme for a <1GB, moderate read volume, super low write volume MySQL database. And of course he wants to put the bulk of the data in JSON fields and manage any schema evolution in our application code.

Can't we just let stuff work? I'm no fan of MySQL, but can't we admit that a ubiquitous and battle-tested piece of technology, applied to a canonical use case, on tiny data under near-ideal circumstances, is probably going to work just fine? At least give it a chance before you spend days designing and documenting a bunch of fancy tricks to save MySQL from being crushed under a few megabytes of data.

View on HN

That case you've seen speaks of the guy's inexperience and lack of understanding.

I have a problem with this rule because what I see happening is people taking it to heart and no longer thinking about what they're doing performance-wise. And then the program is working 1000x slower than it should, at no extra gain (and often a loss) of readability or safety, just because someone decided to use O(n) data structure where O(1) would do, or keeps repeating the same computation thousand times instead of ensuring it's done once.

So to your "let stuff work", I want to also add: "understand the work you're putting the computer through", and "don't do stupid things".

Architecture is not optimization. This gets fuzzy with the decision to use caching, static generation, etc. Ideally, redis or memcached can be added when needed. It will require some changing of the app but hopefully in limited places.

This rule a la Pike is about doing things like writing assembly or manually unrolling loops in noncritical parts of code. However, in some code, almost everything is on the critical path, and that requires architecture. I’m thinking of Carmack’s single function game loop here.

I remember working on a project that claimed to want 10,000 transactions per second. “Okay”, I said, “how much can be accomplished in 100us?”

They looked at me like I was an idiot. “No, it’s going to be clustered and pipelined.”

“Ok, if you can do that maybe you have a budget of 1-5ms.”

“No, we’re going to give each transaction up to a second to get done”

I smiled and admitted that they were obviously a lot smarter than me. Oddly enough the product never saw the light of day.

Don’t worry, we’ll use serverless! So we can support an unlimited amount of transactions per second!

Latency and throughput are not the same thing. A CPU can process multiple instructions per clock cycle yet instructions take longer than a clock cycle.

That’s not obviously absurd. If Amdahl’s law permits then Little’s law tells you whether or not it’s possible.

Yep. As I said I’m not smart.

Wow, you are really good at this.

Let the "geniuses" figure out that to do 10k transactions/sec with processes that can handle 1 transaction/sec would take.....10k processes. Sure, doable, in specific contexts, given enough resources. But they sure as heck aren't free resources!

I think maybe you are misreading the rule, it doesn't say don't optimize, it says when optimizing, don't guess from the code where the bottleneck is, go measure it.

Yeah but some things like how you structure your data(which then drives CPU cache misses) aren't something you can easily adjust/tune.

Usually when you encounter one of those it's a rewrite/rearchitecture of a whole module/subsystem before you see any gains. Been there done that, not excited to repeat it again.

The other rule is choose your data structures wisely.

The number of people who don't understand when to use a list/array VS dictionary/hash table vs lookup object is too damn high. A huge amount of basic optimization is constantly at their fingertips and they nearly always choose to make a list/array and use linq to join/merge multiple relational data sets instead of optimized standard objects.

sure, sometimes your approach is at fault, but as the rule says, sometimes it's just a surprising bottleneck. Having done a lot of embedded work, I've found that many of those can often be fixed in a straightforward manner. Sometimes, you have to change your approach though.

It is all about lack of knowledge because checking where the bottlenecks are is one thing. Knowing why they are there is another.

A retailer app was very slow. The developers proposed a newer faster server with a newer Oracle version.

Then I took a look at one of the slowest queries. Changed it so it could use the indexes in a better way and the query went from +20 to 0.7 seconds.

The developers measured the query was slow, they checked the query used indexes and they were right that a server with more RAM and a newer Oracle version could improve the speed. But they missed that the query had to fetch a lot of data (using indexes) before it could start filtering the data. The only thing I did was to change the query so it could filter the data first.

Bolt - 95 cents

knowing where to put the bolt - 95 bucks per hour.

I've had many similar experiences with Postgres. I always seem to be the one adding the query logging to the apps. Then I feed the prod queries into the query planner. Then run the results through this tool: http://tatiyants.com/pev/#/plans/new Tweak the queries, add/remove indexes. Voilà.

The developers didn't actually know where the problem is because they didn't root cause it. They knew it lay within a certain pretty large bounding box (a particular query), but didn't drill into it further, beyond checking that indices are used (eliminating some inadvertent unindexed search as being the root cause).

If you actually know where (or each one of the multiple wheres if several places collude), that is usually very close to knowing why; often the same.

They should have had the intuition that if a query takes 20 seconds, even in a testing scenario where the system is not bogged down, it must be churning through a lot of data all over the place. Then think: does the query actually need to be looking at a lot of data? Maybe it's wastefuly looking at more records than necessary. They didn't imagine what the machine might have to do to satisfy the query, just accepting it as a black box that the DB has optimized as well as it can be (so just throw hardware at it).

Sounds like ops vs dev (or DBA) right there; it does make sense to a point that ops would grow into that kind of behaviour, given that they can't really allocate resources into performance enhancements for software (especially if it's from an external party or off-the-shelf). To them it's a black box.

But if it's an in-house thing then there should be open communication lines. The SRE paradigm makes sense there, where I see a SRE as part dev, part ops. They can identify perf issues as a software problem and either fix it or send it back to the authors.

It was more like dev & dev.

I am not criticizing anyone because I was just lucky to notice the query could be rewritten. But it was also the fact that I had a little more knowledge about how the db engine handles queries.

So we all looked in the right direction, we all found the bottleneck but the solution was different based on different knowledge.

I would go so far as to say, their "solution" was not a solution at all. Granted, in the absence of being able to recognize alternatives, it may have been the best available option left, but it fundamentally missed the root cause of the query slowness.

Your experience exactly shows why having a diversity of opinions/backgrounds/expertise on a team is a very valuable trait. Had no one realized you could rewrite the query, would scaling vertically and upgrading Oracle been a fatal mistake for the team? Probably not, but damn if it wouldn't have been a big waste of time/money.

This is particularly exasperating for me. I can't tell you how many times in my professional career I've ended up speeding up systems by removing two or three layers of improperly-implemented "caching" and using good ol' MySQL and a basic understanding of algorithmic time complexity to simplify things.

Me too. I've seen a few systems that replaced their simple request requiring 10,000 queries "optimized" by requiring 10,000 cache lookups when they should have just added some joins. The bottleneck is the network latency, not the database. The worst I've seen is an nHibernate cache stored in a session variable, half the database was being serialized/deserialized on every http request. Fortunately that was a small database.

Even with in memory caches I've seen systems grind to a halt by death of a thousand cuts, dictionary based entity attribute systems where each attribute is looked up individually. There seems to be a mentality that constant lookup == free lookup and devs don't seem to realize that constant * $bignumber == $biggerNumber. Caching shouldn't be granular.

Obligatory latency numbers every programmer should know: https://gist.github.com/jboner/2841832

It very much depends. I've seen too many cases when business logic was moved to the database layer, with absolutely disastrous effects. Scaling application backends doing map-reduce on a pre-filtered datasets is way easier than scaling database to handle five-way joins. Yet,I've seen opposing cases too, when rdbms was used as a huge key-value storage , and was pushing hundreds of thousands of rows to tens of backends, sucking it's bandwidth dry.

Having worked with Hibernate recently on a smallish database (that isn't expected to grow too big), my take away is to minimize the network trips. If you have to check something for a list/set of things and all of that can be done from, say, two not so big tables, fetch the superset of records via a join with as much criteria pushed to predicates as efficiently possible and handle rest of the logic in application code. It is so much more quicker. Of course, like most things in software, conditions apply like network latency/total no. of records being fetched by the query/potential for data to suddenly grow etc.

Not to cache simple trips to the database is pretty uncontroversial, the are more controversial cases elsewhere.

Say you consume a Restful API to hidrate order data in another system. Do you fetch orders as:

  /orders?ids=id-1,id-2,id-3

or separate calls to

  /orders/id-x

which can be cached and retrieved by id as a memoized function?

Well if you had to pick only one the second is probably better, but the best would be abstracting away order fetching in application code to always fetch single orders and behind the scenes looking up the cache for singles and pooling all the misses into a single request to the plural endpoint.

If this is a common occurrence the former would almost always be better, the second would be absolutely glacial.

> but the best would be abstracting away order fetching in application code to always fetch single orders and behind the scenes looking up the cache for singles and pooling all the misses into a single request to the plural endpoint.

Unless a significant amount of requests have 100% cache hits then I doubt a local cache will make much of a difference at all, all it's saving is a bit of bandwidth.

I had worked on an application that saw less than 100 writes per minute and about 1k reads per min and used a caching layer in front of the DB. Not only was the cache actually slowing us down, it was also inconsistent with the DB. Can't even begin to express the amount of lost dev time and productivity. We couldn't get rid of it, because someone above was convinced that we would need it to scale in the future.

There are only two hard things in computer science: naming things, cache invalidation, and off-by-one errors.

I'm trying to never implement any caching if I can help it. The database itself does caching already as well.

And if you DO need caching, keep your hands off of the application; add a cache layer in front, or between the application and the database. But don't invent it yourself.

  And if you DO need caching, keep your hands off of the application; add a cache layer in front, or between the application and the database. But don't invent it yourself.

so, redis?

My tech lead once speed up our system by just turning off Memcached. :-)

My problem was people using ORM frameworks and having very little knowledge of the database they were using. Slow as molasses, difficult to figure out what is going on because of many layers of extra "stuff".

Same boat. Basically every day.

It’s insane how often people stick Memchaced/Redis in front of MySQL/Postgres, completely unnecessarily. So many otherwise decent developers assume that MySQL/Postgres is too slow/doesn’t scale, without even trying. Or similarly, how frequently people shard databases unnecessarily. A single MySQL/Postgres server, on a beefy machine, can handle an absolutely massive amount of work, with great performance, assuming you’ve indexed well and don’t have to run super crazy queries.

I’ve worked on some pretty high traffic systems, with pretty large data volumes, and very, very rarely have I actually needed to cache simple DB reads. And it’s hard to properly cache complex ones anyways, because cache invalidation is so hard for complex queries. Likewise, it’s very rare that I’ve needed sharding. SOMETIMES you need these things, but mostly people are adding a lot of extra complexity and cost for no good reason.

I don’t think people realize that databases themselves have a caching layer internally. Redis isn’t magical, you still have to send network packets to the other server.

Even when reading from disk via mmap, hot pages are in memory.

Sometimes postgresql is faster than redis because all it needs to do is read something from memory and spit it out in the right format.

Totally. If the hot pages of your indexes and hot pages of your records mostly fit in memory, DBs mostly read from memory anyways. If they don’t mostly fit in memory, things can slow down a bit, but you’re likely better off vertically scaling your DB (more RAM) than adding a cache.

Looking up a normal sized record by id, with good networking in your data centre, takes ~1-2 ms round trip whether you’re reading from Redis/Memcached or MySQL/Postgres, and either can handle massive read load if sized properly. The cache just ~doubles your costs, is one more thing to patch/maintain, and introduces new sorts of bugs/outages.

I ran into this early in my career when a senior engineer at a startup reviewed my Python PR for operating on a CSV dataset and insisted I rewrite it in Pandas for performance. It was a very simple program with naive Python (a handful of lines), but the Pandas version was far longer and more complex (I had to have the senior engineer help because it took some advanced Pandas-fu, and he spent nearly a full day on it), and it ultimately ended up being an order of magnitude slower because we ultimately had to call back into Python for each cell.

I actually don’t think “premature optimization” is all that bad in the general case, but you have to be educated about it (and I’ve met a lot of people in Python shops who think that Pandas or multiprocessing will cure all performance ails), and you should specifically think about how costly is a given optimization going to be to maintain or back out of if you’re wrong about the performance benefits.

In general, I’ve never worked on a Python project where we didn’t have to do weird, inordinately expensive things to work around performance issues (though I’ve never worked on a rudimentary CRUD app either) nor has “just throw Pandas/C/multiprocessing at it” ever adequately addressed our most significant performance bottlenecks (usually the solution looks something like Spark, all to do something that would have been sufficiently performant with naive Java or Go). This might just be my experience working on nontrivial SaaS apps; maybe if you’re just doing straight data science or CRUD apps or workloads that aren’t latency-sensitive (mind you, we struggled to keep per-request performance in the tens of seconds, so I use “latency-sensitive” very loosely), Python/Pandas will be just fine. We also ran into other problems with Python, such as packaging and distribution; notably our lambda functions were routinely too large because the pandas branch of the dependency tree was itself more than half of the permitted artifact size (to work around, we switched to Fargate tasks, which have a much larger size limit but take 30s or minutes to boot up).

"I ran into this early in my career when a senior engineer at a startup reviewed my Python PR for operating on a CSV dataset and insisted I rewrite it in Pandas for performance."

Did a performance problem exist?

If not, then I would never promote that person to senior engineer. A working program should only be re-engineered for performance if it wasn't meeting the performance contract agreed upon when the program was written.

No, the performance problem didn’t exist. The engineer in question was really smart in many other areas, but “engineering” per se wasn’t his strength. Things are fast-and-loose in startup world.

Yes, startups are the wild, wild, west. I spent from 1994-97 in startups. I loved the lack of bureaucracy. But we spent too much time in firefighting mode. Through trial-and-error I found medium-small organizations in the non-profit world my best fit.

Early on I had a senior guy mentoring me on a project involving a tool called PowerBuilder. He chose a design that didn't fit the problem-space well but it fit the "PowerBuilder Best Practices" so he implemented it. The performance was abominable and he should have known it would be: he too was a smart guy. But he had a hard time seeing "big picture" design.

Do you work at my company? Because I have a coworker that is always proposing _exactly_ that solution. No matter what issues the code has, for him its usage of MySQL is always "the worst moment".

Asking for benchmark just gets a repeat of "our worst moment is MySQL and we can solve that with some NoSQL cache".

My dude's just trying to get a promo, I guarantee it.

We all start out this way, don't we? I think what drove it out of me was having to maintain. Okay, you wrote this thing, now you need to change the way a few features work and add some others. Especially effective is if you take turns being on call, getting paged in the middle of the night when your multilayer architecture has a hiccup and now you have to troubleshoot it.

I wish there was a quick remedy for such people. But usually they are new, based on what I said. So they should be Junior Developer, not Architect. Sure, you can suggest things, but the answer is nope.

A lot of people, though, may never get much experience supporting their own architectures, because they swoop in, then swoop out, never staying at a job long enough. Or support gets assigned to another group. One thing I like about Agile is that the team that creates the code is the one that supports it.

Yeah. I got told to put in a caching for a system that made very slow calls through a handful of proxies to the other side of the world and back.

We went with redis for the "PoC". I'd tried explaining that if a query is only made once every 24 hours and the data can't be cached longer than that because it's considered too out of date then a cache is pointless extra complexity.

He wasn't having it though, so he made us build it and demo it to a room full of people. Fortunately the room understood the simple explanation and he listened to them where he wouldn't listen to the dev team, so a few weeks of work was scrapped there and then.

It seems that a subset of developers are somehow convinced that adding complexity is a good thing for performance and should be done first, when it should really be a last resort. Their usual rebuttal is "it's scalable" and other buzzword-laden phrases. I wonder if it's a form of "big data envy" or just regular "architecture astronautism".

Honestly, it's the other way around usually; simplicity is scalable. A stateless service talking to a database is fine. You can scale out the service, and scale up the database. It's even easier when you use a cloud provider, their relational database offerings can scale from a wordpress blog's database to enterprise scale, tens of thousands of transactions / second.

The problem is that it's boring, and there's a lot of developers that create work and complexity to make their own jobs interesting.

No, no. We need all these layers of proxies, caches and various databases to make things more reliable!

This is taken to other extreme many times.

Example: Google Chrome codebase was allocating lot of std::string and also someone used a Set to check membership of single item. [1]

I mean, if you say like this, many people don't even care about algorithm complexity.

Doesn't help that people want to write Python in the monster that is C++.

https://groups.google.com/a/chromium.org/forum/m/#!msg/chrom...

I very much agree that many people use rules like these as excuses to write shitty code.

From the post you linked though:

> Not reserving space in a vector when the size is known

std::vector::reserve() is actually not something you should always use when you are adding a number of elements as it will (typically) grow the vector to exactly what you ask. If your function then gets called in a loop to append to the same vector multiple times you end up with quadradtic run time that is normally avoided by the geometric growth done when you just append without reserving.

>and also someone used a Set to check membership of single item. [1]

Depending on where it was done (fast path or not), this could be just fine.

You don't know when something comes to hot path.

    if (std::set(itr.begin(), itr.end()).count(element)) { _____ }

is it tempting for someone than something like

std::find(itr.begin(), itr.end(), element) != itr.end())

?? I don't know. That said, C++ STL quite undiscoverable.

>You don't know when something comes to hot path.

That's the point of the first advice though...

Wholeheartedly agree. What you're describing is like the ideal mysql use case. Also, even the most basic next steps performance-wise will probably tithe you over well into the TBs of data. The headroom for mysql is quite a bit higher than most people think unless you're operating at facebook scale.

I see this behavior a lot too.

Inexperienced engineers will nitpick about what is often minor “performance optimization”, clearly not seeing the bigger picture. Example, why should we spend precious developer time to rewrite some code using something that is often less readable when it’s called once every 30 seconds?

To the folks who do this: you are better off spending the time making data-driven decisions and optimizing for the big picture. In other words measure first than come up with an optimization that has a large impact on the system. Not this micro-optimization, I-love-to-tickle-myself stuff.

Learn to see the bigger picture.

> a Redis caching layer and a complex custom lookup scheme

Bane of my life. A few gigs ago, they had a global Redis cache, a Redis cache per server, an in-app cache, and then MySQL. Needless to say, there were many, MANY bugs that came down to cache coherency and race conditions between them.

[edit: it was a global Redis, not clustered.]

Did we both work at a certain edtech company? Seems oddly specific and exactly how I remember our monolith being structured

Alas, that wasn't edtech, it was photos related.

No comment on your particular example, but the opposite approach is more common than I'd personally like and also not ideal -- if you're doing anything nontrivial it's probably worth doing a back of the envelope calculation to ensure you have enough disk, memory, and network throughput for your favorite solution to handle the expected load.

> Can't we just let stuff work? I'm no fan of MySQL, but can't we admit that a ubiquitous and battle-tested piece of technology, applied to a canonical use case, on tiny data under near-ideal circumstances, is probably going to work just fine?

I call that "separation of error-domains", meaning trying to find interfaces where you can separate off parts of your application, so debugging gets easier:

Is the bug in the database, or in our code?

I feels ya.

My most recent team had our hottest dataset in dynamodb. Because cloud. So much effort to get our web service's P99 <50ms.

The whole dataset fit easily into RAM.

Thru (way too much) effort, I was able to introduce Redis. First shared. Then eventually one mini-instance per EC2, running alongside nginx & nodejs.

> <1GB, moderate read volume, super low write volume MySQL database

It looks like an excellent candidate to put it entirely in RAM and trigger sync on writes only, why on Earth would you need anything else.

That's effectively what MySQL is going to do.

For the most part, yes, but I was thinking about explicitly proloading all tables and indexes into RAM or even using the MEMORY engine, if it makes sense in that particular case.

hehehhe I have actually put some data that do not change more often than once a week hardcoded in the code base and commited in source control... whenever a change needs to happen in the data the CI/CD runs and deploys a new version of the app.

You don't wanna know how that was done before. It is like < 1 GB of JSON as well.

I wish we had a deployment process trivial enough to support that. On the other hand, eventually you want to farm that job out to customer success or an implementation engineer, and then it's not so fun having the data in source control.

Please tell me you have git LFS configured to handle this.

Yup !

Never read the release notes for MySQL. Unless phrases like "server exit" or 'regression' don't bother you.

Reply on news.ycombinator.com