Laws of Performant Software
tagide.com
tagide.com
Yes, and then 10 months debugging edge cases where communicating parts are looking at different versions of the "same" data.
Caching is really important, but caching (and cache-invalidation) is really difficult, adding caching to an application that doesn't use caching is not "10 LOC and done".
That's an urban legend.
Needlessly complicated or over-abstracted general-purpose caching frameworks are difficult, but your dumbest imaginable linear LRU fast lookup is both exceptionally useful and can indeed be done in 10 LoC in a lot of cases.
This isn't an "urban legend" it's first hand experience working with companies trying to add caching. No, those companies aren't even trying to write caching algorithms, they're just bundling in a caching layer and hoping that the system behaves in the same way.
It only takes somewhere which writes data (perhaps in a way that bypasses the caching layer so the cache doesn't know it has changed) and re-reads it back quickly for software which used to work suddenly breaks.
Now you might look at that and go "omg refactor it! That's horrible code, that should never ship" etc, but not everywhere is the s.v. bubble with endless amounts of the best developers to throw at problems. Code which worked and solved a business problem ended up shipping, possibly without testers and probably without code reviews.
So adding a caching layer suddenly "breaks" those reports, now who's going to have to fix it, not the person who wrote those reports even if the very behavior of side-effected data changes and db re-reads is precisely a cause of data layer slowness that led to wanting to implement caching...
So those 10 lines need to be in the wrong place?
Why expose the uncached API?
Web or single system implement caching in a defined is easy if you don't have a defined API you probably don't have a good system. If you don't have a good system, why are you trying to implement caching? The not being good part is probably why its slow.
Bundling a caching layer without even trying to write caching algorithms -- particularly if such a layer is serving multiple purposes -- sounds to me like what the parent is calling a "general-purpose caching framework", probably overly abstracted too as these things are wont to be.
I would be super cautious incorporating this kind of black-box stuff. Even if the docs have 10 LOC examples, there can be all kinds of unexpected quirks that you would need to be aware of before doing anything.
I read the OP as talking about specific, single-purpose caching techniques -- e.g. when you need to repeatedly compute a function of arbitrary parameters, it can help a lot to simply store values for the more common parameter combinations.
PS. It is an urban legend, because "cache invalidation is hard" gets repeated a lot, initially as a joke, but it doesn't preclude people who aren't familiar with the subject from taking it as a fact and then repeating it as such.
Voice recognition from scratch is hard. Some lock-free data structures are hard. Caching is not hard. It's knowing what the heck you are actually doing and doing it well is what's hard. By the same measure, C macros would be hard, because some idiot can do #define true false and everyone else will spend the same 10 months trying to understand why the hell things break now and then. Caching is hard is when someone starts messing with other people' code without fully understanding it. But then anything is "hard" under these circumstances.
Sure, if you're implementing Fibonacci, memoization is simple. But if you're trying to memoize the results out of a database, things are going to be a lot more complicated, really fast.
It requires knowledge of what can go wrong, what will go wrong, the use cases associated with a bit of data to be cached, how the application will be deployed, what other caching will be implemented across the system, and a dozen other bits of information unique to each use-case.
Knowing what questions to ask, and of whom to ask them, requires experience.
That sort of goes without saying. If doing something well is hard, then doing that thing is hard. No one is interesting in writing code that doesn't work -- we're always talking about "doing it well".
We don't all going around claiming that quantum mechanics is easy and then backing it up by demonstrating our ability to pull grossly wrong answers out of our asses.
I'm going to borrow from someone much more eloquent than I: How simple it is to declare a static hashtable, and yet how perilous!
ROFL. What are you talking about?
You're talking as if it's a solved problem for all cases. Hint: it's not.
If it was an urban legend people wouldn't write dissertations on it.
In my experience, I just don't hit that many pure functions that don't do things like touch the database (which can change out from under the function), or are called enough with the same arguments that memoization is actually worth it.
But when it does work, it's like magic. I do a lot of work in javascript now, and it's great being able to wrap a function in a single line `memoize` function and instantly improve performance.
Implementing a cache may be easy but debugging a cache, or particularly the interactions of multiple caches (some outside of your control) certainly isn't. I've encountered these problems on many projects and also written about them in detail for my recent book.
Granted, if you aren't taking advantage of immutable data in the first place, it can hurt to get there.
Caching has it's place, but more often than not I see it used as a bandaid on a terrible design.
to me is kinda like you can easily make heavier elements by just adding a few electrons, protons and neutrons...
* Make it Work
* Make it Right
* Make it Fast
"If you really need *high performance*, you need to design for high performance, not just leave it as an afterthought."
It's not simply that they failed to plan for performance, it's that after making it work -they never measured and removed obvious bottlenecks-.
Most pieces of software don't need high performance. Drawing some widgets on screen just isn't that demanding. But it -does- require you to go back afterwards and remove places you introduced inefficiencies. You don't need to code in C and optimize against cache misses for that; you just need to take time after things work to make them not suck, and thats something a lot of software development doesn't take the time to do.
I think its perfectly normal to write slow, non-scalable code as proof of concept. Then continue to attack bottlenecks as you grow (if you grow). Its a lucky startup that has to deal with performance. They can afford to dedicate a couple smart developers just to that issue.
Michael Dell said every time your company doubles in size, you have to reinvent your processes. True for software too.
That said, I'd draw a distinction between vertical scaling and horizontal scaling. The former you should address as needed, as the gains are comparatively limited, and the bottlenecks are unknown (you think you're CPU bound; whoops, nope, I/O. Or whatever); the latter should be designed for if there's a chance you'll need it. Because oftentimes, things that are merely decisions early on (no difference in amount of work) can lead to savings of months of effort and churn down the line if you go for something that scales. Decisions like deciding what data needs to have strong consistency, versus what data can be eventually consistent (and choosing data stores based on that), trying to avoid shared state, thinking about "what happens if there are more than one of these?" and designing/implementing with that in mind (even if some aspects are super hard and you punt on them, there's plenty of low hanging fruit that you can address with minimal effort early, rather than massive later on).
It's not something you can just do later, because thats not how software works. Fast code needs designed to be fast. Not "fixed" at the end.
I've also seen plenty of performance issues ironically caused by performance hacks wedged in early on.
-- sorry for the run on sentence.
Now this does break down, if you need to get to the point where you are bit twiddling, it is not going to be clean as using something higher level.. but you can often put the nasty parts in a static method somewhere and still have the code be very easy to read.
Know your performance goals going in and code accordingly. If you require 100 micro average latency and you coded in Node.js, step 3 will be a rewrite.
Every single line of code I write, I can tell you my performance goals. If indeed it is a simple crud screen by a user, the goal may be "meh, document.ready called when viewed from 100ms browser lag within 1 second". Backend trading code would have different goals..
Asking stakeholders is good. Often then will have no idea, or say something generic like "make it fast".
Try to pin them to something more concrete.
"So how about adding this new feature makes X happen no more than 5% slower than it currently does"
Having this conversation with stakeholders often educates them on the costs of performance as well. Getting 100% of responses sub 200ms is frequently orders of magnitude more expensive than getting 99% of them there, and stakeholders usually get that fast when you show them budget info.
E.g. (99% sub-200ms and 1% _unbounded_) vs (80% sub-200ms and _always_ sub-500ms) means 1% of potentially unanticipated crashes (a hell to debug and explain to customers!) vs a highly reliable system and happy customers.
If you are going to have timeouts with logic, that has down stream implications. If you are going to have truly independent event loops, that is a fundamental architectural decisions.
None of those things match the "make it work, then make it fast". You literally have to design that into the system from jump street as it is part of the definition of "works".
"Know your performance goals going in and code accordingly."
Performance is a first class design constraint just like development time, budget & functionality, if you don't treat it as such from the beginning you are asking for trouble.
So one could say that within a project, 80% of the code isn't performance critical.
But if you don't know what they are it is very hard to back into acceptable performance.
The point you were disagreeing with here was "80% of the software an engineer will write will not require optimization". ie, 80% of any given system hitting the performance requirements naively. So... why are you still arguing?
Generally speaking, I have not found it to be true that you can make something fast if you didn't think about performance first. If you thought about it, and came to the decision "it will be fast enough no matter what we do" bully for you, but for me that happens way less than 80% of the time.
No that is not true at all.
I am writing a crud app to be used by 1 person, some manager of a widget factory. I say "my perf goal is to have page loads in 10 seconds or less". How will that make my code more complex? If anything it will make my code MORE simple, as I can relax all kinds of constraints like "making 72 database queries per page is generally bad".
Once I've answered "yes" to all those questions then I can think about heading back and trying to make it fast. Writing really high performance code that doesn't solve the problem you have or give you the results that you need is, of course, a waste of time.
You'd never say that the you've made the code "work" if you knew it would take 3x of your budget to get there. Similarly with performance, it doesn't "work" if it doesn't meet the perf goals of the project, no matter how relaxed they might be.
Generally speaking I find it much easier to take slow, working code and making fast, as opposed to fast, broken code and making it work.
Backing into acceptable performance after the fact just doesn't work for the problem domains I work in (which on first blush have not been exclusively performance based).
You still mostly want to follow the rough priority above. You absolutely may prototype to convince yourself a performance goal can be met early on, but if the goal is high performance code you are still far better off making the first pass for correctness. Skipping this step often leads to highly performant code that is wrong, and is a pain in the ass to debug.
The "make it work, then make it right, then make it fast" mantra is both overly simplistic and deeply true.
If the engineer knows little about the problem space, then any type of optimization is not warranted until a working version is achieved.
The biggest secret to writing software is "not painting yourself in the corner".
All to often a 'prototype' becomes the finished product without the intervening iterations. If that was the plan from the start it works more smoothly.
- Fred Brooks, "Mythical Man Month"
That's part of "making it work." If you're writing backend trading code, then performance is among the first consideration.
Performance should always be in your mind somewhere, but it doesn't always need to be at the forefront.
If you ignore performance completely until late in the project, you can paint yourself into a corner. This includes cases like knowing that performance is 50x slower than will be acceptable throughout development, but saying "we can add X later for an easy performance win". If you don't actually test that X gets you within reach of your performance target at an early stage, you can end up with a fully built system that is unusable.
On the one hand I understand that most programs don't need to be specially fast. But on the other it leads to such of waste of time for the users. Where it really matters we usually see some kind of rewrite or new program that is designed to be fast and it can take significant slice of the pie. At the same time there is also a case for ease of use - ease of use can make even slow programs not only feel fast, but also take shorter time from decision to install/run said program to achieving user's goal.
There is no silver bullet, but few (or many?) rules of thumb ;)
You don't want to code yourself into a corner by accident. It's fine to knowingly take on technical debt, as long as you have a plan to fix it in the future. Even if this is never required.
I always try to take a pragmatic approach, as things are usually not as binary as these simple rules of thumb assume. The real world is typically very nuanced, which is why engineering can sometimes seem like more of an art than a science, and why experience is so valuable.
Shameless plug - I hope this attitude comes across in my book that focuses on web application performance issues: "ASP.NET Core 1.0 High Performance" (https://unop.uk/book/).
For instance, if you're working in a resource constrained environment, garbage collected vs not garbage collected is a big decision. Or if you're working on a web app, how you layout your database tables or your nosql equivalents is going to have a huge effect, and is much harder to change later.
There's a huge difference between premature optimization vs making decisions that will have performance consequences down the road. If you wait till the end of a project/release cycle to think about performance, the amount you can do about it will usually be disappointing.
If we were talking about cars, then I would say performance includes not only speed but all kinds of things to do with handling. But in computing it always seems to mean "speed"[1], and "performant" always seems to be a bizarre neologism for "fast".
I think people invented it because speed is a many faceted thing. There is latency, there is thoughput, there is user-visible responsiveness, all at multiple interacting levels. But these complexities and vaguenesses apply equally to the quasi-word "performant".
[1] Actually the main article is a partial exception as she inconsistently includes stability within "performance". In one sentence she says "...performance degradation, including crashes, and the unbounded use of resources." But later she says "Code that doesn’t perform, or that crashes."
performant == relative (efficient on the hardware available).
e.g. an algorithm may be performant on a little Cortex-M but certainly is not fast compared to the same code on an I7.
17km/hour is an absolute speed. It is fast for a runner, OK for a cyclist and slow for a car.
"Fast" is relative.
but in the context the original sentence was used, the statement stands.
English is not the most precise language....
- Abstractions are the enemy of performance. You can't get high performance through many layers of abstraction. This relates to my previous point.
- Algorithmic complexity matters when dealing with large data sets. Abstractions can hide the true complexity. (e.g. the famous string append example from IIRC Joel Spolsky).
So the key to high performance software is:
- As close as possible to the hardware.
- The right data structures/layout (taking into account the hardware).
- The right algorithms.
- Measure properly and optimize.
This depends on the kind of abstraction. Layers and layers of crap will never be fast. The correct stack of abstractions, which map well to your problem space, will usually make your code faster because they will make it easier for you to see higher-level optimizations.
> Algorithmic complexity matters when dealing with large data sets. Abstractions can hide the true complexity. (e.g. the famous string append example from IIRC Joel Spolsky).
Very true. Abstractions are a tool to help understand the problem space, not an excuse to avoid understanding it.
> Measure properly and optimize.
A+. If you can't measure it, you can't control it. It boggles me how many people don't get this. "I replaced all the floating point math with integer math because integer math is faster." Really? IS IT? How do you KNOW? Did you benchmark it? "No but..." ahem sorry, I've had this argument too many times. :P
I like to say "abstractions + compilers = performance".
I got involved in a huge discussion about optimizing a web app recently. Things that were learned from the conversation?
1) It turns out moving more frequent cases to the top of an 'if-else' chain in JavaScript offers a greater speedup than I expected.
2) It doesn't matter if you shave 4ms off a request by optimizing your if-else chain if a little later in your code you make 27,000 database queries when you could have made 2.
Knowing how to write a performant app will always be better than writing sloppy code in the fastest means possible.
> yes your bottleneck may not be CPU, but if it is, then language matters a lot
Even if your bottleneck is the CPU, it does not mean it's the programming language. A quicksort in python is faster than a "while not sorted randomly shuffle this list" in C++.
Sure, moving to another language may be faster. But that is beside the point. The article even says "The programming language doesn’t matter as much as the programmers’ awareness about the implementation of that language and its libraries."
So this whole discussion is under the assumption that a developer does not know what it takes to make an app fast. If you're using arrays in operations where you have to insert into the middle a lot, and never have to iterate over all of it sequentially, you probably should be using something like a linked list. Choosing the proper data structure (and learning why) should take priority over the faster language.
Truly optimized C code will be faster than perl/python/php. But given the same problem the perl/pyhton/php programs will tend to use hashed data types (dictionaries in python, arrays is php) and end up with fast code. Sometimes faster than C code because the writing/using hashing in C is more difficult and its not always used.
http://weblogs.asp.net/jongalloway/performant-isn-t-a-word
This matters. Software can perform well at its job (not crash, get the correct answer) but may not perform efficiently (takes a long time, has unbounded resource use, uses a brute force pattern).
To be fair one can also find a fair number of discussions about whether performant is a word in that google search, so "performant as an adjective" is clearly a newish use of the word.
Edit: after reading your reference I can comment that journal/book editors mostly do not subscribe to descriptive linguistics and it is probably good that they do not (to not take chances with parts of language that might turn out to be fads).
If you are a linguist: i.e a social scientist who job it is to explain how languages work in this world. Then you must describe.
If you are a language teacher then your job is to prescribe. And you make a good point that journal editors have good reasons to do the same. (Especially when looking after authors from STEM fields!)
It's imprecise: What is it even supposed to mean? Because it's a made-up word, who knows for sure? Does it mean "better performing?" If so, why not just say that? It's not that many more keystrokes. Better performing in what way? CPU? Resource utilization? Say so. You've put a lot of thought and effort into writing something, why blow it by using an imprecise pseudo-word? The author is undermining his own credibility, telling us "I don't care enough about the topic to even pick an actual word, let alone summarize, in more detail, what I mean to discuss."
My understanding is that performant is totally legal French, meaning efficient or effective, with usage dating back at least 4 decades. If you're not into stealing random words from other languages, I have to question why you're into English in the first place.
It has been used as a noun, but rarely. You are thinking of the -ant formation seen in informant: one who informs. But this is not the only -ant in English, and it is not the one used here.
That which is resistant, resists well; it offers a good amount of resistance. Those who are insistent, insist strongly; they make plenty of insistence. That which is compliant, complies fully; it is very much in compliance.
That which is performant, performs well; it offers (a) good performance.
As to what good performance is:
> Software can perform well at its job (not crash, get the correct answer) but may not perform efficiently (takes a long time, has unbounded resource use, uses a brute force pattern).
This is a reasonable objection—I agree with it, wanting speech to be plain—but here is something to oppose it: I suspect that just about everyone who clicked on this article, including the two of us, knew precisely what the author wanted the word to mean, even if they had an objection to that use of the word. Performant has sprung to life, and it describes an efficient performance.
As for what I think of the word: the English language is already rich with others which would do just as well, which is probably why this one seems so jargony. It is one of those technical words that sounds more like a social signal—"I know what I'm talking about"—than something precise: "This is what I'm talking about."
Optimization is important so far as you need it. If I can launch a bunch of aws instances to get a single-run job done in an hour, then I'll throw hardware at the problem instead of worrying about my code. I care about the analysis results, not necessarily how performant my method is.
If I'll need to run the analysis multiple times or I plan to publish it as a tool, then thats another story.
Database code isn't hard! An HBM file is just as complicated as handling a data reader! Stored procedures (or in-line SQL) are simpler in the long run!
I find controlled experiments / microbenchmarks to be a useful method for finding the initial bottlenecks of a system. Once I know the initial bottlenecks I can make an informed descision on whether that performance is sufficient.
Controlled experiments / microbenchmarks are also essential in establishing a ballpark number for the theoretical maximum performance of your system. From that point on, the performance is yours to lose.
No. It's critical that people question “why do I have to rely on magic?” in the first place. Even in high-level languages, perhaps especially in high-level languages, it's a good idea to write straightforward code.
> there is an appalling lack of candy in the C/C++ ecosystem (...) performance isn’t hidden
C and C++ are very different languages, and there is no shortage of C++ libraries full of incomprehensible magic. When debugging template-heavy C++ code, it can be very hard to determine where expensive and unnecessary object copies are created. Thank you C++, for making copy construction implicit!
> Are you producing too much garbage unnecessarily?
The high-level programmer's best defense against creating too much garbage is programming with values (whose redundant representations in memory can be automatically deduplicated by the runtime system) rather than object identities.
> Are your dictionaries too big to the point of being inefficient?
The problem isn't the dictionary abstraction, but rather the implementor's choice of underlying data structure. If you have a really big dictionary whose keys are strings (a common use case), you want tries, not hashtables.
> string concats can be replaced by string builders in the same amount of lines,
Using string builders is a low-level chore, and defeats the point to using a high-level language. A better alternative is using a persistent list/string data structure that actually supports efficient concatenation.
> Does your program start ad-hoc threads? Use a threadpool with fixed size.
Again, too low-level. A programmer working with a high-level language should be able to spawn as many green threads as she wants to, and let the language's runtime system handle multiplexing those green threads over OS threads.
> Unless you are 100% sure the lines are always of reasonable size, do not use readline.
This is terrible advice. If readline is causing you problems, you are using the wrong string data structure.
---
Okay, I'm done bashing what could be bashed. The last two items in the OP are actually good.
Do this in Go and you'll be surprised how quickly you run out of memory.
Pool implies pre-allocated.
The problem with creating infinitely many threads isn't even space. Even if you had an infinite amount of memory, programs are supposed to complete their tasks in a finite amount of time, so you shouldn't spawn infinitely many threads, because it's an unreasonable thing to want. On the other hand, spawning 100k green threads is a perfectly reasonable thing to want. It's the language runtime's job to multiplex these 100k green threads over 4 or 8 or however many OS threads make sense.
- https://github.com/petkaantonov/bluebird/wiki/Optimization-k...
- https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...
I.e. a ringbuffer. The Disruptor is an efficient and simple way to coordinate this. https://lmax-exchange.github.io/disruptor/
Little bit of background here before I begin. I did my comp-sci and mathematics degree and even though they have been useful in some degree or another they both were mostly a complete waste of time in my experience. For the vast majority of the clients and projects that I've had since finishing my university they've all revolved around fixing framework problems or bugs within large code bases. Just today I fixed a massive Land Titling (Enterprise Java/Oracle) bug where the original underlying framework would leak connections/memory until it fell over after 3-4 hours of usage. Other bugs such as off by 1 problems, data translation, data encoding. The vast majority of the time its been incorrectly designed frameworks or replacing frameworks with another framework.
For most of the work out there (I would guess 90% of it) its maintaining and supporting clients to achieve their business goals. Completely un-interesting stuff but get a good reputation of getting stuff done and they don't even wink at your asking price.
Do simple data structures where you can. Arrays, AoA, SoA, AoS. Process data in bulk where you can. Put complex algorithms only where needed (kd-tree when doing something like raytracing, threading only when necessary and even then only with bulk processing, complex locking or timing mechanisms only when you have to, etc.). Don't follow paradigms blindly.
Basically data should be processed only in a functional way or in "waves". When there are only a handful of variables, a functional way will keep them in cpu registers. When there is more data, going over an array will make the cpu load the data that is to come next into the cache.
Oh, and write C.
[*] gens is a hobby programmer and possibly an idiot
http://synisma.neocities.org/perf_scale_cheatsheet.pdf
welcome feedback and ideas for things to add or change
improving efficiency implies doing less work for the same end-goal. thus, when a program is 'efficient' then it is doing the minimum amount of work that the computation demands. or in other words, we have the best algorithm around for some kind of complexity argument for the task at hand. an algorithm which is efficient is not wasting anything.
performance on the other hand, implies, how quickly the work that is to be done is actually done. basically performance improvement would allow you to do the same amount of work faster (in time).
is the program at any given point in time (during its execution) doing maximal speed of work ? that doesn't seem to make much sense.
infact, in practical terms, what is the 'maximal speed' of work ? <theoretical constructs like Bremermann's Limit doesn't count :)>
programs can perform 'well enough', but that doesn't mean we are all done with it...
The performance of a classifier can be anything from how fast it runs, to its F1 classification score.
but classification performance would be the efficiency of a classifier algorithm right ? i.e. can i get better classifications / generalizations etc. in lesser time.
Classifier A gets F1 score of 0.93 on test set has a lower performance than classifier B, which gets an F1 score of 0.94, regardless of time.
However, you can also say that classifier C, which only achieves an F1 score of 0.929 has better performance, since it is four times as fast.
Performance is the same. Everything performs, you can only talk about higher or lower performance of one system relative to another. Just calling something "performant" implies that it is "high performance", but leaves out what it's performing better than.
I hate that word. lol
( except not: everything has a charge; we assume some zero level charge and we can say that everything has a voltage relative to that level. That's why there are two words: charge and voltage.)