* disable lower c-states
* enable adaptive coalesce in the network driver
* mount volumes with noatime
I once had a project where a SAP system was crawling and literally causing company-wide stoppages. We found a job that literally ran every minute of every day and it processed a table that contained a few thousand tasks. This was something that could be done once per hour, and only during business hours. Furthermore, it was re-processing thousands of records each time it ran. In reality, after a record was processed once, it could be deleted from the table.
We emptied out the table, and scheduled the job to run hourly. The whole company noticed an immediate improvement.
This pattern happens a lot. Someone builds a polling system that hits the server once a second in order to see if a task finished. A cron job runs every 5 minutes. All data is processed in a daily job, instead of doing a 24 hour cutoff.
The world is filled with computers doing useless work. Mostly, no one notices.
I ask business how often should something be updated - they say “real time” (not going into details what real time means really) - it is hard to explain processing all data all the time so everything is fresh takes forever….
After couple of months it turns out they never ever open their “super important dashboard” or do it once in 6 months.
Great after a year of bogging down everything I can clean up and make jobs run once a day instead of once a minute because now it is acceptable.
Bonus points for trying to explain you want to implement "eventual consistency" - keep in mind I have added {“real time” (not going into details what real time means really)} in text to indicate that I am not some junior whining around but someone with deep understanding of computing...
We had a task that checked every minute what could be executed.
Instead of doing that, we scheduled the executing code to execute on that time ( if it didn't exist yet) through service bus. Since events can be planned...
Easy peasy and avoided a micro service named "scheduler", a db and an Serverless function... ( Hoboy)
The memory hierarchy is so stark on modern hardware that the 30-year-old adage that "the fastest code is the code you don't run" is maybe less important than, "the fastest code is the code that doesn't spend much time talking to the memory controller."
And it's even worse once we start talking about accessing memory that's on an entirely different computer. Serialization/deserialization, IPC, network calls, and all those other things we do with reckless abandon in modern service-oriented and distributed applications are just unbelievably expensive.
Last year I took a slow heavily parallelized batch job and improved its throughput by 60% by getting rid of both scale-out and multithreading and just taking it all down to a single thread. Everyone expected it to be slower because we were using a small fraction as many CPU cores, but in truth it was faster because the time savings from having fewer memory fences, less data copying, and less network I/O was just that great. And then the performance gains kept coming because, having simplified things to that extent, I was then in a much better position to judiciously re-introduce parallelism in ways ways that weren't so wasteful.
Another common one is just poor query performance from a database. Lack of appropriate indexes, or other relatively easy optimizations.
Similarly, finding a method of caching that's just bad (in-memory database, with sql queries instead of a dictionary). Isn't so bad for one call, very bad when a a given request (login) makes over 200 calls to this cache for configuration settings. It wasn't a problem per request but in aggregate.
In the systems I work on this has been a big one. In SQL people are pretty good about not writing `select *` in production code, but when querying directory servers, redis, mongodb, etc. people get sloppy. When a system is small, it's enticing to pull in lots of data and work with it in code instead of writing real queries. This doesn't scale.
https://planetscale.com/blog/what-is-n-1-query-problem-and-h...
Aside from the performance gains, it's very satisfying to go from 1,000+ inefficient DB queries to 1-2 optimized queries.
"You might expect that many small queries would be fast and one large, complex query will be slow. This is rarely the case. In practice, the opposite is true."
A big part of why it works out that way is that the query planner can only optimize for the query you give it. If you give it a bunch of small queries, it can only make relatively inconsequential micro-optimizations. The one big query gives it a lot more degrees of freedom and opportunities to make big gains.
Here's another great resource for getting more out of relational databases: https://use-the-index-luke.com/
A lot of people don’t know that a database index doesn’t order the actual rows on disk. It’s just a Btree of pointers.
If you use clustered index for a table query pattern, the rows are actually ordered on disk.
Most DBs load data in 8KiB chunks. So if you query 100 rows that are 100bytes, if they’re not sorted, you actually need to load nearly 1MiB of data even tho the query result is 10KiB.
Speeds up joins and range queries 50x or more, less cache evictions, etc.
You can do this in any database except for Postgres. Postgres doesn’t have the ability to keep rows sorted on disk.
Oh and also does compression which helps quite a bit with network storages (like cloud disks)
It's a pity computers are so fast and have so much memory that people can get away with not caring about minimalism.
Another go-to of mine is to take conditionals out of loops that really only need to be checked once. For example:
for foo in whatever:
for bar in foo:
if len(foo) > some_value:
do_something(bar)
for foo in whatever:
if len(foo) > some_value:
for bar in foo:
do_something(bar)
Improve the data(structure) first if possible.
* Look for O(N^X), that is nested loops even when they are not necessarily expressed as loops on the language level.
* If possible, get rid of ORMs in favor of raw SQL. Not because ORMs are very bad but because almost nobody bothers to learn them; they often start causing issues with any non-trivial amount of load.
* Study data access patterns and figure out where and what composite indexes might help. I am saying composite indexes because I assume regular indexes are more or less always there, often even too many of them.
Especially with the last one I have achieved impressive results without any kind of impressive effort, just setting aside some time to understand the code.
* Instead of writing big complex queries that had nested SELECT's, I split them into smaller bite-sized chunks that could be cached * Better caching strategies - reducing how many caches were flushed when a change was made * Tweaking the index's on database tables to improve WHERE clauses * Storing intermediate calculations into the database (for example, the number of posts a user has could be stored on the user table instead of counting them each time)
When I optimized the database, I could then reduce the size of the DB and the server as they no longer needed to work / wait as much
Think deeply about what you're making the computer do, and ask it to do less things by being smarter about what you ask it to do.
I'd say 95% of the time most of your OOM gains will come from the above.
- Delete as much code as possible
```
#include <time.h>
#include <stdio.h>
int main(int argc, char argv[]) { int i = 0; time_t timep;
/*
* ok so now we are printing something
**/
printf("Greetings!\n");
/*
* this is a for loop from 0..9
**/
for(i=0; i<10; i++) {
time(&timep);
localtime(&timep);
}
printf("Godspeed, dear friend!\n");
return 0;```
now, the canonical
```
$> gcc tz-test.c -o obj/tz-test
```
now do this
```
$> unset TZ
$> strace -ff ./obj/tz-test 2>&1 | grep 'local' | wc
10 77 851
$> strace -ff ./obj/tz-test 2>&1 | grep 'local' | wc
1 5 59
moral, always set TZ to avoid localtime(3) from stat'ing /etc/localtime :o)
Remove end-to-end tests. Replace with contract tests and service-level functional tests.
Much faster feedback! At the same time, better coverage. The only serious problem with the approach is that it upsets the magical thinkers in your org. Often those folks are managers.
Also, avoiding code that has pointers to many small pieces everywhere in memory and lead to a bad cache misses score.
Finally, just good old profiling.
1. Establish what is good enough
2. Measure, don't guess
3. Fix the biggest bottleneck first
4. Measure after fixing
And some general things:
5. Avoid micro-benchmarks (i.e. things not at the entire system level)
6. Be careful with synthetic data
7. Know your general estimates (e.g. cache, memory, disk, network speeds)