The computers are fast, but you don't know it

At one of my first jobs I was a DBA supporting a CRUD app in the finance industry. The app had one report that took forever and usually timed out, I was told to take a look at it. The DB query was just missing a couple indexes so I added those.

After I added them, my boss told one of the users of the app to try out the report and she said it was still broken. He asked what she meant and she said she clicked the button and the page with the results came up right away. She thought it was broken because it didn't take forever.

I blame all the ads, tracking, and bloatware that is prevalent now most of all.

The latency on multiplication (register input to register output) is 5-clock ticks, and many computers are 4GHz or 5GHz these days.

5-clock cycles at 5GHz is 1ns, which is 30-centimeters of light travel.

If we include L1 cache read and L1 cache write, IIRC its 4 clock cycles for read + 4 more for the write. So 13 clock ticks, which is almost 70 centimeters.

------------

DDR4 read and L1 cache write will add 50 nanoseconds (~250 cycles) of delay, and we're up to 13 meters.

And now you know why cache exists, otherwise computers will be waiting on DDR4 RAM all day, rather than doing work.

So the processor in my hand can compute a multiplication fast than light can cross the room?

Thank you so so much. It's insane how it feels like the speed of much of our software hasn't improved or even regressed despite the gigantic advancements made over the years. People really don't seem to care about this.

I had an argument about it with a senior colleague regarding some industry software. He figured it wasn't worthwhile to improve the speed of some table fetching and calculations that people actually had to wait on since it would only amount to a bit more than a second or so on top of the regular slowness of it all.

A second that had been multiplied on at least 20 pc's each going trough it at least a 100 times a day of more than 260 times each year over at least 10 years so far. Turns out more than 5 million seconds is a lot of man-hours which whilst cheaper than ours amount to manyfold what it would have taken to fix it.

It's much easier to reason about state updates if all you have is pure functions. It allows you avoid very annoying and hard to catch bugs. I've seen this personally, when replacing a spaghetti component with a straightforward `useReducer` hook.

Unfortunately, we don't really have a performant way to express this pattern in JS (or even in other languages?). You could use something like elm-lang, but it's not as widespread.

For the cases when it's not, use memo.

Respectfully, this is pretty wrong. Pandas does vastly more out of the box than numpy. Off the top of my head: I/O from over a dozen of data formats, joins/merges, sql queries directly to dataframes, sql-like queries on dataframes, index slicing by time, multi-indexes, much more ergonomic grouping/aggregation functions, ergonomic wrappers around common graphing use-cases, rolling windows.

I'm not even really a power user of it, so there's probably a zillion more things it does that numpy can't out of the box, and I don't wanna spend time writing time and validating if an implementation exists.

Because I needed these operations, I wanted to work with Numpy directly, and didn’t want to write custom implementations each time, I created a library to do it. It also has constructor methods for Python Dicts, any kind of Iterable, CSV, SQL query, pandas DataFrames and Series, or otherwise. As well as destructor methods to generate whatever you need when done. It tries its best to maintain the types you specify, and offers a means to cast as easily as possible. All functions return a single type to allow static type checking. And for performance, there is a “trust me I know what I’m doing” mode for extremely fast access to the data which achieves about a 10x speed up by skipping all data validation steps.

Everything it does outperforms pandas, except for the Joins. It does allow inequality joins and multiple join conditions, but the general solution used isn’t very fast. Anyone reading this who would be interested in improving these component would be welcome to contribute!

https://tafra.readthedocs.io/en/latest/

It gets even better when people start switching between percentages and "percentage points" referring to a measure that's in percentages originally.

Unfortunately, most of those things are easier communicated and harder to get wrong if you try speaking in a more natural way. This is now "twice as fast" or "2.1x faster" is much clearer and can't go past zero :)

Similarly, I think it'd help to switch back from percentages to actual factors (119% = 1.19), and saying "we reduced the time for the computation by 1.19 of original time" would clearly show what's wrong (and saying "by 1.19x" would signal how it's a small reduction, so it's wrong as well).

Finally, I am 94.8% certain people will keep using percentages even where inappropriate, and with too much precision too!

Related anecdote: My blog used to be written using Jekyll with Pygments for syntax highlighting. As the number of posts increased, it got closer and closer. Eventually, it took about 20 seconds to refresh a simple text change in a single blog post.

I eventually decided to just write my own damn blog engine completely from scratch in Dart. Wrote my own template language, build graph, and syntax highlighter. By having a smart build system that knew which pages actually needed to be regenerated based on what data actually changed, I hoped to get very fast incremental rebuilds in the common case where only text inside a single post had changed.

Before I got the incremental rebuild system working, I worked on getting it to just to a full build of the entire blog: every post page, pages, for each tag, date archives, and RSS support. I diffed it against the old blog to ensure it produced the same output.

Once I got that working... I realized I didn't even need to implement incremental rebuilds. It could build the entire blog and every single post from scratch in less than a second.

I don't know how people tolerate slow frameworks and build systems.

Well, it felt slower after the "upgrade". Clicking the start menu and opening something like the Downloads or Documents folder was basically instant before. Now, with Windows 10 and the new SSD there was a noticeable delay when opening and browsing folders.

It really made me wonder how it would be running something like Windows 98 and websites of the past on modern hardware.

I still remember how fast console based computing, an old gameboy or a 90's macintosh would be - click a button and stuff would show up instantly.

There was a tactility present with computers that's gone today.

Today everything feels sluggish - just writing this comment on my $3000 Macbook Pro and i can feel the latency, sometimes there's even small pauses. A little when i write stuff, a lot when i drag windows.

Hopefully the focus on 100hz+ screens in tech in general will put more focus on latency from click to screen print - now when resolution and interface graphics in general are close to biological limits.

What do those tools even do for that long? They can read enough data from the disk to overflow my computer's main memory a few times during it.

Nobody has created a language that is both thousands of times faster than Python and nearly as straightforward to learn and to use. The closest thing I know of might be Julia, but that has its own performance problems and is tied closely to its AI/ML niche. Even within that niche I'm certainly not going to get most data scientists to write their code in C or C++ (or heaven forbid Rust) to solve a performance impediment that they've generally been able to work around.

It's great that you've been able to switch to higher-performance languages, but not everyone can do that easily enough to make it worth doing.

Python allows you to program as if you’re a jazz pianist. You can improvise, iterate and have fun.

And when you found a solution you just refactor it and use numba. Boom, it runs the same speed as a compiled language.

I once wrote one little program that ran in 24 min without numba and ca. 8 seconds with numba.

At least with Chrome's V8, the difference is not that big.

Sure, it loses to C/C++, because it can't vectorize and uses orders of magnitude more memory, but at least in the Computer Language Benchmarks Game it's "just" 2-4x slower.

I remember getting a faster program doing large matrix multiplication in JavaScript than in C with -o1, because V8 figured out that I'm reading from and writing to the same cell, so optimised that out, which gave it an edge, because in both cases the memory bandwidth limited the speed of execution.

As for Electron and the like: half of the reason why they're slow is that document reflows are not minimized, so the underlying view engine works really, really hard to re-render the same thing over and over again.

It's not nearly as visible in web apps, because these in turn are often slowed down by the HTTP connection limit(hardcoded to six in most browsers).

How do you know whether or not speed is a consideration?

Yes, OP delivered impressive efficiency gains. I'm sure he could improve the efficiency even more by dropping into pure Assembly.

But is it worth it?

The prime consideration is not execution speed but maintainability. The further that OP got away from pure Python, the more difficult to maintain the code became. That's a downside.

Now, OP describes an important technique because in the real world, you have a performance budget. Code needs to execute at speeds that return quickly enough to the user, or long execution is financially expensive (i.e. cloud computing resources), etc. But optimizing beyond what the budget requires is wasteful in terms of time needed to do the optimization as well as harmful in terms of negatively impacting future maintainability.

Most of the time it's not that you need a faster language, it's that you need to write faster code. I was working on a problem recently where random.choices was slow but I realized that due to the structure of my problem I could convert it to numpy and get a 100X speedup.

As a fun exercise this year I've been doing Advent of Code 2020 in C, and my god it's crazy how much faster my solutions seem to execute. These are just little toy problems, but even still the speed difference is night and day.

Although, I still find Python much easier to read and maintain, but that may just be I'm more experienced with the language.

If there isn't a compiler in the box (JIT or AOT), I won't be using language XYZ, unless forced by customers.

The only reason I use Python is for UNIX scripting.

This kind of blanket comment that "scripting languages are too slow" makes it sound like you shouldn't use them for anything, but they are perfectly adequate for many tasks. I'm more likely to have network and DB slowdowns than problems with scripting languages.

So you dont need to pretty much ever reinvent or even use a hackerrank algorithm, you need to understand that the database compute instance has a fast cpu and lots of RAM too

I wonder what would be the software engineering landscape today if hardware specs were growing like 10% per year...

Nowadays you need vector operations, you need to utilise GPU, you need to utilise various accelerators. For me it is black magic.

Another perspective on premature opt: When my software tool is used for an hour in the middle of a 20-day data pipeline, most optimization becomes negligible unless it's saving time on the scale of hours. And even then, some of my coworkers just shrug and run the job over the weekend.

I once owned a small business server with a Xeon processor, Linux installed. Just for kicks I wrote a C program that would loop over many thousands of files, read their content, sort in memory, dump into a single output file.

I ran the program and as I ran it, it was done. I kept upping the scope and load but it seems I could throw anything at it and the response time was zero, or something perceived as zero.

Meanwhile, it's 2022 and we can't even have a text editor place a character on screen without noticeable lag.

Shit performance is even ingrained in our culture. When you have a web shop with a "submit order" button, if you'd click it and would instantly say "thanks for your order", people are going to call you. They wonder if the order got through.

Habit is a very powerful force.

Performance is somewhat abstract, as in "just throw more CPUs at it" / it works for me (on my top of the line PC). But people will happily keep on using unergonomic tools just because they've always done so.

I work for a shop that's mainly Windows (but I'm a Linux guy). I won't even get into how annoying the OS is and how unnecessary, since we're mostly using web apps through Chrome. But pretty much all my colleagues have no issue with using VNC for remote administration of computers.

It's so painful, it hurts to see them do it. And for some reason, they absolutely refuse to use RDP (I'm talking about local connections, over a controlled network). And they don't particularly need to see what the user in front of the computer is seeing, they just need to see that some random app starts or something.

I won't even get into Windows Remote Management and controlling those systems from the comfort of their local terminal with 0 lag.

But for some reason, "we've always done it this way" is stronger than the inconvenience through which they have to suffer every day.

If you stick to only doing arithmetic and avoid making lots of small objects, javascript engines are pretty fast (really!). The tricky part with doing performance-sensitive work in JS is that it’s hard to reason about the intricacies of JITs and differences between implementations and sometimes subtle mistakes will dramatically bonk performance, but it’s not impossible to be fast.

People building giant towers of indirection and never bothering to profile them is what slows the code down, not running in JS per se.

JS, like other high-level languages, offers convenient features that encourage authors to focus on code clarity and concision by building abstractions out of abstractions out of abstractions, whereas performance is best with simple for loops working over pre-allocated arrays.

I would like to see all of the actual code he omitted, because I am skeptical how that would happen. It's been a while since I've used pandas for anything, but it should be pretty fast. The only thing I can think is he was maybe trying to run an apply on a column where the function was something doing Python string processing, or possibly the groupby is on something that isn't a categorical variable and needs to be converted on the fly.

Though I will never understand webpages that use more code than you'd reasonably need to implement a performant lisp compiler and build the webpage in that (not that I'm saying that's what they should have done, I just don't understand how they use more code)

Implementations of languages like javascript, ruby - and I would presume python and php - are a lot faster than they used to be.

I think most slowness is architectural.

By the next morning, I'd found it was doing an O(n^2) operation that, while probably sensible when the app had first been released, was now totally unnecessary and which I could safely remove. That alone reduced the 20 minutes to 200 milliseconds.

(And this is despite that coworker repeatedly emphasising the importance of making the phone battery last as long as possible).

It's a ~community language without the backing of an 800lb gorilla to offer up both financial and cheerleading support.

I love the idea of Nim, but it is in a real chicken-and-egg problem where it is hard for me to dedicate time to a language I fear will never reach a critical mass.

https://appleinsider.com/articles/22/06/11/stage-manager-for...

  "The function looks something like this:"

Then he says:

  "I replaced Pandas with simple python lists and implemented the algorithm manually to do the group-by and sort."

For all this decreased performance, what new features do we have to show for it? Oh great, I can search my Start menu and my taskbar had a shiny gradient for a decade.

This talk by the creator of micropython [0] gives his reasoning for why to implement python on microcontrollers despite it being hundreds of times slower than C. Starts @ 3:00

- it has nice features like list comprehension, generators, and good exception handling

- it has a big, friendly, helpful community with lots of online learning resources

- it has a shallow but long learning curve. It's easy to get started as a beginner, but you never get bored of the language, there's always more advanced features to learn.

- it has native bitwise operations

- has good distinction between ints and floats, and floats are arbitrary precision, you're not restricted to doubles or even long longs. (I'll add that built in complex numbers is a plus)

- compiled language, so it can be optimized to improve performance

[0] https://www.youtube.com/watch?v=EvGhPmPPzko

Emotionally though (once you have the environment set up), it’s just such a breeze to write it. It’s like executable pseudo code with zero boilerplate. You can focus purely on the algorithms and business logic. Compared to many other languages the line count is often 50-80%, even if you include type annotations! This doesn’t only apply to plain imperative code, using the dynamic features you can also turn it into your own DSL where needed.

Then there is obviously the huge eco-system around it, there is not a single service, file format or database that doesn’t have a good python library for it. While go might have equally wide library choices, I wouldn’t be so sure about nim, go on the other hand has a lot of other wtfs even though it provides a lot of good fresh tech.

Would I use it for a big service with potentially lots of performance requirements? No. But there is no doubt why it’s so popular. For many applications where the the outcome of the program is more important than the performance or environment, like glue code, simple intranet applications or exploratory coding, it is still the perfect choice. You also have to consider what it is replacing, often the alternative would be even worse; bash-scripts, Excel or Matlab.

Another way to put it is that it’s a very good Swiss Army knife that is good at everything but not best at anything.

Almost always yes, because software is almost always used many more times than it is written. Even if you doubled your dev time to only get a 5% increase of speed at runtime, that's usually worth it!

(Of course, capitalism is really bad at dealing with externalities and it makes our society that much worse. But that's an argument against capitalism, not an argument against optimization.)

I doubt the author's C++ implementations beat BLAS/LAPACK, but since they're not shown I can only guess.

I've done stuff like this before but the tooling is really no fun, somewhere between 2 and 3 I'd just write it all in C++.

Changing the interface just to get parallelism out seems not great - give it to the user for free if the array is long enough - but maybe it was more reasonable for the non-trivial real problem.

Note that I'm not saying that their second version of the code wasn't faster, just that this has nothing to do with python vs. pandas.

it has thankfully started: https://www.blauer-engel.de/en/productworld/resources-and-en...

I think KDE's Okular has been one of the first certified software :-)

To some extent they can claim to deliver a unique feature where there is no replacement for the algorithm they are using.

I overheard this quote recently: 'I'd rather have today's algorithms on an old computer, than a new computer with old algorithms'

no surprise pandas was "slow"

That is not to say aiming for generally unbranchy code is not a good thing - that often implies well designed code and well chosen data structures anyway

If you provide an end result response from your web app to a user's browser in 50ms-100ms (before external latency) then things like 200 microseconds vs 4 milliseconds have less of a meaningful difference. If your app makes a couple of internal service calls (over HTTP inside of the same Kubernetes cluster) it's not breaking the bank in terms of performance even if you're using "slow" frameworks like Rails and get a few million requests a month.

I'm not defending microservices and using Kubernetes for everything but I could see how people don't end up choosing raw performance over everything. Personally my preference is to keep things as a monolith until you can't and in a lot of cases the time never comes to break it up for a large class of web apps. I also really like the idea of getting performance wins when I can (creating good indexes, caching as needed, going the extra mile to ensure a hot code path is efficient, generally avoiding slow things when I have a hunch it'll be slow, etc.) but I wouldn't choose a different language based only on execution speed for most of the web apps I build.

This is a provocative framing but I'm not sure it makes sense. Functions aren't resources; they don't have throughput or utilization. It would be bad if a core could only call the function 300-600 times per second, but that is why we have async programming models, lightweight threads, etc. So that the core can do other stuff during the waiting-on-IO slices of the timeline. Which, as you mention, dominate.

It would also be bad if a user had to wait on 300-600 sequential RPCs to get back a single request, but like... don't do that. Remote endpoints are not for use in tight loops. There are cases where pathological architectures lead to ridiculous fanout/amplification, but even then we are usually talking about parallel tasks.

There is overhead to doing things remotely vs. locally. But the waiting isn't the interesting part. It's serialization, deserialization, copying, tracking which tasks are waiting, etc. A lot of performance work goes on around these topics! Compact and efficient binary wire protocols, zero-copy network stacks, epoll, green threads, async function coloring schemes, etc. The upshot of this work is also, as is typical in web/enterprise backend world, not so much about the latency of individual requests (those are usually simple) but about the number of concurrent requests/users you can serve from a given hardware footprint. That is normally what we're optimizing for. It's a different set of constraints vs. few but individually expensive computations. So of course the solution space looks different too.

Granted, this is exacerbated when architectures don't make a good division between control/compute/data planes.

Control plane, which is exposed to users, should almost certainly be limited to a single (or handful, at most) microservice calls. Preferably to the fastest storage mechanism that you have, such that what latency it does add is minimized entirely.

Converted the list to band performed a simple binary search to find it.

A basic python script. could handle about 4,000 records a second.

Corporate IT reached out to Oracle. Built a custom solution that cost probably a couple hundred thousand.

They tried to force us to use it. They were a little upset when I asked if they could up the performance by a few thousand percent.

I was on their shit list after that until I had leave.

And then your actual function starts, and returns after roughly 10s.

I think you underestimate just how inefficient enterprise can be. The extra time taken in connections between layers is not even a consideration.

[1] https://en.wikipedia.org/wiki/COMEFROM

Should be the opposite. Overhead as a proportion of total time goes down the more useful work is involved.

Doesn't this mean it's less of a problem? Like, isn't that good?