Making Reasonable Use of Computer Resources: Part 2
vfoley.xyz
vfoley.xyz
> And I think we did things that most programmers would be able to do:
> Get a base line for how fast we can zcat a file
> Use perf top to find some bottlenecks that can be addressed
> Look at the data to understand what the common case is
These are reasonable statements to make for people writing software in low level languages. They will frequently use those tools and make those considerations. The original code the author is referring to was written in Python by a person who specializes in data science, using mostly Python (according to their CV). The subsequent port into Rust was done because of claims of it being faster, but I wouldn’t necessarily expect someone primarily working in Python and performing a one-off port into a language they don’t regularly work in, specialized for a space they don’t regularly work in (systems programming) to know these things. It’s always cool to see performance and code length golfing exercises, but the argument that “most programmers would be able to do” the intuited evaluative scoping steps isn’t necessarily fair — not all programmers are working at lower levels of abstraction that warrant being necessarily aware of those tools.
> programmers should try and write their software in a way that makes reasonable use of the resources of the computer.
As programmers, I think we should build tools to help us with that. This kind of article will appeal to people already sensitive to performance, just like talking about software craftsmanship will appeal to people already proud of their job. But if we want to change the industry for the better, we have to enforce that with tools.
Since Rust is used here, we could imagine something like clippy (a linter) but for performance. You could have lots of options, with a few defaults performance levels. For example, at performance level 0 (minimal performance), lines() is fine. At level 1, it could become a warning. At level 2, an error.
The problem I see is that while lines() is easy to fix, the part about Unicode would be harder to do, and that was where most of the performance improvements were found. Maybe that can be fixed at the library level?
The reasoning involved is deeper than what an automatic tool can accomplish. It requires knowledge of computers (disk access speed) and of the input (mostly not unicode). We can imagine inputs for which this optimization does not work and also architectures where it will be less effective. This is the kind of reasoning experienced humans are (still) better than machines.
Not for the part where you have to use something else than lines(). It's ~10% of the time spent in the code. As I said, it's not the biggest improvement, but for something that can easily be automated, 10% is great.
For the part about unicode, I agree that it's harder to find a solution. Maybe the library for JSON could first check if a \u or \U is in the object, and if so decode it as unicode? I may be missing something here, but that doesn't seem too hard. Of course it would be a bit slower than just decode unicode if everything is unicode. But if the majority of JSON is like the one in this article, this would be a sensible default.
For sure, we can't replace humans. That's not the goal. The goal is to offload easy and repetitve tasks to tools, and also to encode tacit knowledge in them.
Of course it's nice to speed it up further, but probably not worth the effort (unless you do it for fun).