C string handling is a mess of viciously surprising APIs, juggling those particular footguns is almost certainly not your least bad option.
There were maybe a few thousand or tens of thousands of emails, and so, I came to look at the issue with my usual attitude which is that if it isn't running instantly on a GHz computer of any kind, something must be horrifically wrong. Not to mention running out of memory.
It turned out that essentially every email was cc'ed to several thousand people; you know, the thread went to a whole company or something. The file size itself wasn't huge in the scheme of things, but our perl script (written by someone much more elevated that me, with a master's degree) read the whole file at once into memory, and expanded the headers to perl hashes, multiplying the size.
But there was no reason the whole thing had to be in memory. Only that people learn to program in CS 101 these days, I guess, as if memory is infinite, because gigabytes might as well be infinity, but if you multiply a certain moderate overhead times tens of thousands by tens of thousands, suddenly all your gigabytes are gone.
Another thing I remember, was when I was given a T-SQL report that typically ran on a few thousand documents, and was asked to run it on all of the millions on all our databases. It was hundreds or thousands of times too slow, but it was running a SQL statement in a procedural loop per document and it could be turned into a single statement.
So my experience has basically taught me that if something is within a factor of two of optimal, it's good enough, but an incredible amount of code, regardless of high or low level, is way worse than that.
But I've never gotten much pay or glory for fixing this sort of thing. Sometimes I daydream about being a high paid consultant who goes and turns three nested loops into two, or two into one.
The customer was ecstatic, but I still think it is 2 orders of magnitude too slow.
Here's my favorite WTF: https://christiantietze.de/posts/2020/01/string-index-offset...
A simple fix in the doc seems to solve the confusion:
"Returns an index that is the specified distance from the given index, unless that distance is beyond a given limiting index [in which case it returns nil]".
It does say "returns an index ... unless ..". So yeah, a bit too terse. But with is the issue with the API?
// /!\ Warning! Do not use in production! /!\
let s = "Swift"
if let i = s.index(s.startIndex, offsetBy: 5, limitedBy: s.endIndex) {
print(s[i])
}
Seriously? 'print(s[nil])' should print nothing? How about 'let c = s[nil]'. Should that just silently pass? That is the runtime error, btw. It won't even get to print(). (Valid to question the entirely non-informative error, however.)
let s = "Swift"
print(s.index(s.startIndex, offsetBy: 4, limitedBy: s.endIndex))
print(s.index(s.startIndex, offsetBy: 5, limitedBy: s.endIndex))
print(s.index(s.startIndex, offsetBy: 6, limitedBy: s.endIndex))
Optional(Swift.String.Index(_rawBits: 262401))
Optional(Swift.String.Index(_rawBits: 327681)) <- this is unexpected
nilThe problem is that it returns non-nil, but the _limit_ is broken in this case: using s.endIndex as a limit means you can get non-nil but bogus indices returned. And yes, this is the fault of the docs for using a broken last arg to the API, but there's no really clean way to use this API as designed, afaict. At least not if you want to limit to "end of string" as opposed to "some index into the string that I already know to be valid".
let s = "Swift"
if !s.isEmpty,
let index = s.index(s.startIndex, offsetBy: 5, limitedBy: s.index(before: s.endIndex)) {
print(s[index])
}
Had it been a char const /size_t or a pair of char const * for first/last it should be both safer and faster. I prefer the later as a pair of pointers doesn't require updating both when iterating with the first ptr.
I am a C programmer who have implemented string to number parsing for this very reason. I know exactly what it does and how fast it is.
If you do use code you didn't write, The chance of a standard library being poorly implemented, is probably lover then most other libraries, so picking a non standard lib as a grantee against bad performance seems misguided.
Basically the only silver bullet for productivity increase is shared code. We just have to place higher emphasis on software correctness/quality instead of the functionality churn in certain parts of the software world.
https://github.com/akkartik/mu
Mu takes the dwm[2] principle of avoid-config-files/modify-sources-to-reconfigure to the limit. The idea is that there are at any given time only 3 languages in the computer:
1. A self-hosted notation for a subset of 32-bit x86 machine code
2. A memory-safe statement-oriented language where most statements map 1:1 to machine-code instructions. Written in level 1 above.
3. A high-level interpreted language.
The vision is to fix 1 and 2 but allow 3 to fork wantonly. If you want a Lisp for your HLL, make a fork. If you want a Python-like syntax, make a fork. But, and this is the important part, in any given computer/repo there is only ever one language. Only one non-trivial parser. (Levels 1 and 2 have extremely uniform syntax.)
As best I can tell, the #1 way to avoid the need to run a fuzzer is to avoid writing parsers. Just say no.
[1] https://lobste.rs/s/to8wpr/configuration_files_are_canary_wa...
What about a program written in 3 that can compile to either 1 or 2? Why would it hurt anything to have a different language somehow made possible to run here?
(Level 2 is not interpreted, it's compiled. Skipping level 2 would be totally in line with my principle above. But it would be more painful, I think. You'd basically be programming in machine code.)
Not trying to dismiss the idea at all, just I don’t yet see “the light”, so to say.
My basic worldview is that we need to have 100-1000x more people reading and auditing open source. The original value of open source was in the ability of people to read the source. If we don't use that ability then we don't really get the benefit.
The world today focuses on APIs and ignores implementation. I would argue that's the biggest source of problems today, with security holes, data breaches, user-hostile UX and so on.
If you accept that being able to read sources matters, hopefully it makes sense why reducing the number of languages matters. Every new language you add is more code to read, another language to learn and become proficient in, a new source of gotchas and subtleties to spend 10 years learning. Another set of tools, another set of moving parts that might not build on your computer because of some subtle version mismatch.
It's a hard problem. So let's make it easier for ourselves by relying on fewer languages, and being more thoughtful about the dependencies we introduce into our projects.
What’s your proposed solution to the problem with low and high level languages? Is the level 3 language a higher level one? Because I’n not sure there could exist a one language to rule them all, because of the inherent difference between the two domains.
You're absolutely right that languages are only part of the problem. Beyond the language choice, Mu provides guardrails to help you pick up the underlying technical knowledge by just trying things and seeing informative error messages (often failing tests) in response. That's the hope, anyway.
Right now the first HLL is still in progress. I spent some time with a postfix-based language before changing my mind. Now I'm working on a Lisp-based HLL. So I'm not dogmatic about what the HLL should be, and in time there will probably be multiple options in separate forks/repos.
Rather use buffers with known lenghts, and for strings you need to known the string (=unicode) rules. Nobody does that. Know libunistring. I have my own, because libunistring is too slow, but know it.
For my string libraries I rather follow the STL, with ranges/views and boehmgc core. const vs dynamic strings. So I will not step into the accidental strlen and buffer-overflow trap.
E.g. For input buffers know if they are zero-terminated and const. With the GTA post I pointed out the libfuzzer design flaw, giving you an ASCII input buffer which is not zero-terminated. Even strtol/strtod cannot be used then. You need to copy the buffer, terminate it, and then you can use the broken string libc. Not talking about sscanf, which I usually use only as sscanf_s if available. Or _snscanf/_snscanf_s. Microsoft does many things wrong, but its libc is far superior to glibc, bsd or musl. musl is better than glibc, but also lacks in this regard.
* https://news.ycombinator.com/item?id=26302744
"Use a library" is observably not a panacea in practice.
This is a sign that writing (trivial) parsers is a core competency for you. However, that doesn't meant that writing all parsers is your core competency. Especially not for an industry standard like JSON that should have plenty of libraries that take care of the problem.
Writing parsers is the same level of complexity as interview FAANG level questions. Any decent developer should be able to write a parser from scratch.