Our Git Hash Bug
tmendez.dev
tmendez.dev
I wish YAML wasn't so common in Python ecosystem...
Jsonnet, I haven't used personally but I know people who have raved about it.
Ones I know less about include KCL, CUE, and Nickel.
It can be hard to find an implementation of this if you're working in a not-so-popular language (looking at you Swift) but JSONC has had the best developer experience of anything I've tried so far.
It's basically JSON with single and multi line comments and trailing commas.
TOML nesting is indeed a joke. YAML & JSON "look clean", but try to match up nesting in a large document without help from text editor highlights, then see how easy it is in XML.
XML has universal support everywhere, has first-class tooling for everything that's being re-invented in these other languages, but it's not "cool" any more.
On a higher level, the fact that so much different kind of information could exist at each level was nothing but headaches. In YAML, or in JSON, it's pretty straight forward. You have an object, it has children, children have types/values etc.
In XML, you have to keep in mind what the tag of the element, what its attributes are, and then what the child elements are, and then whatever the heck CDATA is.
I think my fellow posters are looking at the past through nostalgic rose tinted glasses. XML was terrible and I am glad it's not used as widely anymore.
Features like schema feel very natural in XML whereas JSON schema feels outer worldly; and is seldomly used
Being able to clearly distinguish between multiple semantically different types of strings is a blessing
And coming back to your example, how is the knowledge in JSON of which attribute is an array, which is an object, which is a string and when order of attributes as well as duplicates matter and when not any easier than XML?
1. No consistency in whether a value is an attribute or a child element, they seem completely interchangeable and redundant. I'm sure there's some nuance I'm missing here but I've worked with my fair share of XML and it has yet to be relevant to me at least. This adds a lot of mostly arbitrary decisions to the design phase, makes it difficult to clearly refer to specific values, and means it's impossible to directly parse XML into an object structure in most languages like you can with JSON.
2. It's awful to read and write. Everything is just a mess of tags and any structure is quickly lost. Line splitting is awkward and often not even attempted (seriously I had coworkers push back when I applied basic formatting to a config file because they claimed it was easier to read with lines 3x the width of anyone's screen). Needing to name the object in both the opening and closing tag wastes space and is absolutely ridiculous when writing by hand (sure editors can kinda handle this now but not perfectly and sometimes I'm using vi over SSH because that's all there is).
I really don't understand why so many people are still so attached to it myself.
The opposite problem exists in Minecraft chat message formatting - they used JSON for markup, when it should have been XML.
XML is too big of a hammer for the space it fills.
[1] https://www.tbray.org/ongoing/When/200x/2003/03/16/XML-Prog
That's kinda weird way to phrase it; being painful to work with has nothing to do with "coolness"... that's a legitimate complaint.
Despite all the tooling and support, XML is painful to work with.
Basically any mainstream configuration syntax.
The really big problem with xml wasn't actually the verbosity of xml itself but the fact that it was popular in a time before rails popularized "convention over configuration"'
It doesn't have schema support however you arguably don't need one, because the software that reads the config specifies the type of keys when the value is read and casting takes place then. If the software expects a string, it reads the value as a string, if it expects a number it's parsed as a number and so on.
HOCON has hierarchical merging, include files, a more convenient syntax, ability to read environment variables, substitutions, comments and a few other convenience features. In Conveyor, a tool for packaging desktop apps I wrote that uses it, it's also extended so you have "hash-bang includes". Those are includes that specify a program to run instead of a file, the output is then included and parsed at that point. This lets you escape from declarative config to a fully dynamic computation if you need to. You can disable this feature with a command line flag if you don't trust the config you're parsing (and also env var substitution).
You can also render the whole config to regular JSON if you need to.
I find that set of features to nicely balance config complexity with read/writeability. The main issue is that the main library is not well maintained, and the best implementation is for the JVM. You could give it a C API these days with Native Image but nobody has.
Main downside vs yaml is that IDEs and editors can use YAML schemas to give auto-completion whereas they don't do that for HOCON.
There's a little tutorial here, with a slider that shows how you can start with json and transform it. The right hand side is valid HOCON at every step:
You need a format that doesn't let you make this mistake in the first place, like [everything except YAML].
try this:
yaml.dump({"gameServerVersion": "556474378"}, sys.stdout)
gameServerVersion: '556474378'
I agree you should not do that, but it is common in the YAML world - if you're using YAML you've already decided you don't care if things are reliable.
Given that they were using string templating, this would have been caught earlier using a different format.
Of course there are other string templating mistakes that other formats would not catch (e.g. forgetting to escape strings), but they are still better than YAML.
Any format would fail if used like that, be it YAML, JSON, XML...
… the first couple of times.
After causing the same bug multiple times its the user at fault, not the tool.
YAML does a decent enough job.
I think these are the relevant YAML spec sections from a quick glance. If someone wants to correct me, feel free.
YAML spec on double quotes
> The double-quoted style is specified by surrounding “"” indicators. This is the only style capable of expressing arbitrary strings, by using “\” escape sequences. This comes at the cost of having to escape the “\” and “"” characters.
YAML spec on plain style
> The plain (unquoted) style has no identifying indicators and provides no form of escaping. It is therefore the most readable, most limited and most context sensitive style. In addition to a restricted character set, a plain scalar must not be empty or contain leading or trailing white space characters.
Times every yaml user in the multiverse.
Just like convincing your boss to switch the backend over to K8s and then realising a year later it was a mistake.
Everyone does it.
Absolutely not. If the tool is so bad that users commonly make a mistake then the tool should prevent that mistake.
This is a basic UX tenet that unfortunately many people do not know.
The accepted safe default is to use double quotes. If folks don’t know that then that’s on them, not the tool. It’s in the spec.
Good workmen don’t blame their tools.
Edit — I know the UX tenet. I’ve worked in places where they thought using YAML for end users was a good idea. It’s not. It never will be. It’s a backend tool for engineers. So I agree with you in that specific UX case.
But this isn’t an end-user UX case. It’s backend platform configuration.
Use the right tool for the job, and use it the way it’s been designed —> Double quote strings in YAML.
Good workmen that are given shoddy tools absolutely blame them.
Good workmen talk shit of particular tools and brands all the time, they just avoid having them in their toolbox. They don’t blame their tools, not all tools.
My first thought was that git hash started with a zero, and the parser stripped it off because it was parsing it as a number and not a string.
But then I saw "infinity" and was like "wait is the git hash all numbers except there's the letter 'e' in there somewhere"?
Ugh.
Always quote your YAML strings!
I just wrote a configuration format writer for an app I maintain, The C libyaml library (ugh, yes, I have to write this in C) lets you specify the quoting style it will emit for strings... always choose some form of quoting.... always.
Also it seems kinda short-sighted for this company to use the short git hash. The short hash is nice for display purposes, but there's always the possibility of a collision (git is smart and will print out a revision that has no current collisions within your repo, but that doesn't mean it won't happen in the future). But for a configuration setting like that, there's really no reason not to use the full hash.
We have a logfile parser that tries to parse a value and fell back to float().
But if you passed it a hex value that happened to only have decimals and 'e', then Python interpreted it a an exponentiated number, which happened to be larger than what it could natively represent, so returned `Infinity` !
Obviously the bug was on our end with the naive `float()`, which after deliberation should not be used in this generic parser case, but it was interesting tracking down this bug from the DB values all the way up to the parser.
I don't think json supports comments, which makes it a non-starter for a lot of folks.
What else would be needed?
However, this could've been caught by a validator. Whatever is loading this config file should know that `gameServerVersion` is a string and when it got a number, it should've thrown an error. It could also further validate that it gets a hex string lest something feed it "Infinity".
#!/usr/bin/env python
import sys
import re
from typing import Annotated
from pydantic import BaseModel, AfterValidator, validator
from pydantic_yaml import parse_yaml_raw_as
def validate_commit_id(s: str) -> str:
assert re.search(r"^[0-9a-f]{9,32}$", s), f"{s} is not a commit id"
return s
CommitId = Annotated[str, AfterValidator(validate_commit_id)]
class Model(BaseModel):
gameServerVersion: CommitId
def main():
text = sys.stdin.read()
conf = parse_yaml_raw_as(Model, text)
print(conf)
if __name__ == "__main__":
main()
$ cat conf.yaml
gameServerVersion: 556474e378
$ python parse.py < conf.yaml
pydantic_core._pydantic_core.ValidationError: 1 validation error for Model
gameServerVersion
Input should be a valid string [type=string_type, input_value=inf, input_type=float]
$ cat conf.yaml
gameServerVersion: "556474e378"
$ python parse.py < conf.yaml
gameServerVersion='556474e378'
$ cat conf.yaml
gameServerVersion: "Infinity"
$ python parse.py < conf.yaml
pydantic_core._pydantic_core.ValidationError: 1 validation error for Model
gameServerVersion
Assertion failed, Infinity is not a commit id [type=assertion_error, input_value='Infinity', input_type=str]
Never store the short hash anywhere. It's good only for immediate commandline use, and nothing else.
In JavaScript, Number.MAX_VALUE (the biggest number that can be represented without returning Infinity) is a 309-digit number, and Git SHA-1 hashes are only 40 characters long.
However, this does mean that the chance will be even smaller than you say. Assuming that all other conditions are met (sole "e", every other character is a digit)
* if the sole "e" is in any place from the 2nd to the 37th, the resulting number is almost guaranteed to be returned as Infinity, unless the number happens to begin with "0e". (Leading zeroes, either in the significand or the exponent, would be required for this to not be a guarantee. Any number beginning "0e", on the other hand, will evaluate to 0 and not become Infinity.)
* if it's in the 37th place, the resulting number (assuming no leading zeroes) has around a 27% chance to be representable without being returned as Infinity.
* if it's in the 38th or 39th place, the resulting number is guaranteed to be small enough that it won't be turned into Infinity.
* if it's in the 1st or 40th place, it probably won't be parsed as a number so will not return an error.
I'm also not interested in doing the full calculations, and in the end this isn't going to make a significant difference to the chances IMO (except in the "0e" case mentioned above, which seems like it'd make a noticeable difference), but it's interesting to think about the edge cases.
With 10 hex digit like on the blog, that'd be 0.9% so about once every 100 builds
With full 40 digit: 2.7e-8 so about once every 100M builds
Assuming my math is correct.
Here's to you, and to whoever else runs into this problem next.
md5('240610708') == md5('QNKCDZO')
https://news.ycombinator.com/item?id=41510252#41510814
https://stackoverflow.com/questions/22140204/why-md524061070...
log_config:
file: '"/var/log/my_custom_service/mcs.log"'It's always a gamble when using a programming language from with a small community. Will this one get big?
But nobody else cares about how yamerl parses YAML for Erlang and lots of tool try to re-normalize YAML as they process it; so if we put e.g.
config:
option: "this_must_be_a_string"
config:
option: this_must_be_a_string
Edit: Now that I think about it, it might have been only digits with a leading zero that was dropped.
Same goes for CLI (shell), same for SQL. All the many bugs with them is because they are not for machines. Proper formats for machines don't have those issues (MsgPack, Protobuf, hell even XML/HTML is better).
I tend to joke that working in the industry made me lose all the trust I ever had in me. I don’t believe any deployed code unless it can be confirmed by two independent sources and a runtime telemetry.
Here, I expected hash collision which is rare, but in my lifetime I’ve seen my share of “it really shouldn’t happen but somehow it happened”. Couple weeks ago I’ve seen author at a random library implementing queue (with ACKs!) for UDP packet transmission.
Truly an amazing career for chaos keepers.
I want an interchange format that lets you declare the types on either side:
foo : f32 = bin : 0x42280000
There you have it. Whoever is using teamcity is doomed to fail like this. Not a YAML problem, but a management problem.
2. Use monotonically increasing variables for versions of software.
I always use s-exprs for problem-free config files but sadly everybody else takes one look and says "but all those parentheses! I'd rather have my strings interpreted as very large numbers, thank you."
But thinking deeper, given that the hash is computed from a small amount of entropy (the commit time), plus a seed (the committed code changes), and the previous value (previous commit hash), this is actually fairly similar to the definition of a PRNG.
That is to say, it's not an RNG, but for some approximations it's indistinguishable from one.
I don't think they are. The use of "RNG" is referring to how a "random" commit sha lead to this bug. Commit SHAs are, for all intents and purposes, random.
You could test this against thousands of SHAs and never encounter one that meets the criteria to trigger the bug.
To get this specific bug - one 'e' in the 7th digit and 9 digits less than 'a' - is 1/16*(10/16)^9 = 0.009095. Just under 1 in 1000. But if the 'e' is in the 2nd,3rd,4th,5th,6th you'd get the same bug, so actually about 6 in 1000.
But around 1% of the hashes will be all decimal digits and will parse as numbers ((10/16)^10 = 0.009). That's common enough that I've seen that one cause errors in our code too (somebody had code reading the hash then trying to append it to a string).
For instance consider this bug:
secretKey = Hexadecimal(Crypto.Rand())[0:16]
The person likely intended to generate a secret_key with 16 bytes of entropy. Instead they generated a secretKey which is 16 bytes long, but only contains 8 bytes of entropy. I would call this a RNG bug.
Or clickbait since the title sounds more interesting than “oh YAML is dumb”.
As a supporting example: the game Super Metroid alternates every frame between checking collisions from left-to-right or right-to-left. The direction of collision checks can make a difference for speedrunning tricks, and there's no (practical) way to control for it, so it's referred to as "RNG". From a player's perspective, the fact that it's just a frame counter rather than an LCG or something is irrelevant -- it's a luck-based factor that's outside of the player's control.
And the same frame counter is used as a source of entropy elsewhere in the game, so there's an argument that it's not even wrong to call it RNG. Similarly, a Git hash is a SHA-1 of the repo contents commit message, commit date, etc., and a cryptographic hash and pseudo-random generator are very similar constructions...so calling it RNG is a little cute but not exactly inaccurate.