5,805 karma · joined September 23, 2013
UTF-8 is already a great wire format.
I've never found a "binary JSON" that's significantly better than this; I mean you can beat it, but you need awkward encodings (prefix indices & other weird shit). You end up burning nearly-byte for any particularly clever integer encoding.
Most data structures are just nested arrays of integers. If you need an integer keyed OBJECT you're SOL, but I just play fiddly games with astral plane UTF-8 characters. (Yeah yeah yeah ad hoc encodings are nasty news.)
If you've got a BUTT LOAD of data just fire up a compressing SQLite DB like a normal human.
A buddy of mine works for a large grocer down here (HEB). The way HEB builds is that they get a build contract and a bunch of lawyers, and the GC has to demonstrate that they have enough assets that HEB can go after to make them (HEB) whole. If the GC goes over estimate, then that's on the GC, not on HEB. HEB gets a building or they take the money back.
As an individual, you can "act like HEB" by going with a major builder. Down here that'd be Pulte or someone like that; alternately, there's a lot of high quality factory-built custom homes. I'm trying to convince my wife that we should build a metal building on post-and-beam with a crawl space. We can build a minimum viable house for 25¢/$ compared to a custom, and the quality is straight-up an order-of-magnitude better. The rest of the house can be finished out using off-the-shelf components.
Let's just cut out the middle man.
Everything else, I stole from Bob Nystrom: I keep a local copy of the token's string in the token, aka, `char word[64]`. I try to minimize "decision making" during lexing. Really, at the consumption point we're only interested in an extremely small number of things: (1) does the lexeme start with a letter or a number?; (2) is it whitespace, and is that whitespace a new line?; or, (3) does it look like an operator?
The only place where I've ever considered goto-threading was in keyword identification. However, if your language keeps keywords to ≤ 8 bytes, you can just bake the keywords into `uint64_t`'s and compare against those values. You can do a crapload of 64b compares/ns.
The next level up (parsing) is slow enough to eat & memoize the decision making of the lexer; and, materially, it doesn't complicate the parser. (In fact: there's a lot of decision making that happens in the parser that'd have to be replicated in the lexer, otherwise.)
The result, overall, is you can have a pretty general-purpose lexer that you can reuse for a any old C-ish language, and tune to your heart's content, without needing a custom rewrite, each time.
I guess I'm really confused. It's not like English is some Arabic language where the orthography is in a second nearly unintelligible languages? Or, Chinese or Egyptian hieroglyphs... ?
Also, I want a cookie & a pony.
I have three wild-eyed theories: (1) eukaryotism is an unbelievably exotic step; (2) and/or the moon is required and ultrarare; or, (3) advanced civilizations eschew yellow stars for being inconveniently short-lived: maybe they prefer brown dwarfs, white dwarfs, or black holes for their energy gradient.
In general: use your sneakers; if necessary, push opponents away with with your palms and tightly held fingers.
Modulo the launching of material, scientific needs, etc., there no reason it couldn't've been built by Brunel in Victorian times. Well... scale, as well, I suppose.