creationix

Yep. I did try binary formats first. I tried existing ones like CBOR, I tried making my own like Nibs. The text encoding is an operational concern, not a technical one.

This is the same reason I've been advocating for JSONL at work. It's not ideal technically, but it's a good balance of technically good enough while being also human friendly when things go wrong.

- https://vercel.com/blog/how-we-made-global-routing-faster-wi... - https://vercel.com/blog/scaling-redirects-to-infinity-on-ver...

RX is one step towards less human friendly, but more machine friendly. I try to keep things balanced in my designs.

Exactly. Low heap allocations when reading values is one of the main driving factors in this design!

Serialized just means encoded as a stream of bytes so that it can be transferred between systems. There are absolutely cases where you want to be able to query a value directly like a database instead of parsing the entire thing to memory before you can read it. Think of this as no-sql sqlite.

Thanks. I had this for older versions, but forgot to write it up again for the latest version.

One old version that is meant to be more human readable/writable is jsonito

https://github.com/creationix/jsonito

I'll add similar diagrams and docs for the format itself here.

oh, sorry about that. I forgot to include the description of the format with examples.

I did add some small examples to the repo.

https://github.com/creationix/rx/blob/main/samples/quest-log...

The older, slightly outdated, design spec is in the older rex repo (this format was spun out of the rex project when I realized it's actually a good standalone format)

https://github.com/creationix/rex/blob/main/rexc-bytecode.md

Thanks for testing it out! Yes, the website could use some love to make everything more discoverable.

The primary use case is not compression, it's just a nice side effect of the deduplication. This will never beat something like zstd, brotli, or even gzip.

My production use cases are unique in that I can't afford the CPU to decompress to JSON and then parse to native objects. But with this format, I can use the text as-is with zero preprocessing and as a bonus my datasets are 18x smaller.

It's a gradient. I did design several binary formats first, but for my use cases, this is actually better. There is nuance to various use cases.

> None of the space savings/efficiency of binary

For string heavy datasets, it's nearly the same encoding size as binary. I get 18x smaller sizes compared to JSON for my production datasets. This was originally designed as a binary format years ago (https://github.com/creationix/nibs) and then later after several iterations, converted to text.

> Being able to copy/paste a serialization format is not really a feature i think i would care about

Imagine being paged at 3am because some cache in some remote server got poisoned with a bad value (unrelated to the format itself). You load the value in dashboard, but it's encoded as CBOR or some binary format and so you have to download it in a binary safe way, upload that binary file to some tooling or install a cbor reader to your CLI. But then you realize that you don't have exec access to the k8s pods for security reasons, but do have access to a web-based terminal. Again, to extract a binary value you would need to create a shell, hexdump the file and somehow copy-paste that huge hexdump from the web-based terminal to your local machine, un-hex dump it, and finally load it into some CBOR reader.

A text format, however is as simple as copy-paste the value from the dashboard and paste into some online tool like https://rx.run/ to view the contents.

Exactly. My for use cases, this format is amazing. I have very few floats, but lots and lots of objects, arrays and strings with moderate levels of duplication and substring duplication. My data is produced in a build and then read in thousands or millions of tiny queries that lookup up a single value deep inside the structure.

rx works very well as a kind of embedded database like sqlite, but completely unstructured like JSON.

Also I'm working on an extension that makes it mutable using append-only persistent data structures with a fixed-block caching level that is actually a pretty good database.

if you data is lots and lots of arrays of floats, this is likely not the format for you. Use float arrays.

Also note it stores decimal in a very compact encoding (two varints for base and power of 10)

That said, while this is a text format, it is also technically binary safe and could be extended with a new type tag to contain binary data if desired.

Or in this case, just do `rx file.rx` It has jq like queries built in and supports inputs with either rx or json. Also if you prefer jq, you can do `rx file.rx | jq`

As with most things in engineering, it depends. There are real logistical costs to using binary formats. This format is almost compact as a binary format while still retaining all the nice qualities of being an ASCII friendly encoding (you can embed it anywhere strings are allowed, including copy-paste workflows)

Think of it as a hybrid between JSON, SQLite, and generic compression. This format really excels for use cases where large read-only build artifacts are queried by worker nodes like an embedded database.

- this encodes to ASCII text (unless your strings contain unicode themselves) - that means you can copy-paste it (good luck doing that with compressed JSON or CBOR or SQLite - there is a scale where JSON isn't human readable anymore. I've seen files that are 100+MB of minified JSON all on a single very long line. No human is reading that without using some tooling.

True, the particular use case where this really shines is large datasets where typical usage is to read a tiny part of it. Also there is no reason you couldn't write an rx parser that creates normal mutable objects. It could even be a hybrid one that is lazy parsed till you want to turn it mutable and then does a normal parse to normal objects after that point.

A new random-access JSON alternative from the creator of nvm.sh, luvit.io, and js-git.

Yep, I've got lots of experience in this area. It works very well for read-only use cases and is interesting for read-write like tedit. Also it's amazing for build systems like the one built into tedit. I wish the webdev world in general would move to that model. It would save so much compute people would joke it would slow global warming.

Exactly, you don't know if that's true.

From my personal experience talking to actual mozilla employees and reading blog posts by mozilla employees, that statement looks pretty true. Even LGBT employees supported him through the ordeal. Yes mozilla the company had some problems with it's policies, but from what I've heard directly from insiders, Eich was partnering with the LGBT people to solve their problems and never acted in and discriminatory manner against them. Loosing him hurt their cause and the company.

Please do some research on what actually happened instead of blindly jumping on the popular bandwagon.

Of course there were employees who were unhappy when they learned about his donation. In fact they were quite surprised because of how professionally he did his job.

But I have never heard of any thing he's ever done to act in a discriminatory manner on the job. I have heard of cases where he actually protected such people and helped improve things internally.

Wait, you're yelling "huge false equivalence" and then compare supporting what was a popular political viewpoint (Even Obama and the majority of California voters supported it back when Eich made the donation) to being as objectively wrong as murder?

What kind of self-reinforcing social bubble have you created that makes you think those two are equivalent? You say "crime" and "was clearly guilty" for actions that are explicitly protected under the law. Do you know what crime is?

Why do you constantly assume that he voted to take away rights? If you've been paying attention to what he did do in relation to same-sex couples you'd know what he has been working for years to get them equal rights. But through other means. This is hardly the same man that was portrayed in the attacks.

Your entire argument is based on a false assumption that just because he doesn't think redefining marriage is the only way to get those rights that he wants to take rights away.

There is a base assumption that anyone who disagrees with redefining marriage is unacceptably evil and wants to strip rights from a minority. This is false.

Redefining marriage is not a right, it's a tactic to give same-sex couples the same rights as married couples. Not everyone agrees with this tactic.

And guess what, the redefinition of marriage complete is now across the entire country and these people still don't have equal rights in many areas! Maybe he was on to something?

Don't assume someone is fighting against you just because he doesn't agree with a particular tactic.

"Many religions are going to have to change"

Pause and think about this for a moment.

If you believed that there really was a God in heaven who spoke with you and has revealed to you the purposes of live and the purpose of marriage, a God who created the world, gave you life and offers eternal riches in the next life.

If you really believed this, would you bend to social pressure because political winds have shifted against some of the teachings of your God?

If a church bends to social pressure then it's a fake. Pay close attention to the ones that act like politicians and say whatever makes them popular and look at the ones who stick to their beliefs.

We shouldn't punish people for believing what they believe!

You can try to reason with people and have civil discourse and explain your position. But do it without resorting to straw-man attacks, ad-hominim attacks and blatant false accusations in order to garner the mobs against them.

We shouldn't drive away would-be allies simply because they have a diverse perspective. We should welcome their diversity and utilize their wisdom and experience.

Diversity is a good thing. This is why it's important we don't drive out everyone. This means to protect the rights of the gays and the religious.

I'll be running a workshop on April 8th in San Francisco if anyone wants to learn more about luvit in person. https://www.eventbrite.com/e/questing-with-luvit-a-libuv-asy...

Thanks for posting this. If anyone has questions, I'll try to answer them here, on twitter or in the comments on the blog post.

Actually dukluv is the other way. It's a re-implementation of of a node-like environment using duktape instead of V8. I think the original question here was to write a node addon that allowed you to run arbitrary code in a duktape sandbox.

And thank you! I've wanted this for years, but haven't yet gotten up the courage to actually tackle it. ES5.1 is big and ES6 is massive.

A lot of these one-man projects aren't because nobody else in the world can work on them. It's usually because everyone else is busy working on something else.

If I ended up having to maintain duktape I'm confident I could understand and improve the code with a little rampup time. This isn't rocket science. The hardest problem is finding time and funding. If a company depends on the project they have incentive to provide those things.

It's worlds smaller and easier to work with.

Integrating duktape into my project was literally just including the C file. It added just a couple hundred KB of overhead.

Compare that to SM or V8 that require massive custom build systems and use up to 10mb of ram for a simple repl.

It a couple orders of magnatude slower, but smaller than V8. I love the low overhead and have been using duktape for some time.

It could be node compatible. I'm just not done.

Personally though I prefer the libuv APIs over the node APIs.

If anyone is wanting a node.js like experience with duktape, I've been working on libuv bindings for duktape.

https://github.com/creationix/dukluv

Thanks for posting here. I hope people like this idea.