mccanne

Hey really cool! Tcpdump co-author here. If you're interested in the origin story of the tcpdump language, bpf, and pcap, I had fun putting together a talk a number of years ago... https://www.youtube.com/watch?v=XHlqIqPvKw8

For sure... the use of JSON here seems orthogonal to Jamie's point of constructing complex nested values in SQL with a single backend query. Modern SQLs support all this as native types (presuming the result can fit in a homogeneous relational type, which it can in this case), e.g., Jamie's query can be written with a record expression (duckdb dialect):

  with theTitle as (
    from title.parquet
    where tconst = 'tt3890160'
  ),
  principals as (
    select array_agg({id:principal.nconst,name:primaryName,category:category})
    from principal.parquet, person.parquet
    where principal.tconst = (from theTitle select tconst)
    and person.nconst = principal.nconst
  ),
  characters as (
    select array_agg(c.character) as characters, p.u.name
    from principal_character.parquet c
    join (select unnest((from principals)) as u) p
      on c.character is not null and u.id=c.nconst and c.tconst=(select tconst from theTitle)
    group by p.u
  )
  select {
    title: (select primaryTitle from theTitle),
    director: list_transform(
                list_filter((from principals), lambda elem: elem.category='director'),
                lambda elem: elem.name),
    writer: list_transform(
              list_filter((from principals), lambda elem: elem.category='writer'),
                lambda elem: elem.name),
    genres: (select genres from theTitle),
    characters: (select array_agg({name:name,characters:characters}) from characters),
  } as result

And if you query typeof on the result, you'll get:

  STRUCT(
    title VARCHAR,
    director VARCHAR[],
    writer VARCHAR[],
    genres VARCHAR,
    characters STRUCT(
      "name" VARCHAR,
      characters VARCHAR[]
    )[]
  )

Relevant discussion from a few years back

https://news.ycombinator.com/item?id=28221654

And somewhat ironically here, SuperDB not only implements sum-type decoding of JSON in package unpack, but it also implements native sum types in a superset of JSON that we call Super JSON (with a query language that understands how to rip and stitch sum types for columnar analytics... work in progress)

https://superdb.org/docs/formats/data-model/#25-union

Nice article!

Decoding sum types into Go interface values is obviously tricky stuff, but it gets even harder when you have recursive data structures as in an abstract syntax tree (AST). The article doesn't address this. Since there wasn't anything out there to do this, we built a little package called "unpack" as part of the SuperDB project.

The package is here...

https://github.com/brimdata/super/blob/main/pkg/unpack/refle...

and an example use in SuperDB is here...

https://github.com/brimdata/super/blob/main/compiler/ast/unp...

Sorry it's not very well documented, but once we got it working, we found the approach quite powerful and easy.

Really cool though typing ">" after "|" is a pain https://github.com/brimdata/super/blob/main/docs/language/pi...

Necessity is the mother of invention. MapReduce-based systems were developed because the state-of-the-art RDBMS systems of that age could not scale to the needs of the Googles/Yahoos/Facebooks during the phenomenal growth spurt of the early Web. The novelty here was the tradeoffs they made to scale out and up using the compute and storage footprints available at the time.

"We thought of that" vs "we built it and made it work".

Yes, the name conflict is unfortunate. We named our project Zed in 2021 before the Zed text editor was a thing.

Apologies if our examples in the article weren't rich enough or clear. The beauty of the error type is that you can wrap any value in an error and make the error as rich as you'd like, even stacking errors from different stages of an ingest pipeline so you can see the lineage of an error alongside data that wasn't subject to errors. e.g., imagine an error like this:

  error({
      stage: "transform",
      err: "input error",
      value: {
          stage: "normalize",
          err: "input error",
          value: {
              stage: "metrics",
              err: "divide by zero",
              value: {
                  sum: 123.5,
                  n: 0
              }
          }
      }
  })

... and you can quickly deduce that your "metrics" stage is dividing by "n" even if n is 0 and you can fix up your logic as well as fix the errors in place after fixing the bug in the ingest pipeline.

Yes, great point! The idea is that you can fix up a problem with data in place, while you're updating your ingest pipeline to handle whatever is causing the problem. You can do a transform on the errors into clean data, delete the errors, and commit the changes atomically. In the meantime, queries and searches can still run on the data that isn't problematic and even if there are errors inside of a hierarchical value, queries can be run on the portions of the value that are clean and intact while the errors are being addressed.

Hey, thanks for bringing up storage efficiency. This is an often cited objection and a big motivation of the Zed project actually. In Zed, data is organized around a self-describing type system rather than schemas, and unlike the relation model, heterogeneous data can land side-by-side at the storage layer. And unlike NoSQL JSON-based systems, Zed is comprehensively typed, which allows us to form vectors for heterogeneous data and get OLAP-style analytics performance even with heterogenous data. We're working on the vector analytics layer as we speak and hope to have an "MVP" of the Zed lake with vector support in the coming months.

TL;DR Zed stores data efficiently based on types not schemas so first-class error values of type "error" fit in anywhere with ease.

A quick look at the forthcoming release of zui-1.0. Explore gnarly JSON data with ease!

Simon, brilliant observations here and kudos on sqlite-utils.

I'm all for layers, a fundamental approach in our field to tame complexity. And the SQL model and SQLite have stood the test of time and are solid foundations.

I'm just wondering could we be stuck in a local maximum where the presumed answer is always the relational model? Maybe if we built the relational model on top of a different set of lower-level primitives (a type system instead of schemas and tables) we could escape local maximum we're stuck in? Just a thought.

There are a few somewhat ad hoc perf measurements here regarding the sqlite-utils and sqlite... https://zed.brimdata.io/docs/commands/zq/#73-performance-com...

I'm not a SQLite expert so if I did something wrong, please holler and let me know :)

Hear, hear!

Author here. All good points. Yes, you can build a super-structured type system on top of tables. EdgeDB does this well. And you can put JSON into relational columns. Then you might ask what the "type" of that column is? Well, if you want deep types, the row type varies from column to column as the JSON values vary and you have to walk the JSON to determine the type. SQL implementation are beginning to try to do deal with this mess by adding layers on top of tables. We're saying, maybe we should think differently about the problem and build tables on top of types as a special case of a type system. This also gives a very nice way to get data into and out of systems without having to go through the messiness of ODBC and special casing tables vs tuples vs scalars etc.

Author here. Agreed! Validation is important. While I didn't make this point in the article, our thinking is schema validation does not require that the serialization format utilize schemas as the building block and you can always implementation schema (or type) validation (and versioning) on top of super-structured data (as can also be done with document databases).

Author here. This totally makes sense. The challenge here is you need to store the type definitions somewhere (e.g., in the .proto files) and any system that processes protocol buffers needs to know which proto to apply to which messages. The theme of super-structured data is that this type structure should be native to the serialized data and our premise is this leads to better DX (though Zed is early and the jury is out). Perhaps flexbuffers is a closer analogy, which I should have mentioned in the article.

zq good/ -

Wow, thanks.

Coincidentally, after hearing of a friend's woes dealing with massive amounts of CSV coming from a BPF-instrumental kernel, I played around a bit with integrating Zed and BPF. Just an experimental toy (and the repo is already out of date)...

https://github.com/brimdata/zbpf

The nice thing about Zed here is any value can be a group-by key so it's easy, for example, to use kernel stacks (an array of strings) in a grouping aggregate.

(p.s. for the record, the only thing I have to do with the modern linux BPF system is the tiny vestige of origin story it shares with the original work I did in the BSD kernel around 1990)

That's a cool idea. We've had many collaborators using Zed lakes for search at smallish scale and we are still building the breadth of features needed for a serious search platform, but I think we have a nice architecture that holds the promise to blend the best of both worlds of warehouses and search.

As for list() and values() functions, Zed has native arrays and sets so there's no need for a "multi-value" concept as in splunk. If you want to turn a set into an array, a cast will do the trick, e.g.,

echo '1 2 2 3 3' | zq 'u:=union(this) | cast(u,<[int64]>) ' -

[1,2,3]

(Note that <[int64]> is a type value that represents array of int64.)

Hi, all. Author here. Thanks for all the great feedback.

I've learned a lot from your comments and pointers.

The Zed project is broader than "a jq alternative" and my bad for trying out this initial positioning. I do know there are a lot of people out there who find jq really confusing, but it's clear if you become an expert, my arguments don't hold water.

We've had great feedback from many of our users who are really productive with the blend of search, analytics, and data discovery in the Zed language, and who find manipulating eclectic data in the ZNG format to be really easy.

Anyway, we'll write more about these other aspects of the Zed project in the coming weeks and months, and in the meantime, if you find any of this intriguing and want to kick the tires, feel free to hop on our slack with questions/feedback or file GitHub issues if you have ideas for improvements or find bugs.

Thanks a million!

https://github.com/brimdata/zed https://www.brimdata.io/join-slack/

There are a few examples in the ZSON spec...

https://github.com/brimdata/zed/blob/main/docs/formats/zson....

And you can easily see whatever data you'd like formatted as ZSON using the "zq" CLI tool, but I just made this gist (with some data from the brimdata/zed-sample-data report) so you can have a quick look (the bstring stuff is a little noisy and an artifact of the data source being Zeek)... https://gist.github.com/mccanne/94865d557ca3de8abfd3eb09e8ac...

Also, if any of you find problems with the Zed spec(s), we'd love to hear about them. "Now" would be a good time to make changes / fix flaws.

This is a very real problem being addressed here and I am intrigued by all the great comments in this thread.

In the Zed project, we've been thinking about and iterating on a better data model for serialization for a few years, and have concluded that schemas kind of get in the way (e.g., the way Parquet, Avro, and JSON Schema define a schema then have a set of values that adhere to the schema). In Zed, a modern and fine-grained type system allows for a structure that is a superset of both the JSON and the relational models, where a schema is simply a special case of the type system (i.e., a named record type).

If you're interested, you can check out the Zed formats here... https://github.com/brimdata/zed/tree/main/docs/formats

I only saw the bits around 230pm, but if by "startup school" they were trying to explain to the inexperienced entrepreneurs how they can be exploited by third-tier investors who enjoy mocking them, cutting them off while they are trying to explain their innovative ideas, and showing how smart they are to other people in the room, then that parody was pure genius. But I'm guessing most people missed the joke.

hey Steve here, the BPF+ work was done mainly by Andy Begel... really good stuff but it never got incorporated into libpcap, and I never otherwise published how the original optimizer worked. I did, however, give a retrospective talk on it a couple years back at sharkfest...

http://sharkfest.wireshark.org/sharkfest.11/

Click on "Keynote Video" for a really boring talk :-)