Easier data debugging with Zed’s first-class errors | Hacker News Reader

Easier data debugging with Zed’s first-class errors

brimdata.io

33 pointsjameskerr17 comments

Storing the errors alongside other transformed inputs is a really good idea. But manually patching up and/or deleting individual records is a really bad idea! How is that going to help your data integrity?

Surely you want to fix your importer and re-run it. If there was a smart tool that was able to avoid duplicating work and only re-import the affected records, that'd be great, but that doesn't seem to be what's shown here.

Oh wow ! I was just trying to articulate exactly this kind of approach and looking for it. I would love to see a coherent approach to data errors and this seems like a step in the right direction.

One question - the blog post covers basically debugging the ingestion of data part. My quite usual issue with older data is that at some point, you discover an issue with it (say it's slightly false, but not too much) - so you want to somehow let users know, or allow to select only the data without the issue (but still let them know how much of it they miss) - is this framework helpful in this situation ?

Yes, great point! The idea is that you can fix up a problem with data in place, while you're updating your ingest pipeline to handle whatever is causing the problem. You can do a transform on the errors into clean data, delete the errors, and commit the changes atomically. In the meantime, queries and searches can still run on the data that isn't problematic and even if there are errors inside of a hierarchical value, queries can be run on the portions of the value that are clean and intact while the errors are being addressed.

It took me a minute to realize that this isn’t about the Zed text editor.

Oh it isn’t?! :(

Yes, the name conflict is unfortunate. We named our project Zed in 2021 before the Zed text editor was a thing.

This article would improve by linking to what Zed is supposed to be: https://zed.brimdata.io/docs

"Wouldn’t it be great if you could see errors in place instead of mysterious NULLs?"

It's not a good ad when the error message is inadequate even in the supplied example and you need to hack around it.

Apologies if our examples in the article weren't rich enough or clear. The beauty of the error type is that you can wrap any value in an error and make the error as rich as you'd like, even stacking errors from different stages of an ingest pipeline so you can see the lineage of an error alongside data that wasn't subject to errors. e.g., imagine an error like this:

  error({
      stage: "transform",
      err: "input error",
      value: {
          stage: "normalize",
          err: "input error",
          value: {
              stage: "metrics",
              err: "divide by zero",
              value: {
                  sum: 123.5,
                  n: 0
              }
          }
      }
  })

... and you can quickly deduce that your "metrics" stage is dividing by "n" even if n is 0 and you can fix up your logic as well as fix the errors in place after fixing the bug in the ingest pipeline.

My point is that you know the real source of the error, so there's no point for these terse messages. The example in the article, you know at point of conversion that you get a float, but instead you need to do a dance to get the original. Here it's a divide by zero - you can tell me the name of the variable too.

There's no need to repeat the infamous MSSQL "String or binary data would be truncated" message saga here - there's no reason you can't give much more verbose errors by default.

For example, the error could have been '1064 is not a string value' or at least 'float64 is not a string value' etc.

Your observations are definitely valid. Structured errors were introduced in Zed some time ago, but as you highlight, they're not yet being generated in all the places they could be and with appropriate detail. There were a couple existing to-do items in the Zed backlog to make these enhancements in packages where it would have a high impact, but we've now created an additional Epic https://github.com/brimdata/zed/issues/4608 to gather these up and remind ourselves to address this in a comprehensive way. Thanks for helping to shine a light on this!

Thought it was about Zed Shaw for a moment and got ready for something colorful

This is really cool. I’d imagine this helps with datasets skewed by errors and actually allows incident losses to be quantized in ways that are usually pretty cumbersome. I think SREs would love this

Interesting. Seems like a hassle to store efficiently.

Hey, thanks for bringing up storage efficiency. This is an often cited objection and a big motivation of the Zed project actually. In Zed, data is organized around a self-describing type system rather than schemas, and unlike the relation model, heterogeneous data can land side-by-side at the storage layer. And unlike NoSQL JSON-based systems, Zed is comprehensively typed, which allows us to form vectors for heterogeneous data and get OLAP-style analytics performance even with heterogenous data. We're working on the vector analytics layer as we speak and hope to have an "MVP" of the Zed lake with vector support in the coming months.

TL;DR Zed stores data efficiently based on types not schemas so first-class error values of type "error" fit in anywhere with ease.

Unfortunate product name

Reply on news.ycombinator.com