Hacker News Reader

Comment by cratermoon | Hacker News Reader

> the real problems stem more from schemas than data formats

Oh, how the turntables turn. Ontologies are hard, and always fail in some way[1]. There are strategies like entity resolution[2] that attempt to take items from dataset A and items from dataset B and figure out if they refer to the same thing, but they are expensive and time-consuming and even when they work, only approximate.

Anyone has ever done a data migration project of any complexity knows that there must be provisions for dealing with data quality issues, especially if effort includes reconciling data from multiple sources.

1 https://oc.ac.ge/file.php/16/_1_Shirky_2005_Ontology_is_Over... 2 https://doi.org/10.1145/3418896

View on HN

We did a great many of them over the decades; I find the mongoldb ones we encountered the most horrible (even compared to vague 80s + 90s plain text formats as they usually at least had some documentation as to what the developer was storing and how), but in general, we found the most reliable and, when you are used to it, the fastest (in the end) is creating types for everything and convertors from one to the other type. We did try other (more sloppy) ways before it and they always kind of result in bad quality. We have noticed that making (very exact) types for everything when doing complex conversions works well. Most notably, we try to avoid stringy types and add a lot of validations in the deserialisers. You can run conversions 10 years later on new batches and still get understandable errors.

> try to avoid stringy types and add a lot of validations

Right there you've stated a key practice that may, by itself, be 80% of the solution. Of course there's no "right" set of of types, but just putting the work into thinking about the types and the essential constraints will drive towards exposing and eliminating assumptions that will come back to bite later.

Another huge chunk of problems can come from over-reliance on language primitive types. Is that numeric field really an integer, meaning that any signed value representable in 32, 64, or however many bits, including zero, is valid? Probably not. To give an example: A person's age is not an signed integer, it's a duration, we don't expect it to be negative. Maybe we should flag values above 90 or 100. If you're doing a one-off, that's the sort of thing you can code once and forget. If you dataset migration is your business, your Age type needs to be more robust, and perhaps there's even a need for the behavior of Age to vary with the domain.

Which language did you use to encode these types?

We work in C# mainly; unfortunately; I have been in favour of F# for a long time, but it's just not popular enough so it would make our (already large, with decades of work) codebase less maintainable. However it does not really matter as long as you can make the types (classes in the case of C#) precise enough. You can indeed (see the other commenter) make (an) Age classe(s) which will give a nice error message if you do something illegal, for instance;

    int i = -2; 
    Console.WriteLine((Age)i); 
    Console.WriteLine((Age)200);
    Console.WriteLine((AdultAge<UnitedStates>)17);

will throw three Exceptions. We define every single piece of data like that.

> There are strategies like entity resolution[2] that attempt to take items from dataset A and items from dataset B and figure out if they refer to the same thing

The more code I have written, the more convinced I become that "general" solutions almost never pay for themselves when you compare the implementation and/or performance complexity required to make them work.

It's almost always better to "just write the damn code" and solve the exact problem you actually have.

If every time you do it is a one-off, yes. If your business is essentially nothing but aggregating datasets (which may themselves be aggregates of still more datasets), then by the third time you've solved the exact problem, you're going to see commonalities. You abstract those out so that the fourth, fifth, and sixth problems don't have to re-write that part. After a while, you've built out enough abstractions that you're programming the migration process itself, not each migration.

Add in the fact that your business depends on doing it at scale, ingesting live data as it is generated, and developing an entity resolution system is almost a no-brainer.

Yes and no.

They don't pay if you try to generilize before you have a problem.

If you have a frequent use case that follows the same patterns, then you probably can benefit from an abstraction layer.

But trying to build an abstraction layer before even having a use case then yes, it usually doesn't pay off.

Reply on news.ycombinator.com