By ultra-wide I mean tables with thousands of columns.
I've got my basic concepts down, now looking to read the literature on what has been written before.
Anyone have any pointers on people or terms to search?
Are you writing huge amounts of data?
How are you reading the data?
I'm assuming many of the columns will be empty for most records?
No. All this data is reviewed by at least 1 human. It may be added by a crawler and skimmed for QA, but it is higher level data as opposed to event stream data. Small quantities of data currently in 10MB range but I suspect won't exceed 1GB
> How are you reading the data?
Easy to hold it all in memory.
> I'm assuming many of the columns will be empty for most records?
Yes. Currently 90% sparse. I expect eventually will follow the 80-20 rule.
> why you want to use ultra wide tables?
This is a long winded answer. The short version is "I don't know". It's a gut brain thing that I'm trying to move upstream to my symbolic brain and get the math onto paper. It feels to me like there is something "magic" in a wide dataset. I'm trying to formalize what that magic is, or maybe I'm hallucinating it.
Years ago I had the flash of insight (I wasn't the first, nor will I be the last) that all data structures, code included, could be represented with just words and indentation. I called my implementation Tree Notation and languages built on this syntax Tree Languages. (S-Expressions without parentheses is a 90% accurate description).
But the idea failed to catch on. It could be a bad idea. I'm still not sure. Perhaps it was missing something. I started collecting information about every notation and language I could find. This has helped me make Tree Languages better (though I still don't have a clear model as to whether the underlying notation is great or not), but also that dataset became a product itself and I shipped it at PLDB.com.
With the PLDB dataset, I feel like I can now quickly compute an accurate research cost estimate to any question about programming languages (and increasingly the answers are instant from the data already there).
I feel like there is a non-linear effect driven largely by having hundreds of columns, and I think it would scale where the product would be far better with another 2x of columns (and also fill in more missing data, of course). We are just kind of adding data and will have some empirical results as to whether that is right (perhaps it's wrong and the product won't substantially improve and the returns of adding more data will be diminishing—not sure). But I'm trying in life to take a more theoretical look at things to, and try to figure out formally why this gut feel is right or wrong.
One idea at formalization is that the value of a dataset increases exponentially with the number of independent columns (Value = n^2). Obviously that's the best case, the average case impact of a new column would be worse by some factor. There's probably some known math around this from the Decision Tree literature.
Another idea is that adding columns, even if sparse, is valuable because then you make known the unknowns. Wide tables could maximize the chances of collecting columns that offer a wide variety of perspectives to maximize the odds of finding the best models that most accurately predict the data.
Anyway, as you can tell it's not very clear to me yet, but I'm confident someone has written some math somewhere about the shape of the return curve from adding more columns to a dataset.
Yes and that might be it and if so I don't think that's a huge benefit because joins are easy (with a little practice).
> Your data size is smaller than I expected and I'm still not sure I understand your use case. With that size of data you could choose pretty much any model and it would probably fit your requirements.
This is helpful feedback. I realize I'm not phrasing my question correctly. Nothing I'm trying to do is hard on the computing end.
What I'm trying to model is the ~value~ of a "computable encyclopedia".
Imagine if Wikipedia was just infoboxes.
If you had a single table 10 million rows long, but with maybe ~100,000 columns, would that be: _meh_ or a _step change_ in value? My thinking is the latter. If I wanted to learn about "bicycles" I could ask dozens of ad hoc structured questions from such a table and get instant answers as opposed to reading and digesting narratives.
My hunch is that such a structured dataset would be a significant increase in value over Wikipedia, but I'm not sure if I'm right and haven't yet been able to model from first principles why this would be right.
I could be wrong.