Knowledge Compressor

These methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive.

The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as:

  lbug> MATCH (a:E)-[t:TRIPLE]->(b:E) RETURN a.name AS subj, t.predicate AS pred, b.name AS obj, t.props AS props, t.scope AS scope;