My most common use cases involve getting PDFs or HTML files and I have to parse the metadata to store along with the embedding.
Would I have to run a process to extract file metadata into JSONs for every embedding/chunk? Would keys created based off document be title+chunk_no?
Very interested in this because documents from clients are subject to random changes and I don’t have very robust systems in place.
Extract metadata as usual, then return the result as JSON or a Pydantic object. DataChian will automatically serialize it to internal dataset structure (SQLite), which can be exported to CSV/Parquet.
In case of PDF/HTML, you will likely produce multiple documents per file which is also supported - just `yield return my_result` multiple times from map().
Check out video: https://www.youtube.com/watch?v=yjzcPCSYKEo Blog post: https://datachain.ai/blog/datachain-unstructured-pdf-process...
Could your metadata come from something like a Postgres sql statement? Or an iceberg view?
Just connect from your Python code (like the lambda in the example) to DB and extract the necessary data.
Forgive my ignorance, but what is "json-pair"?
It's simpliy about linking metadata from a json to a corresponding image or video file, like pairing data003.png & data003.json to a single, virtual record. Some format use this approach: open-image or laion datasets.
Hopefully, @jerednel can add more details.
My retriever functions will typically use metadata in combination with the similarity search to do impart some sort of influence or for reranking.