ibegtin

Hi HN, I've made major progress in implementing Python lib, Iterable Data for reading and writing data files row by row in a consistent, iterator-based interface. It provides a unified API for working with various data formats (CSV, JSON, Parquet, XML, etc.) similar to csv.DictReader but supporting many more formats.

I use it extensively inside our dataset search engine Dateno and it's helpful since I had to operate with wild range of data formats and something more or less universal was really needed.

Repo: https://github.com/datenoio/iterabledata

I'd love feedback from folks who reads, converts and processes data regularly and existing tools are not ideal for these tasks!

Now I am lookng for

Open source library and tool that helps to identify semantic types of the columns the datasets or database tables. Useful to find PII data or to automatically document existing undocumented databases or data files.

Just launched officially, new global dataset search engine with 10M datasets, near-realtime search and 13 facets to filter results. More datasets, open API and more open source upcoming soon.

I have used OpenRefine for a long time to refine / clean / manually process table data. But I can't find anything similar for NoSQL (JSON/JSON lines) data. I wrote an experimental Python library that mimics MongoDB as column-based, not document-based, NoSQL RDBMS, and I wrote a headless wrapper that allows manipulating columns/fields as you may do it using OpenRefine. It's highly experimental and not performance optimized yet. I am looking for feedback and strong critique.