Ask HN: ML – how big is your training data and how do you store/manage it?

3 pointsmrmrcoleman3 comments

Big in terms of disk size. Manage in terms of update, extract, share, collaborate.

Big by what measure? Videos, images, and text have very different profiles.

What do you mean by manage? Sharing, splitting, cleaning, versioning, auth, providing for distributed training are all options here.

Can you please elaborate one your Ask HN

Thanks, updated.

For text classification:

* unsupervised: 50 million to 1 billion characters

* supervised: labels for 2,000-50,000 documents.

Data stored in Arangodb, accessed through Python scripts that run in Jupyter notebooks (e.g. using GUI widgets)