ishcheklein

There are two projects that take this direction - Splitgraph and Dolt.

Interesting, what are the benefits of using this vs Sagemaker?

"Powered by open source OrgStack's core validation engine is open source and available to the community." - repo is not available?

Feels like an empty landing page to test the idea at this moment.

Okay, thanks! I think I got the idea. Have you seen Quilt btw? They had the same message initially - Docker for Data, packaging data, etc. Implementation was very different though.

I somewhat don't like this analogy btw, see how you mentioned Docker changed life for DevOps in the first place (vs engineers), the same here - data scientists don't care about packaging data - there should be strong incentive to do so.

A few specific questions:

1. where does query execution happen - always client or remote as well? 2. in a global "Github for data" case is there some discovery mechanism for existing data? 3. do you provide a public storage to cover the case for Github for data? or is it now more like torrent - peers host and pay for data storage?

Btw, what is major direction for you - Github (public collaboration and sharing) or internal versioned warehouses (or some other internal case?).

Thanks! Still trying to wrap my mind around use cases :) What is the main adoption path for the tool do you see?

People who use flat files and they are large enough and/or need some provenance? Or people who already have Postgres and they want to version it? Or is it about production use case (like Docker) - make snapshot to deliver it consistently?

Btw, regarding the FQA - thanks! From my take on Splitgraph though one the main DVC's difference is that it deals with flat files. From software engineering perspective - it is very close to Git lfs on steroids + some higher level features similar to makefiles, ML metrics, etc.

Here is one of the DVC maintainers :) Congrats! It's great to see more tools for codifying data in different scenarios.

To be honest, since you introduce a new workflow and a few new concepts it's not that easy to get the right perspective in 5 minutes (I know the same problems exists with DVC and we've been iterating on docs a lot). Mind a few questions?

Do I understand it right, that is mostly focused on tabular data? Kinda git checkout for an SQL table?

How does exactly it handles security? Will it be another hell like Adobe Flash was?

Nice! I moved ~5years ago to the US from Moscow - was nice to get back virtually :)

I'm co-founder of Iterative.ai /DVC.org (open source tools for ML) and doing conferences/meetups is an essential part of the business for us (kinda marketing for dev tools). I would definitely recommend you to connect and talk to amazing folks from the https://tulu.la/ team. They are building (still in stealth) a product to run online events with experience as engaging as possible. I'm biased, since I know founders personally (and happy to be their friend!), but they started their company with this vision even before all this coronavirus stuff, they definitely have a big vision in mind for a long term and have been passionate about this space for years. An example would be this online conference they organized a while ago - https://divops.org/. I would happy to connect you (and I'll send a link to this conversation to them).

DVC (Iterative) | REMOTE | FULL-TIME | OPEN SOURCE | dvc.org (iterative.ai)

Our mission is to build widely adopted tools that solve complexity of dealing with data and ML. Think of Hashicorp (Terraform, Vault, etc) for MLOps. We love open source! The whole team is highly technical and you can find our profiles on GH.

We are VC-backed, well-funded company. Adoption for DVC is growing fast.

Please, find the full job description here - https://remoteok.io/remote-jobs/76126-remote-senior-software.... and feel free to ask any questions.

To apply send a direct email to ivan@dvc.org.

DVC (Iterative) | REMOTE | FULL-TIME | OPEN SOURCE | dvc.org (iterative.ai)

We are Hashicorp (Terraform, Vault, etc) for MLOps. Our mission is to build widely adopted tools that solve complexity of dealing with data and ML. We love open source! The whole team is highly technical and you can find our profiles on GH.

We are VC-backed, well-funded company. Adoption for DVC is growing fast.

Please, find the full job description here - https://remoteok.io/remote-jobs/76126-remote-senior-software... and feel free to ask any questions.

To apply send a direct email to ivan@dvc.org.

Thank you! It all makes sense.

A technical question - how does your system work in the IRIS or other examples you have? It looks like it hides under the hood a few things compared to this implementation - type of the model (tensorflow) and the logic that fetches it from S3, right? Let's imagine we would want to make a DVC repo (just to store model versions to start) out of one of your examples instead of the DVC get started, how would we do that with the current implementation (through metadata + custom init)?

Nice! Thanks for creating the integration. I'm one of the maintainers of the DVC project and would be happy to answer any questions.

It's mentioning DVC inside it, I would be happy to discuss and answer any questions as one of the maintainers.

An excellent read on open-source ML workflow tools - why do we need any new tools at all? and how should we approach creating them? It builds upon another great talk by Patric Ball - Principled Data Processing - https://www.youtube.com/watch?v=ZSunU9GQdcI.

From a very high level perspective - Pachyderm is a data engineering tool designed with ML in mind, DVC is a tool to organize and version control an ML project. Probably, one way to think would be Spark/Hadoop or Airflow vs Git/Github.

For those who familiar with cookiecutter-data-science, there is a good example how DVC can version and manage data -https://github.com/drivendata/cookiecutter-data-science/issu... + this PR to play with it - https://github.com/drivendata/cookiecutter-data-science/pull...