Example: 100 TB, write us every 10 mins.
Or, 1tb, parquet, 40% is rewritten daily.
Example: 100 TB, write us every 10 mins.
Or, 1tb, parquet, 40% is rewritten daily.
Maybe Pachyderm or Dolt would be better tools here.
There's an ongoing discussion about replacing/configuring the hash function, and it looks like there might be some movement toward replacing the hash and other speedups in 3.0
https://github.com/iterative/dvc/issues/3069
> We not only want to switch to a different algorithm in 3.0, but to also provide better performance/ui/architecture/ecosystem for data management, and all of that while not seizing releases with new features (experiements, dvc machine, plots, etc) and bug fixes for 2.0, so we've been gradually rebuilding that and will likely be ready for 3.0 in the upcoming months. - https://github.com/iterative/dvc/issues/3069#issuecomment-93...