daviesliu

Good point.

Most of file storage picked GPL, for example, Ceph, GlusterFS, MooseFS, so we followed them.

Right now, the metadata is not replicated to S3, you can replicated it to another Redis, or backup the persisted RDB and AOF to S3.

Redis can be persisted with RDB and AOF, can also be replicated to another machine. In the cloud, you don't need to worry about that, hosted Redis are ready to use.

The is an ongoing effort [1] to improve the persistency and availability in general, which is expected to be GA in 2021.

[1] https://github.com/RedisLabs/redisraft

Good, that's why we choose Redis for the metadata. When Redis is deployed in same DC or VPC, the latency could be about 0.2ms - 0.5ms.

Later on, We will try the client cache of Redis, which could also reduce the latency for some metadata operations down to a few microseconds.

Since POSIX is so complicated that we would not trust the test we wrote to prove the compatibility.

So we use several third-party test suite, including fsx, pjdfstest, fsracer, flock. We also use third-party tools for benchmark, fio, mdtest and others.

The architect of JuiceFS is very close to GFS, which use a single node master for many years, even now. Since Redis is only responsible for metadata, a single node can serve hundreds of millions of files, and tens of thousands of IOPS, that should be enough for many use cases.

The term `distributed`, means that JuiceFS is not a `local` file system, or can only be used by single machine. JuiceFS should be qualified as distributed system, even the core part is the client, which could be used by many machines in the same time.

The Kubernetes CSI driver will be released soon.

The underlying S3 bucket still have intelligent tiering, you can also put life cycle rules on it.

For small files (less than 4MB), the number of Get/Put/Delete is the same comparing using S3 directly. For larger file, each object in S3 is about 4MiB. Most of S3 Client will do the similar thing to GET/PUT small parts in parallel to speed things up. Overall, JuiceFS should use the similar number of GET/PUT request comparing to use S3 directly.

Second, all the List and Head request go to Redis, they are free, so you may save some cost on API costs.

Third, the frequently read data will be cached in your local disks, so you will also save some cost on GET/PUT requests.

Comparing to SeaweedFS, JuiceFS is more feature complete rather than basic read/write functionalities.

The core of seaweedfs is to manage many small blobs, you can use SeaweedFS together with JuiceFS to have a full featured POSIX file system.

I'd say Route 53 is more of an eventual consistent object store rather than a database :-)

Comparing to seaweedfs, JuiceFS is more feature complete rather than basic read/write functionalities.

The core of seaweedfs is to manage many small blobs, you can use SeaweedFS together with JuiceFS to have a full featured POSIX file system.

The file format is described in https://github.com/juicedata/juicefs#architecture

We will provide tools to dump the metadata as JSON, then you could recover your files using that.

Good question, we have not try xfstests yet, will put it on TODO list.

Founder of JuiceFS here, we have spent 4 years building it and open sourced it today.

Yes, JuiceFS splits files into 4MB blocks and put them into S3.

JuiceFS supports GCS and can be used in GCP, we have customers use it for data scientists. JuiceFS is free if you don't have data more than 1TB.

Discloser: Founder of JuiceFS here

> interop is too valuable

Good point, JuiceFS already provides this "transparent mode", which is called compatible mode.

There are different when they are read/write from multiple clients in same time. HopsFS can use the meta service to synchronize with each other under low latency (about ms), but ObjectiveFS have to use S3 to do the synchronisation, which has much higher latency (> 20ms).

We have chatted on these with the founders of ObjectiveFS before creating JuiceFS, they did NOT recommend to use ObjectiveFS in big data workload with Hadoop/Spark, that's why we started to build JuiceFS since 2016.

JuiceFS can used in AWS EMR cluster, we have customers using JuiceFS to address the consistency and performance issues from S3.

There is a blog post [1] talking about this use case. Unfortunately, we have not publish a English version, you can read it using google translate .

[1] https://juicefs.com/blog/cn/posts/globalegrow-big-data-platf...

Disclaimer: Founder of JuiceFS here.

We see similar results using JuiceFS [1], are glad to see that another player has moved on the same direction with us, since JuiceFS was released to public in 2017, and is available on most of public cloud vendor.

The architect of JuiceFS is simper than HopsFS, it does not have worker nodes (the client access S3 and manage cache directly), and the metadata is stored in highly optimized server (similar to NN in HDFS, can be scaled out by adding more nodes).

JuiceFS provide POSIX client (using FUSE), and Hadoop SDK (in Java), and a S3 gateway (also a WebDAV gateway).

[1]: https://juicefs.com/docs/en/metadata_performance_comparison....

Disclaimer: Founder of JuiceFS here

Juicesync support HDFS, can copy data using multiple machines, which is game killer if you have petabytes of data.

Good point.

Juicesync does support less service, but has better coverage for enterprise (Rclone does much better for individuals).

Btw, we are on the way to open source JuiceFS.

Juicesync is simper (fewer command line options) and faster (using multiple threads and machines) than rclone.

disclaimer: worked on juicesync

Have you publish the workload tracing from Spotify?

We are very interesting on the results you got, looking for a way to replicate that using our setup.

> Also, Huawei in their early ages stole one of Cisco router's source code and priced out Cisco with an exact same product compatible with Cisco systems, one Chinese academic invovled in modifying admitted it in an article. One of the role model stories.

Source?

I have done the things before. Before returning the work laptop to employer on the last day, I cleaned out all the company stuff to left my personal things (this is my only laptop with my personal stuff in it), then move them out and clean them finally.

Is this common?

In my opion, Cephfs and GlusterFS have much better support for Pandas and others, you don't need change any code between local development and train an distruted way, rather than a path.

Second, the Python library for HDFS provides only subset of filesystem API, which limits what you can do with it.

The rbd image can't be used as a shared storage for database.

JuiceFS [1] could be a replacement for S3QL, which has better support for B2, can also be mounted on multiple machines in the same time.

If you don't have more than 1TiB, JuiceFS is totally free.

Disclaimer: Founder of JuiceFS here.