Hacker News Reader

Comment by CoolGuySteve | Hacker News Reader

How much does it cost to use on S3? For example, how many GetObject and other misc non-free API calls does it use? And does it store data using intelligent tiering?

View on HN

For small files (less than 4MB), the number of Get/Put/Delete is the same comparing using S3 directly. For larger file, each object in S3 is about 4MiB. Most of S3 Client will do the similar thing to GET/PUT small parts in parallel to speed things up. Overall, JuiceFS should use the similar number of GET/PUT request comparing to use S3 directly.

Second, all the List and Head request go to Redis, they are free, so you may save some cost on API costs.

Third, the frequently read data will be cached in your local disks, so you will also save some cost on GET/PUT requests.

The underlying S3 bucket still have intelligent tiering, you can also put life cycle rules on it.

If a lifecycle rule deletes an object from the bucket, does the redis metadata server gracefully handle this (i.e. bucket state drifting)?

JuiceFS access objects in S3 using an unique id to generate key, so the result of lifecycle will not change the way JuiceFS accessing it, except the StorageClass Glacier or Deep Arching, which is not accessible instantly, will hurt the user experience for JuiceFS.

Thanks. Sorry I wasn't being very clear, I meant: what happens if an object is permanently deleted from a bucket by an out-of-band process, invisible to JuuiceFS (e.g. by a user operating on the bucket directly or by a misconfigured life cycle rule that deletes old objects after X days/months/years)?

Does JuiceFS's metadata server handle this loss of synchronisation gracefully?

Reply on news.ycombinator.com