huntaub

S3 Mountpoint is exposing a POSIX-like file system abstraction for you to use with your file-based applications. Foyer appears to be a library that helps your application coordinate access to S3 (with a cache), for applications that don't need files and you can change the code for.

Storage Gateway is an appliance that you connect multiple instances to, this appears to be a library that you use in your program to coordinate caching for that process.

These are, effectively, different use cases. You want to use (and pay for) Express One Zone in situations in which you need the same object reused from multiple instances repeatedly, while it looks like this on-disk or in-memory cache is for when you may want the same file repeatedly used from the same instance.

Yes, definitely. S3 has a time to first byte of 50-150ms (depending on how lucky you are). If you're serving from memory that goes to ~0, and if you're serving from disk, that goes to 0.2-1ms.

It will depend on your needs though, since some use cases won't want to trade off the scalability of S3's ability to serve arbitrary amounts of throughput.

Woah buddy, I worked with Andy for years and this is not my experience. Moving a large product like S3 around is really, really difficult, and I've always thought highly of Andy's ability to: (a) predict where he thought the product should go, (b) come up with novel ways of getting there, and (c) trimming down the product to get something in the hands of customers.

Also, did you create this account for the express purpose of bashing Andy? That's not cool.

Some quick questions that came up in the last post, that I wanted to go ahead and address:

How are you different than existing products like S3 Mountpoint, S3FS, ZeroFS, ObjectiveFS, JuiceFS, and cunoFS?

Archil is designed to be a general-purpose storage system to replace networked block storage like EBS or Hyperdisk with something that scales infinitely, can be shared across multiple instances, and synchronizes to S3. Existing adapters that turn S3 into a file system are either not POSIX-compliant (such as Mountpoint for S3, S3FS, or GoofyFS), do not write data to the S3 bucket in its native format (such as JuiceFS, ObjectiveFS – preventing use of that data directly from S3), or are not designed for a fully-managed one-click set up (such as cunoFS). We have massive respect for folks who build these tools, and we’re excited that the data market is large enough for all of us to see success by picking different tradeoffs.

What regions can I launch an Archil disk in?

We’re live in 3 regions in AWS (us-east-1, us-west-2, and eu-west-1) and 1 region in GCP (us-central1). Today, we’re also able to deploy directly into on-premises environments and smaller GPU clouds. Reach out if you’re interested in an on-premises deployment (hleath [at] archil.com).

Can I mount Archil from a Kubernetes cluster?

Yes! We have a CSI driver that you can use to get ReadWriteOnce and ReadWriteMany volumes into your Kubernetes cluster.

What performance benchmarks can you share?

We generally don’t publish specific performance benchmarks, since they are easy to over-index on and often don’t reflect how real-world applications run on a storage system. In general, Archil disks provide ~1ms latency for hot data and can, by default, scale up to 10 GiB/s and tens of thousands of IOPS. Contact me at hleath [at] archil.com, if you have needs that exceed these numbers.

What happens if your caching layer goes down before a write is synchronized to S3?

Our caching layer is, itself, highly-durable (~5 9s). This means that once a write is accepted into our layer, there are no individual components (such as an instance or an AZ) failure which would cause us to lose data.

What are you planning next for Archil?

By moving away from NFS and using our new, custom protocol, we have a great foundation for the performance work that we’re looking to accomplish in the next 6 months. In the short-term, we plan to launch: one-click Lustre-like scale-out performance (run hundreds of GiB/s of throughput and millions of IOPS without provisioning), the ability to synchronize data from non-object storage sources (such as HuggingFace), and the ability to use multiple data sources on a single disk.

How can I learn more about how the new protocol works?

We’re planning on publishing a bunch more on the protocol in the coming weeks, stay tuned!

Basically, we are building this at Archil (https://archil.com). The reason these things are generally super expensive is that it’s incredibly hard to build.

My (limited) understanding was that the industry previously knew that it was unsafe to share GPUs between tenants, which is why the major cloud providers only sell dedicated GPUs.

This is actually not the case. The TLS stream ensures that the packets transferred between your machine and S3 are not corrupted, but that doesn't protect against bit-flips which could (though, obviously, shouldn't) occur from within S3 itself. The benefit of an end-to-end checksum like this is that the S3 system can store it directly next to the data, validate it when it reads the data back (making sure that nothing has changed since your original PutObject), and then give it back to you on request (so that you can also validate it in your client). It's the only way for your client to have bullet-proof certainty of integrity the entire time that the data is in the system.

Thanks! Under the hood, when you mount an Archil volume, you connect to a fleet of instances that we're managing with SSD drives attached, which cache reads+writes before hitting the underlying data in your S3 bucket.

Hey, I'm Hunter -- the founder of Archil. I'll be around in the comments to answer any questions that people have about the platform, or how things have changed since the Fall.

This is exactly what I wanted for our team when I was at AWS. There are so many versions of operations which are just slightly too dangerous to automate, and this provides a path to iteratively building that up. Congratulations!

i-series instances have direct-attached drives

I think that worrying that a self-hosted file system has a backdoor to exfiltrate data is an odd concern. Security concerns are (obviously) normal, but you should not be exposing these kinds of services to the public internet (or giving them access to the public internet), eliminating the concern that it's giving your data away.

It's a very different architecture. 3FS is storing everything on SSDs, which makes it extremely expensive but also low latency (think ~100-300us for access). JuiceFS stores data in S3, which is extremely cheap but very high latency (~20-60ms for access). The performance scalability should be pretty similar, if you're able to tolerate the latency numbers. Of course, they both use databases for the metadata layer, so assuming you pick the same one -- the metadata performance should also be similar.

My guess is going to be that performance is pretty comparable, but it looks like Seaweed contains a lot more management features (such as tiered storage) which you may or may not be using.

Yes, I think this is probably true. I've worked with a lot of different hedge funds who have a similar problem -- lots of shared data that they need in a file system so that they can do backtesting of strategies with things like kdb+. Generally, these folks are using NFS which is kind of a pain -- especially for scaleability -- so building your own for that specific use case (which happens to have a similar usage pattern for AI training) makes a lot of sense.

Well, for active data, the idea is that the replication within the system is enough to keep the data alive from instance failure (assuming that you're doing the proper maintenance and repairing hosts pretty quickly after failure). Backup and recovery, in that case, is used more for saving yourself against fat-fingering an "rm -rf /" type command. Since it's just a file system, you should be able to use any backup and recovery solution that works with regular files.

I think that's a pretty odd concern to have. What would you imagine that looks like? If you're running these kinds of things securely, you should be locking down the network access to the hosts (they don't need outbound internet access, and they shouldn't need inbound access from anything except your application).

IMO this is the problem with all storage clusters that you run yourself, not just Ceph. Ultimately, keeping data alive through instance failures is just a lot of maintenance that needs to happen (even with automation).

I think that the author is spot on, there are a couple of dimensions in which you should evaluate these systems: theoretical limits, efficiency, and practical limits.

From a theoretical point of view, like others have pointed out, parallel distributed file systems have existed for years -- most notably Lustre. These file systems should be capable of scaling out their storage and throughput to, effectively, infinity -- if you add enough nodes.

Then you start to ask, well how much storage and throughput can I get with a node that has X TiB of disk -- starting to evaluate efficiency. I ran some calculations (against FSx for Lustre, since I'm an AWS guy) -- and it appears that you can run 3FS in AWS for about 12-30% cheaper depending on the replication factors that you choose against FSxL (which is good, but not great considering that you're now managing the cluster yourself).

Then, the third thing you start to ask is anecdotally, are people able to actually configure these file systems into the size of deployment that I want (which is where you hear things like "oh it's hard to get Ceph to 1 TiB/s") -- and that remains to be seen from something like 3FS.

Ultimately, I obviously believe that storage and data are really important keys to how these AI companies operate -- so it makes sense that DeepSeek would build something like this in-house to get the properties that they're looking for. My hope is that we, at Archil, can find a better set of defaults that work for most people without needing to manage a giant cluster or even worry about how things are replicated.

It's interesting, after my time at Amazon (8 years) -- I struggled to visualize decisions without a document, because you get so used to reviewing things in that way. However, it's EXTREMELY heavy-handed. People will give you comments on the structure of your document (to "raise the bar") instead of the content of the document, and often you can't get documents reviewed until they are aesthetically in the right place for people to digest them. Ultimately, I think it makes sense when you're at a large organization with high attrition (such that you need to keep track of all of the decisions which are made), but otherwise, it's probably not worth it to do a formal document writing process.

Interesting -- is that normally done with database updates + polling vs. something purpose-built?

What are some example use cases where having the ability for the database to push updates to an application would be helpful (vs. the traditional polling approach)?

You've totally hit the nail on the head. This is the real moat of S3, the fact that they have so much front-end throughput available from the gigantic buildout that folks can take advantage of without any capacity pre-planning.

Are you talking about getting metadata from many objects in the bucket simultaneously? You might be interested in S3 Metadata https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingM....

I generally see object storage systems advertise 11 9s of availability. You would usually see a commercial distributed file system (obviously stuff like Ceph and Lustre will depend on your specific configuration) advertise less (to trade off performance for durability).

You're totally correct, but these products also need to be specifically designed against these failure cases (i.e. it's more than just MTTR + MTTF == durability). You (of course) can't just run deployments without validating that the durability property is satisfied throughout the change.

I think that this is, to Andy's point, basically about simplicity. It's not that your business necessarily needs 11 9s of durability for continuity purposes, but it sure is nice that you never have to think about the durability of the storage layer (vs. even something like EBS where 5 9s of durability isn't quite enough to go from "improbable" to "impossible").

For example, in AWS, you can get a similar FSx for Lustre file system for just 11% more cost, which could be worth it to avoid the management costs of running your own storage cluster.