Hacker News Reader

Comment by ochoseis | Hacker News Reader

This does sound nice, but the article barely mentions a pretty major limitation from the release announcement[0]:

> it doesn’t support writes in this first release, and in the future will only support sequential writes to new objects.

[0]: https://aws.amazon.com/blogs/storage/the-inside-story-on-mou...

View on HN

It wouldn't be an AWS announcement without a whopper of a caveat, it's like the Amazon equivalent of Google canceling things.

To be fair it sounds like this isn’t for that use case.

AWS really does have its own quirks as a company.

- The tools feel fragmented, not aligned and often don't compose well

- Always sharp corners (Billing, ops, you name it)

- Products in various state of usability / maturity, sometimes a product is launched by barely usable.

I've never worked for Amazon, but if what is said about amazon is true, then the culture is reflected in their products, teams siloed owning their own services, then it does make sense that their services have those quirks.

That comparison seems unfair... AWS has a habit of launching early and iterating, and they almost never shut stuff down. In this case it's a filesystem implementation of a HTTP API, so the capabilities reflect the main API. Deciding to build and launch reads before writes isn't really a bad decision, it helps those who have the read use case try this out and give feedback immediately.

> In this case it's a filesystem implementation of a HTTP API

When you put it like that, it makes me wonder why they don't support WebDAV.

I think there’s a bunch of gateways available already. The example use case for this seems to be mounting a 10 petabyte “hard drive” with CSVs to a reporting machine.

>only support sequential writes to new objects.

"Doesn't supports writes in the first release" is a letdown, but the latter part above seems expected. I suppose you could abstract what would have to happen to fake arbitrary seek() and overwriting portions of a file, appending, etc, but it would encourage things that wouldn't work well.

Partial writes would need to buffered and flush()ed. That is not unlike other file systems, until you fully know it is persisted - or am I missing something?

That you can't update an S3 object. There is no equivalent to open() on an existing file, seek() to some position, and modify a few bytes.

That S3 storage is almost immediately consistent on a planetary scale is pretty impressive already. This would make it a lot more complicated to do that.

OTOH, you can always engineer around such limitations by storing smaller objects you can completely ingest and store on a single operation.

It’s not supported by S3 to partially update an object; you would have to completely replace the entire object. While they could do some magic under the hood to make it seem like it works, it’s better to just reject these operations and let the application developer decide what’s best.

Maybe I'm missing some context here, but can't you generate X number of presigned URL's with your chunk size and if you provide the chunk ordering it's automatically done on S3 side, but I though you could manually implement the chunk order assignment also.

I would suspect the team making this tool is not exactly the same team maintaining S3...

So adding "server side operations" would require work from the other team to be added.. which certainly could be done if they decided it was needed.

As a first MVP release this project seems neat for read use cases.. and feedback from the community can help ensure read operations are solid while they are also working on the write side -- which they admitted was lacking in this initial release, but want to improve.

For that use case, they have EFS. If you are reading from one of more S3 blobs and gradually building another blob, you can keep that blob on EFS and, when finished, dump it back to S3.

My understanding it goofys and s3fs support these types of operations?

Still, a win for reliability / performance of whatever aws will be supporting

> it doesn’t support writes in this first release, and in the future will only support sequential writes to new objects.

To me it just sounds like Azure Files with less features. You can just mount a share on any machine supporting NFS or SMB or just use the REST interface. [0]

[0] https://learn.microsoft.com/en-us/rest/api/storageservices/f...

Fair point, defs should be mentioned by any serious article.

I have use cases where forbidding writes is a feature, and if it means more stability and performance it’s a trade-off that’s worth it to me.

Reply on news.ycombinator.com