Hacker News Reader

Comment by vlovich123 | Hacker News Reader

I’m an engineer on the R2 team. Happy to answer any questions in this thread.

1. TFA goes R2 clients will be able to enable caching (in the near future). A couple questions:

1a. Cloudflare Cache is free for users (up to 500MB per file for unlimited files), but if R2 uses it transparently, are reads from cache metered per R2 pricing? And what about the file-size limits that apply to free/pro accounts?

1b. How is strong-consistency guaranteed despite caches in front of the bucket? I guess cache-invalidation can be done through Durable Object (DO) Alarms or some such? Or, would this guarantee no longer hold?

2. TFA mentions in the passing that R2 utilizes DO (for presumably metadata / journal?). Cloudflare once wrote in detail about how DO is used in a pretty novel way to power wrangler tail, and so I was wondering if there's something novel about R2's use of DO, too?

3. In-Worker API: Are customers metered for both In-Worker API (duration and invocation) and R2 (Class A / Class B)? If so, that makes Class B operations almost twice as expensive than S3 (excluding egress).

4. Any ETA on "Public buckets"?

5. Has Cloudflare rolled out newer server designs just for R2 (aka Dropbox's MagicPocket)?

6. Is serving media content (say, audio and video) an acceptable use-case for R2?

7. Any plans for Kinesis Firehose (a managed-ingestion pipeline) esque product to front R2?

Thanks.

1a. I don't think we have anything to share with regards to the logistics of putting cache in front of R2 or how that product will work.

1b. Unavoidably as you guess you'd be giving up the consistency. Presumably the consistency guarantees would be part of the cache-control header you associate with the object. Cache purging would probably work similar & we have exciting upcoming news about major performance improvements for that. But again, since we don't have any idea of what that product looks like, hard to say.

2. There is. I have a bunch of technical blog posts written but we decided to punt on providing that detail for a few months just to give everyone some breathing room. R2 does have one important extension we got the DO team to build just for us (not 1 instance of the code instantiated). That piece will only be available for R2. R2 is also driving performance improvements & API improvements to DO that will almost certainly percolate out to external users.

3. The equivalent model would be AWS Lambda + S3 so I think we're significantly cheaper (+ Workers is faster than AWS lambda - R2 is still in open beta so not useful to compare perf against S3 yet). Yes, if you're using it to emulate public buckets, then it'll be a bit expensive (depending on egress charges you might otherwise face)

4. Soon. It's very high on our list alongside pre-signed URLs

5. Nothing to share at this time.

6. Yes. From Matthew Prince aka eastdakota [1]

7. Nothing to share at this time.

[1] https://news.ycombinator.com/item?id=28682885

If I remember correctly, there was some discussion around R2 acting as a proxy in front of s3. So, if R2 does not yet have the asset, you would automatically pull it in from S3, and from then on R2 would serve it directly.

Is that still on the cards?

Yes, still in the cards. Hoping to release this before GA - it'll be a part of our migrator.

Not a Cloudflare employee, but: Yep, Cache Reserve is in private beta https://blog.cloudflare.com/introducing-cache-reserve/

Cache reserve is a similar concept but a bit of a different product (although some people might be happy with the no-code approach that enables). Migration is a bit of a different thing.

Thanks for the awesome product! I have a few questions:

1. Are the rate limits for all users on a given bucket, or by source IP as some other providers do?

2. What is the plan for the rate limits post beta?

3. Do you support the x-amz-content-sha256 header. i.e. If a put contains that header and the content hash doesn't match is it rejected?

4. What does it mean to support pre-signed URLs? Can't a client pre-sign a URL that anyone else can use? My understanding is they are the same as normal usage from the server's perspective, just the key holder signs the request, and the other person actually makes the request.

5. Why is the read limit smaller than the write limit?

I'm also an engineer on R2 team.

1 & 2 - We're reserving some ability to rate-limit during beta as we figure out how best to scale our systems under load. I'll defer to vlovich & greg-m for post-beta plans.

3. We do support that header. Mismatching content hashes will be rejected during write. If you ever experience otherwise, please report it.

4. We only support sigv4 auth via header currently: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-u...

With presigned urls, we will support sigv4 in url parameters https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-...

5. See #1. We'll be improving the throughput of both operations, but currently reads have higher throughput than writes.

Thank you, that makes sense.

So presigned URLs that use auth header are fine (We use those extensively in Peergos, as presigned with an auth query parameter isn't cached by browsers as the query string will change with subsequent requests).

On the subject of sigv4, are you going to allow longer than 7 day lifetimes for presigned URLs?

Any advice/guides on how to get this working with Rails ActiveStorage?

Should be the same as any other S3 backend. We don’t have examples for ActiveStorage per se but we have boto3, AWS JS SDK, and rclone [1].

[1] https://developers.cloudflare.com/r2/platform/s3-compatibili...

Does the worker api support multipart uploads?

Not yet. The current in-worker api is described at https://github.com/cloudflare/workers-types/blob/master/inde...

Reply on news.ycombinator.com