Supabase Storage v3: Resumable Uploads with support for 50GB files | Hacker News Reader

Supabase Storage v3: Resumable Uploads with support for 50GB files

supabase.com

188 pointscreativedg57 comments

Hello HN, Supabase Engineer here.

This release brings resumable uploads to Storage. If you users are uploading large files or uploading from a mobile network, this improves the user experience by continuing the upload from exactly where it was interrupted. As a side benefit, you also get progress events exposed in a neat way.

And we continue to use more of Postgres. We use it as a database to store object metadata, RLS policies for Authorization[1], the SKIP Locked feature for building a queue [2] and advisory locks for implementing a distributed locking mechanism [3].

Will be hanging around to answer any questions.

[1] https://supabase.com/docs/guides/storage/access-control

[2] https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock...

[3] https://supabase.com/blog/storage-v3-resumable-uploads#enter...

>you also get progress events exposed in a neat way

Yes! :D

hey hn, supabase ceo here

This release introduces a few new features to Supabase Storage: Resumable Uploads , Quality Filters, Next.js support, and WebP support.

As a reminder, Supabase Storage is for file storage, not to be confused with Postgres Storage. Resumable Uploads is the biggest update because it means that you can build more resilient apps: your users can continue uploading a file if their internet drops or if they accidentally close a browser tab

This implementation uses TUS[0], which is an open source protocol. We opted for this over s3's protocol to support the open source ecosystem. This means you can use several existing libraries and frameworks (like Uppy.js[1]) for multi-part uploads.

It also has some neat technical details, using Postgres Advisory Locks to solve concurrency issues.

The Storage team will be in the comments to cover any technical questions.

[0] TUS: https://tus.io/

[1] Uppy: https://uppy.io/docs/tus/

I was just going to ask if you'd considered tus for this :)

Kudos on the launch - great addition

credit to the Fabrizio and Inian on this one. I'm personally impressed how fast they were able to implement this since we shipped a major update to Storage last Launch Week.

From what I understand, the TUS protocol wasn't necessarily simple, but the TUS community has made it a lot easier with their node server which they recently converted to Typescript: https://github.com/tus/tus-node-server

Yeah the Typescript port gave a good starting point. However there were some missing pieces which we had to build into our server - https://github.com/tus/tus-node-server#extensions

We plan to contribute these back to the Node TUS server after launch week.

Thanks for working on such an awesome project and releasing such useful features!

Yet, what stops us from using supabase is not the set of features, but the state of the current apis. Last night I was evaluating the python sdk and some of the examples were broken, baseline features like RLS are unimplemented (https://github.com/supabase-community/supabase-py/issues/58), and progress seems slow (it seems to have been release two years ago and it is still in alpha). Even though supporting many platforms is boring (i.e. no flashy feature announcements) and costly (many FTEs are required for maintenance let alone adding features), are there any plans to bring feature parity more of the sdks to allow more developers to leverage this awesome platform?

client libs are one of our next focuses. we've been discussing internally how to do this. So far we have concentrated on the JS/Dart libs because that makes up a large portion of our users. Over the last 3 months the community has added (and documented) libraries for Swift[0] and C#[0].

We hope to continue making this a community-driven endeavor, so if anyone reading this would like to become a maintainer for one of the libraries (especially Python), please reach out. If we can't find maintainers we'll find a polyglot who can work on this full-time.

[0] Swift: https://supabase.com/docs/reference/swift/introduction

[1] C#: https://supabase.com/docs/reference/csharp/introduction

Will you consider supporting C++? I’d like to use the supabase APIs, specifically the real-time APIs, from my ESP32s and other C++ projects.

We don't have a lot of C++ experience in the team, but I'll share your feedback with the Realtime team - perhaps we can find someone in the community to help

> So far we have concentrated on the JS/Dart libs because that makes up a large portion of our users.

I once had a computer magazine publisher tell me they only published Windows content because none of their readers used Mac or Linux.

sounds like they were just getting started and knew the efficacy of focus :)

It sounds similar to how my ISP only offers 25mbps upload to go with 900mbps download because "none of our customers use any more than 25mbps".

Of course they won't use more than 25mbps if you literally throttle them to that speed. What a joke.

> We hope to continue making this a community-driven endeavor, so if anyone reading this would like to become a maintainer for one of the libraries (especially Python), please reach out. If we can't find maintainers we'll find a polyglot who can work on this full-time.

Are these types of positions paid?

yes, we pay maintainers. See "giving back" in our blog post from last year:

https://supabase.com/blog/supabase-series-b

This is great - the caps on image size / mime type per bucket is super useful

Yes, I agree!! Very very useful!

Awesome to see supabase constantly improving. Been using supabase for the past few weeks and have really enjoyed it!

I was a bit surprised, however, that there's not currently a good way to reference storage objects from my postgres tables. I found that the recommended way is to store the object's path (as a string) in the database. While that works, it isn't optimal as I'd like to enforce consistency between the object and the table referencing it.

I've tried referencing the id of the corresponding row in the storage.objects table, but (1) apparently the schema supabase uses to manage storage.objects may change, and (2) it still requires separate (non-atomic) operations - or additional triggers - for keeping things in sync. Using buckets (corresponding to tables) and folders with ids is another way to work around it, but still feels suboptimal.

Not 100% sure what the best solution would look like, but ideally the supabase client could emulate storage operations for objects "attached" to a given table record, and supabase (the backend piece) could implement them as atomic operations (e.g., uploading the actual storage asset, storing the necessary metadata, and updating my table row to reference the newly-created storage object; exposing a helper function to return the URLs for any storage objects attached to a record; etc).

Anyway, just a suggestion. Keep up the great work!

this is a similar issue we face with Auth. We want to give direct access to the data/tables, and at the same time we need some flexibility to alter the tables on rare occasions.

We've dabbled with the idea of offering versioned views which have a "set" interface (eg `storage_v1`, `storage_v2`). We're still debating all the ways that we might do it.

All that to say - we're aware that this experience can be improved and we're working on it.

Hi. I was checking out the pricing for supabase database.

It seems like it may be more costly than firestore, although you are offering postgres as the underlying data store.

I'm wondering if I'm reading that right.

When it says 500 connections, is that 500 clients connected at one time? Just trying to compare the pricing vs firebase.

Thank you

It's very hard to compare Supabase + Firebase pricing because it depends on which features you're using. For example, API requests are free on Supabase and paid on Firebase.

The 500 connections you quoted is for the "Realtime" feature (which provides multiplayer functionality). You're correct, this is 500 clients connected simultaneously

Thanks appreciate the reply!

Btw did you start working on supabase in 2020? I was looking at the YC catalog.

That’s seriously impressive amount of work if it’s only been a couple of years.

thank you. yes, we started in Jan 2020

Any chance we could choose to use B2 instead of S3?

Even if Supabase keeps the same pricing (though why would you) the folks at BackBlaze have had my respect for a long time and I’m happier when I can stay out of the AWS universe.

yes, it's planned. I believe the only blocker we saw last time we looked was a 1-hour downtime-window every week (not that always went offline, but that they could use that time for maintenance). Do you know if that's still the case?

If you’re talking about https://www.backblaze.com/scheduled-maintenance.html it’s not intended to affect file usage of B2 (the bucket functions may go offline for up to 15min, but ul, dl, and listing of files are fine. One related thing to be aware of: the official documentation suggests adding retry to every file upload as some uploads naturally fail (usually for load balancing))

yes, I believe that is the one. We've engineered the server for additional providers and so it should be easy enough to add (and we'd welcome any PRs)

one caveat - we are shipping as fast as we can right now so that we can have feature-parity with Firebase. It's a balance of "breadth vs depth" and right now we are going for breadth. We will definitely offer more providers on the platform but the resilience of s3 is beneficial for our engineers right now - it means that they can focus on features rather than devops.

Quick question; since you use AWS under the hood, is there any way to “bring your own aws account” for any of these things? (Example use cases: relevant if I want to use sns alongside your storage solutions, or if I have aws credits I want to use)

not yet, but this is something we will work on (probably next year)

This team ships, every few years, there is a company HN loves, few years ago it was CloudFlare/Stripe, Supabase feels to be in the same class. Linear as well.

Supabase and Linear do deserve the hype IMO and Cloudflare still does.

Stripe has gotten a bit more iffy for me, with some of their poor practices around frozen accounts

I get the iffyness, I suspect there is probably more to it then the public sees, for example it could be their underlying layer or something else.

Cloudflare and Stripe have just reached that stage where there isn't much to hype about. They're now dealing with problems that are gnarly, as opposed to "we just launched a storage service!" Not to diminish Supabase, Fly, etc. and their tremendous work, it's just a completely different lane of expectations.

To me its more than just release new features and improvements (though Cloudflare could do well improving the speed around this a bit, a nice cadence around services and a public-ish roadmap would be ideal IMO).

More importantly, I think Cloudflare, Supabase and Linear are innovating and really nice to use, generally speaking[0]

[0]: they all have edge cases and what not, and sometimes failures, don't get me wrong

I'd add Tailscale and Fly.io to the category, too. Well-written technical blogs are truly the way to HN's heart.

> Fly.io ... well-written technical blogs

I'm not sure I would class fly.io as well-written technical blogs. Well-written yes, technical no.

I used to think the fly.io ones were good, but having read a good few of them I've found they follow the same repetitive structure:

    - Very light on detail (fly.io blogs are carefully written to *sound* like they're giving you detail, but in reality it's all a 50,000ft view, and a lightweight one at that.  Given the length of their average blog post, they could and should do better.).
    - Constantly going off-tangent, whether randomly talking about sandwich fillings, food-types, or just a paragraph with the author's rant about how they dislike a particular technology.  The first couple of times its cute, but after that, not so much ...

Would you mind sharing specifics about the first bullet? What's the last post you read you thought could have gone deeper? We definitely want to go deep, some of our posts are better than others.

tbh you seem too focused on making HN happy, makes me wonder about your focus

I appreciate it. I do struggle with focus. Thank you for giving me a nudge.

In his defense, HN is his target audience so making them happy is important too.

fwiw Kurt, I love reading Fly.io posts. your "globally distributed postgres" post especially[0], which had plenty of depth and no sandwhiches

[0] https://fly.io/blog/globally-distributed-postgres/

Yeah, supabase is great. I think if they could get the auth story a little more polished (it "works", but its definitely weak/buggy in some aspects compared to alternatives, and hard to use), and made some advanced use cases with Prisma easier to do (or better documented) since that's such a common pairing, they'd be even easier to recommend.

Right now I still use Supabase because I think they're a solid "Postgresql as a service" offering, but they're just SO CLOSE to being so much more than that. They have all the pieces, they just don't all work quite the way folks expect. They'll get there, I'm sure.

Supabase looks so good and keeps getting better. I started playing with it a couple months ago and could see myself loving it. One of my biggest problems right now is inertia: I’m comfortable with my existing tools (RDS, S3, lambda) and thinking about building a new product while also adding new tools is a tough pill to swallow. But the product sure makes a strong case!

Resumable uploads is a big deal for what I’m working on. But I’m also blocked by another issue, maybe a Supabase engineer on the thread can weight in. I’m using ffmpeg on AWS lambda to transcode media files after successful upload to S3. Lambda makes it easy to add ffmpeg binaries via lambda layers. Supabase functions don’t seem to provide any such option. Is there a way to solve this within Supabase as it exists now?

I’m also curious about function timeouts. If I have a file conversion that takes a couple minutes, is a Supabase function the right way to go?

That is the same inertia we see while selling to early stage company.

I also see quite some people build out in supabase and migrate to AWS, especially if they want SOC2 or some compliance certifications. The Compliance automation tools work really well with known cloud providers.

Just as a note, Supabase received SOC2 Type 2 compliance last week (we've had Type 1 for a long time already) and now we're working on HIPAA.

I was trying to do the same thing (ffmpeg to do file transformations on upload) and couldn't figure it out with Supabase functions. After poking around in the Deno discord for awhile, it seems that ffmpeg.wasm for Deno just isn't much of a thing yet. I was able to get it working with Vercel functions, but it was a total nightmare and the runtime made it a non-starter. I landed where you started: just do it in Lambda!

This was almost exactly my trajectory. It’s a shame cause Supabase is so close to being a slam dunk for me but this complicates it. Are you using S3 for storage or were you able to stick with Supabase and use upload complete callbacks to trigger the lambda?

that's great feedback. Our Functions team has seen this - we'll see what we can do

Thank you! This was one of the two major blockers for me, the other being lack of presigned upload URLs and I see that landed with this release!

> Is there a way to solve this within Supabase as it exists now?

No, not right now. However we do have a planned path to this. The 3 features we will release are:

1. Custom execution limits/timeouts for Edge Functions.

2. WASM support in Edge Functions

3. Storage Webhooks, which can trigger an Edge Functions

Features 1 & 2 will be possible with the Edge Runtime we released yesterday[0]. Feature 3 is already possible using our Postgres Database Webhooks[1], and we will also provide additional support for this directly in the storage server

[0] Edge Runtime discussion: https://news.ycombinator.com/item?id=35525222

[1] Postgres Database Webhooks: https://supabase.com/docs/guides/database/webhooks

Supabase Launch Weeks are so friggin cool fr.

Looking through the supabase storage docs, I wasn't able to identify if storing to local disk was an option? I know it launched with S3 as a supported backend, but I'm curious to know if this ever expanded.

Or perhaps supporting Minio as a backend, which I could run on the same machine.

Hey, Supabase Storage engineer here

Yes, we support S3, Minio and any S3 compatible storage servers. As well as the File System (local disk) - This is for the self hosted version.

When using Supabsase managed version we are storing your files in a S3 bucket managed by us

In the future we might add the option of "bring your own bucket" to Supabase Storage and we'll upload the assets there.

The bucket could be located in a minio server of your choice, for example.

Is this what you were looking for?

The local disk option when self hosted is what I was looking for. Couldn’t see that mentioned in any of the docs.

This looks great! Thank you

Reply on news.ycombinator.com