BunnyCDN has been silently losing our production files for 15 months | Hacker News Reader

BunnyCDN has been silently losing our production files for 15 months

old.reddit.com

180 pointsspeckx47 comments

I tried using their Magic Containers product and there were issues that showed a lack of attention to detail as well.

It's supposed to scale globally (magically!) but I found multiple cases where particular nodes were problematic and the health checks didn't detect them (in fact to start with the health checks didn't even work properly if you had multiple containers, they did fix that). The support was quite slow too, after finding multiple product issues they'd escalate to developers and then come back a month later and ask to retest, but some of this took multiple round trips. I was only using this on a side project, but definitely wouldn't consider them for anything critical, even if they are quite cheap.

Reply in thread from Bunny: https://old.reddit.com/r/webdev/comments/1sglytg/bunnycdn_ha...

Handwaving of professionally agreed upon SLAs as an edgecase and providing poor customer support; thats a paddlin

Also a user of their CDN, storage and container service for about 4 years.

My experience was more positive. One time I had a minor issue with their storage where I couldn't replace a file or something. This was fairly early after the product launched. They fixed it and gave me free credit for reporting it.

If the story is true, remaining a user seems…terrifying?

Because they fixed a minor issue many years ago?

I believe "the story" was intended to refer to the horror story in the main post rather than the minor issue described in the comment above.

100%

That's scary. I'm in the process of moving all of my services to Europe, and I had considered BunnyCDN, but after this I'm not so sure anymore.

I also tried Hetzner Object Storage. I love Hetzner, they're great, except for their Object Storage service, which is completely unreliable (errors, slowness, etc.). I'm surprised that Hetzner still hasn't retired that product until it's properly fixed.

My last chance is with Scaleway. Xavier Niel's products are always good, so fingers crossed...

I guess you may have alot of files, but to me object storage is so cheap I keep copies on Aws S3, wasabi, r2 and a 16 TB HDD on my hetzner server.

Admittedly 4 might be too many. But at some point I switched to r2 for the free public egress and deleting one of Aws or wasabi has never been a priority and I don't want to do it without putting in the time to quadruple check I'm not deleting anything important.

But at the very least id have 3. I'd hate to discover that my S3 was blown up in a war and then copying everything my HDD was the last straw that pushed an aging drive over the edge.

Bunny's CDN is great. The issue in the post is about their file storage service not the CDN.

For my storage needs in the EU I'm using Upcloud's object storage. Very happy with them. Then a Bunny cdn zone if I need to share the files publicly.

There is also OVH (not affiliated, just happily using their VPS), but I would consider switching to 2 providers, as any provider can have data loss or just lock your account at anytime for any reasons. 2 providers with free egress so you can easily replicate data between them.

The classic rule is 3-2-1:

- Three copies

- Two types of media

- One off-site or offline

>My last chance is with Scaleway.

I have been using for sw for object storage for a while. I haven't had a a issue with them with object storage.

only problem I had was with a another product called cockpit(they ship metrics about object storage there), which they bundled with object storage product, which cannot be disabled.

This month I had a 12 euro surcharge, because they enabled something in cockpit by default a while ago and suddenly start to charging for it.

Scaleway is a disaster in terms of handling payments.

At some point there was a bug in their payment system and I couldn't pay an invoice. The payment failed the first time due to lack of funds on my account (this initial failure was my fault). Once my account had funds, retrying failed with a vague error on Scaleway's side.

I reached out to support, and it was a known limitation: if your initial payment fails, you an only retry with a subset of their supported payment methods. I didn't have access to any of these payment methods, and I made it clear that I could pay, but only with the payment methods we'd been using thus far.

They kept sending me emails reminding me of this overdue bill and threatening to send collection agency, all while I kept exchanging messages with support complaining that their system gave an internal error.

The bill was, by the way, for less than €10. But a collection agency will charge you hundreds for the "collection" service.

They were aware of the error in their payment system, but it seems they had no intention of fixing it. After months of back and forth, they just cancelled the invoice.

The amount of time and stress in dealing with was ridiculous.

On LowEndTalk: https://lowendtalk.com/discussion/comment/4767400/#Comment_4...

Interesting timing given the post yesterday about someone switching to BunnyCDN from cloudflare

That is interesting.

To be fair the Reddit user account is 8 years old. Not conclusive but does suggest it may be organic

you can purchase reddit accounts with age/content.

Not a great way to handle the issue originally, but I think the CEO's response is more than fair.

I haven't had to contact Bunny's support so it's a bit disappointing to see this type of runaround from the support, but as an actual issue it seems pretty minor in the end.

I've been using Bunny for ~four years after looking at video hosting. In the four years we've saved approximately 50 000€ compared to if we'd gone with Cloudflare video streaming service and it's just been rock solid for us.

The CEO's response is alright, the VP of engineering's response is a bit more worrying in how it mainly tries to downplay the issue.

I’ve used bunny for a few years … happily. I wonder if this is a bug due to some meta data of the files like the names or something. Very weird. Good thing you had metrics to catch it.

I upload all object storage stuff to bunny for live but also to backblace for backup.

I’ve always wanted to implement fail over client side for any asset over to bacblaze but seems like a lot of overhead

Wait...this has been going on for 15 months without resolution or recompense and you haven't pulled up stakes and moved to almost anyone else? I get "it's a lot of work to move" and maybe "we hope they'll figure this out so we don't have to move", but in my world that excuse runs out waaaaaay before 15 months. The people I'm accountable to would have hauled me out back and put me out of their misery after, maybe, 6 months on the outside.

> ...but in my world that excuse runs out waaaaaay before 15 months.

I expect the combination of

  Yes, we should've migrated away sooner, we never had the capacity to do so and hoped Bunny would just get their shit together.

and

  The loss rate isn't enormous in percentage terms, but it's consistent and ongoing.

means that detecting and dealing with the loss is substantially less work than moving away. [0] I expect that Management is fully aware of what's up and is making the call here.

[0] "Just" add a retry if the post-upload verification step fails! Sure, it's slower, but it works, right??? mournful sob

I'd guess that it hasn't become an issue with any of their clients. If any big client had been impacted once, they would have sped up moving. If a big client had been impacted twice, they would have moved already.

That's fair, and I agree (hope?) if it was customer impacting there would be a miraculous allocation of scarce resources to move platforms. But in my world I'd get asked "why are you still screwing around with a vendor that clearly sucks" even if it was for a non-prod playpen of no consequence. YMMV.

I'm a very grateful bunnycdn customer. they are great, lovely UX and great performance and price. we use them to store our image files and other documents.

what's bunny? a CDN by Hug Hefner?

A time for scrutiny...

Reply on news.ycombinator.com