Year old startup overloaded GitHub – Incident report
lovable.dev
lovable.dev
> Incident report for the GitHub outage on January 2-3, 2025
Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application to be up, instead of taking full responsibility for it.
Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bother to check if there are support tickets open with the customer, they just go and disable their access.
A title like "GitHub caused our outage" would still make it clear the downtime wasn't the direct action of anyone on the team, yet still take responsibility over that it happened. Instead, labeling it "Incident report for the GitHub outage" just seems like straight up blaming someone else.
And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault.
No, GitHub won't ever take responsibility for what you have between you and your projects/apps/companies users. Nor did they do so in this case.
If I use service X for doing Y, and I write an email asking if it's OK that I upload 1000s of files every day, even if they say OK today, but then next week turn around and say "Nah", I'm still responsible for my users, who trust me and my team. Service X has no responsibility towards those users at all. It sucks though, I agree with that.
> And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault.
Besides DNS and BGP I suppose, what services are "unavoidable" exactly? Git hosting isn't some arcane distributed network technology needing years of reading/experience to understand, the CLI ships with a web ui you can basically copy-paste to have your own Git hosting.
I'd say you are responsible for everything that you use and depend on. And if you think "Ah, I'll just chunk 10K repositories at GitHub a day, they say it's fine" and then don't have any plan in case GitHub suddenly says it isn't OK, you are responsible for the falloff if shit hits the fan.
As another example, let's say you build a house in a hurricane-prone area. It's your responsibility to ensure the owner buys hurricane insurance, as mandated by law. It's not your responsibility to build a nuclear-bunker-grade house that is impervious to hurricanes. It is easy to point the finger and say "you should have thought of that", but in practice it is easier to deal with such catastrophes as they happen.
As for the repositories, there are no good SLA terms for such storage. Nobody offers this service because it's expensive to offer. Not on this scale. So if you need it, you have to invest a lot in hardware or colocation or specialized clouds. Plus admin. A whole datacenter, and suddenly you're an ant vs three goliaths.
Your customers don't care if it's the service's fault and will blame you.
unless you're a super important partner, the people on call might never have heard of your little app and just decide that it's the only safe thing to do to protect the reliability of the system.
Who relies on support people to determine the basis of their business when it’s obvious that they were concerned with the high usage rate and that it might cause problems for their customers?
As for the OP, they should have conducted load testing and implemented rate limits on their end, rather than blindly relying on someone’s word that GitHub was ready to handle all their product's load for free.
https://news.ycombinator.com/item?id=42646297
And now this post with an exaggerated title. Seems like they're shilling and trying to make Lovable sound like a product with such huge traction that it "even brought Github down". They keep making outlandish claims on social media too, like reaching $4m ARR in 5 weeks etc. This company is very suspicious.
Looking more into it, Henrik Westerlund works on "growth" at lovable (via LinkedIn) and posted Lovable to Product Hunt https://www.producthunt.com/@henrik_westerlund. So it appears they are shilling.
For 8 hours, no one is aware the service is down. It then takes ~3 more days to fix it. One of their first decisions is to 'make as much noise as possible on social media' (?), and every step seems to create additional problems (corrupt repos etc.) Nothing appears to be well thought out, the blog post reads like they weren't ready for this at all, panicked and chaotic decisions without understanding the tech stack on a deeper level (race conditions, rate limits etc.) Not a lot of confidence in the team behind a project that looks like nothing more than a glue between an LLM and a storage backend.
> Seems to onerous.
I would only say that about a non-critical personal project.
Getting a flag like this hardly “overloaded” anything. Alerts for these things usually trigger well below any actual risk to the system.
They probably should have had a backup location from day 2 though, I agree. If nothing else, in case of a GitHub outage.
Maybe I read the landing page very wrong, but it seems to be a "app building toolkit" of some sorts? Not just "creation of git repos".
They could have made the GitHub repository creation happen when the user does some action, instead of at the stage of "create app" which probably every single user does at least once, even people with no intention of actually building apps.
Or better yet, offer their own viewer for Git repositories they themselves host. It's not overly difficult, and the `git` CLI tools even ship with a web UI you can take inspiration from.
Their product has change tracking which is clearly powered by git repos on GitHub.
So again, I ask, for something that is a SPOF, why rely on a third party?
Most folks seem fine with it, at least it still lives on like normal as far as I know. I think engineering principles flew out the window a long time ago, all people care about now is shipping as fast as they possibly can.
My assumption is that lovable is not free, so using a cheap or free service while it is taking money from their customers fits squarely into the categorization of poor engineering and possibly incompetence.
"How can somebody who does X just do Y?", implying that Y is somehow bad or wrong in the context of X, or contrasts with the experience level implied by X.
Maybe it's true of the subject. Maybe they should have known better. But I think it's beneficial to take a step back and view it from the perspective of readers: Many people - even people familiar with X and Y - may have no idea why Y is bad in the given context, or, more relevantly, why this should be obvious, as you imply by using the word "just". I think there's a lot of benefit to be gained by trying to observe yourself, and notice when you are writing in this style, and add more context.
I suppose they did respond pretty fast, but if I were them I'd have liked to have had the S3 option in my back pocket earlier. Maybe I'm just being too risk-averse here...
>We're currently investigating issues. Please stay tuned until this error banner has been updated.
When I try to create a new project, it says "An unknown error occurred with the code sandbox"
Something about S3-backed repos didn't work out?
UPD. "Under maintenance: Lovable is currently not able to reach the cloud provider for our previews, fly.io" Now it's fly.io's problem, not GitHub's