Salesforce Global Outage
status.salesforce.com
status.salesforce.com
Seemingly meaningful IDs
Search
Region filter
Email update signup
Predictable URLs for instance status so they can be deep linked in runbooks
What appears to be the actual live instance status.
What appears to be the actual live service status in each instance.
An update log with frequent detailed updates.
So yeah you're exactly right, the snark is not deserved if you ask me, and I'm 82% snark.
I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach.
Smaller is, in fact, better, because it's easier to operate reliably.
"Simplicity is the ultimate sophistication." -- Da Vinci
Depending on your time and appetite for tinkering with all of this, it's not hard to imagine a home setup that fails over to a cheap Hetzner or DO VM. A manual failover at the DNS level isn't overly complex, and could be scripted.
Keeping a database in sync between home and the instance might be simple or more complex depending on needs, but would it really be that hard to have Claude help you setup a replicating Postgres server? If your database (or data files) are 1 gigabyte and don't update that often... maybe just rsync it every night or something
There's a thread you and others are pulling on here, and we need to pull it. Hosting doesn't have to be the domain of the big vendors anymore.
I think people forget that those large environments are there for a reason. To make sure the service stays up in the face of problems outside your own control.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
My entire point is that you have no redundancy in your system and you also aren’t big enough to have any pull with the vendors who can fix these types of outages so you’re basically at the mercy of your providers with no recourse.
That’s why these systems are built the way they are.
And generally four nines is considered the gold standard these days. I can tell you for sure that both Netflix and Ebay would lose money anytime they drop below four nines because I have at some point been responsible for both. You’re correct that Reddit has a lot more leeway and outage time before they start losing money but not that much leeway.
"What are you optimizing for?" is always an important question, as is "The Five Whys."
Over multiple ISPs, so far internet outages for more then a few minutes is very rare (though the few minutes would make me not want to host something requiring high availability; and a cut cable is really annoying because there simply is no quick fix), power outages even rares, I experienced 3 in 40 years, and the longest was 6 hours.
Ignoring for now how you are synchronizing the database and filesystem, and how doing so may well result in your duplicate experiencing the same failure as the original, you can maybe recover from a small class of availability issues that could knock you out of an SLA.
But that assumes you can get online and can fully orchestrate the transition within less than 53 minutes of it starting. Including the time you took to become aware of it. And including the time to diagnose and decide that a switchover would resolve the problem. Including the time it takes for DNS caches to expire and point to the new host. Including the DNS caches which may ignore your TTL. And including all these things again when you switch back.
And assuming, of course, that it doesn't happen again for a whole year.
Built how? Because I can state with confidence that I have cleaned up a ton of failed upgrades/zombie terraform deploys of these serverless kubernetes wonders that followed every best practice under the sun, and these things are not really considered even moderately reliable (as designed by imperfect mortals under real world conditions), meanwhile professionally, people who stand to lose a lot of money should their systems go down generally operate systems whose architectures were designed decades ago, are generally horizontally scaled monoliths, and are extremely conservative in software choice.
Also downtime often is no biggie, as long as it's planned and or don't lose (too much) customer critical data.
Like nodobody cares if your test db cluster goes down for the weekend. We even shut down our db instances to save money.
Even if a solar flare takes out an entire continent or two I think it's safe to say that the bittorrent network will still be running in some form. Can you be so certain about any given SaaS product?
(I hope you'll agree that the middle east outage is a true outlier)
Salesforce is not simple. It's wildly, overly complex. It's amazing it has any 9's at all and not 8's or 7's. Salesforce offers three 9's, which allows for 43 minutes downtime per month. The current outage is at 8 hours (and counting) so Salesforce is now at 98.9% uptime for the month - there's an "8" in there now. Not good, but considering the complexity of Salesforce, it's still kind of amazing.
It turns out business environments are wildly overly complex.
But the GP is essentially correct - there is a 2% of salesforce that could be built run and keep 80% of salesforce users happy. Except that you could not charge enough to be able to advertise on F1 cars and take SVPs out to dinner.
So you could not actually make 80% of them happy - they would ever buy it.
Yes, you largely do - they’re the commits that get rushed to, and through.
This take that showstopping technical debt is unavoidable is very new, and will age like milk.
No it's not. The push and pull between shipping and paying down technical debt is as old as there's been software to sell. Sales has been selling features that don't exist quite yet ever since they've been talking to customers, and engineering has been pushing back on implementing them yesterday since there's been features to implement. Showstopping technical debt is merely a side effect of who wins that argument in a given org.
My point is the technical debt is stopping the show way more often.
Not “this never existed before selling”,
but “we never had the team in place who could do this right in the first place”.
You can quibble about who is responsible, but the fact remains.
I think that the best you can hope for is a CI process that verifies a minimum quality standard. There are I am sure plenty of CI funnels that measure cyclometric complexity, but “rewrite this because it works but is over complex” is a hard sell, compared to “we need to rethink the whole architecture, give me three months”
I remember Cisco before iOS used to have hundreds of branches for their router, one branch for each major customer that was demanding specific features. It was unmanageable, but that's what you needed to do to win those "enterprise customers".
It also turns out customers aren't very good at articulating their needs and putting them into a cohesive vision of the product. But they sure have specific demands to get stuff in. I'm not blaming the customer, this is just how this world works -- All of the "enterprise software" apps are extremely complex with hidden knobs and weird behavior that was pushed in by a customer twenty years ago all over the place.
Ouch. I heard of a company in my home town (small B2B service provider) doing something similar - they paid well but I didn't think it was worth it.
What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale.
In contrast, something like a bank or social media isn't really reducible - every user needs to be able to interact with every other user in a consistent manner.
So running a midsize bank's backend which processes 10m transactions per day, might be as if not more complex (all consistent, repeatable, and must never fail), that having a product which is a 10-10k org's IT infra replicated a thousand times.
And yes, lots of people have worked at banks and other fintech companies of this scale, including me.
I am not an expert, as I never worked on the 'core' systems but I know folks who did, and everyone told me there's an arcane database monolith that sits at the heart of these, very expensive and exotic big box SW & HW (at least for us unwashed rubes used to EC2 instances)
This is only true if you exclude problems like “finding a CDN node from the device,” “managing congestion,” etc. as part of the problem statement
They do though! They mostly try to avoid it since hitting the network to serve any kind of latency would unacceptably increase latency, but you wildly underestimated the amount of complexity there is to running a CDN.
the OP mentioned 1B requests/day, where there are systems handling 1B requests a second.
Also, mono-tenant architectures is no golden bullet either. Such architecture (often coming from a formerly on-premise product that was SaaS-ified) can easily become hell to operate as it multiplies the integration points (DB parameters, URLs, allowlists, etc).
It's also quite wasteful in terms of resource utilization and hosting costs.
Flexibility and abstraction come at a high cost. It doesn't really matter though, world domination at all costs is the name of the game.
AI coding changes this a lot.
I think AIs / agents (and more importantly how we are learning to use them effectively) may help in that regard, but only if they are able to manage that complexity. This'll depend on context window sizes, their ability to explore a codebase, and how well their operators can provide relevant information.
But that's only what they can consume (so codebase, documentation, etc), on top of that are the people that work with / for these systems for decades and who know a lot about things outside of what's written down.
The level of scale and complexity a big tech SRE has to deal with on a constant day-to-day is a very imbalanced proposition. A lot of people, in my experience, are not fully comprehending.
You have to be a jack-of-all-trades and a master of all.
One thing that I find interesting is that they launched their platform language Apex (a sortof subset of Java) in 2007 when TDD was the hot new thing, so TDD is baked into the platform - your Apex code must have at least 75% test coverage, and the tests must pass, before you are allowed to deploy to prod.
They leverage that test coverage when they are upgrading the platform - they have an internal process called The Hammer where they run all customer-created tests against customers own unique configs on the current platform and then again on the next version of the platform to see if any customer tests are being broken. Look it up, its really interesting.
I'd argue they (and many larger, older, established etc systems) may not have, but it's part of how it works so while it may not be the best it's the one that is working right now and earning them money - working (and earning) software always trumps correctness etc, in practice.
> We're no longer pursuing restarts as a path to remediation.
Oh you have
> We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue.
At least it didn't fix the problem so they can actually start finding the real cause.
> We're no longer pursuing restarts as a path to remediation.
Why isn't the AI they sell telling them what's wrong? Why do they need to take shots in the dark to "see if that resolves the issue"?
"Oops, we forgot to tell it that this is the hyperscaled Salesforce production environment and that its choices need to project competence and consider brand embarrassment. WILLFIX"
So hopefully it's not done often.
It's a joke, but like, after nearly two decades of engineering I have something break on me, due to updates. I figure the updates broke it, call the vendor, and they go... did you try rebooting it again?
Rebooting it a second time fixed it.
Flashbacks to "Thank you for calling Three-Ten-DELL. Have you tried turning it of and turning it back on again?"
https://www.salesforce.com/dreamforce/
Sept 15-17
They've gotten a little better in recent years.
Something I wonder about is whether SRE responses were delayed due to having to be emergency-change-approved because Dreamforce was on. I don't recall a global outage ever occurring during a change freeze when I worked there, so /shrug.
Also see: Oracle
Fix: Rolling some unspecified fix they proved in testing out over the fleet seemingly very slowly (After their earlier attempts to roll something out faster failed).
It’s optional but everyone uses it. And it was flaky for an hour or so, like two months ago.
Layoffs are usually short-term money-savers at long-term cost, but hot damn this is the shortest short term I've ever seen.
Scroll down. >_<
Isn't it actually amazing, and not "the most salesforce thing ever"?
⎿ You've hit your session limit · resets 2:53am (48°52.6′S, 123°23.6′W Etc/GMT+8)
/upgrade to increase your usage limit.LWC is worlds better. And the local tooling with the cli and VSCode extensions is miles better than the old Eclipse/Sublime FMT days.
Now, don't talk me about VSCode Extensions. This is the perfect example of an awful dev experience. apex-jorje-lsp.jar with a JVM to parse Apex taking GB of memories, extensions taking dozens of seconds to load (when they load) ... In fact, the only decent LSP is aer, a simple decently working Go binary rather than the monster Salesforce shipped. The one good tooling Salesforce built in the last 15 years is, to some extent, the SF CLI - which came after the `force` CLI from the same guys who built `aer`, anyway. And nowadays, people can use that with their preferred editor from Zed to Vim with shortcuts from built upon the SF CLI.
So no, Salesforce didn't do great with tooling, they just did the bare minimum waiting on the (small) community to give them the right ideas.
I think the biggest difference was around providing security and stability barriers between front-end components on the same page, with the intent of allowing you to compose a page that contains your own components and those of other third party applications you've installed with guarantees about how they can (and can't) interact. Not sure they couldn't have tacked that onto another framework, but it comes with enough trade-offs and compromises that I'm not sure anyone else would have wanted to upstream it, so they would have been forking something anyway.
Aura wasn't much fun to work with, was never really feature complete, and not advocating for it... but it actually kind of made sense if you thought about front end with the context of how salesforce did security and multitenancy in mind.
And the tooling is heavy, but I’ll repeat, it’s all miles better than it was back then.
We traded janky and sparse for a full suite of pretty great tooling that can be a resource hog.
Remembering the relative simplicity of the old stuff as “better” overall is just rose-tinted glasses.
(Nowadays any of those need to fit in an "agent dropped all tables. Apologized" moment.-)
Please VCs stop with the AI FOMO and find a few good startups to just go destroy Salesforce and give folks a simple inexpensive replacement.
Where customers do use the features and integrations it’s often a giant mess that needs a whole separate ecosystem of consultants and “partners” to get the thing working and maintaining it.
Salesforce does a lot more than that
But mostly it's a mix of the integration network effects you mention, cost of reimplementation if you want to leave, and the good old "nobody gets fired for buying IBM" dynamic.
Data querying for creating marketing campaign audiences or generating some reporting reporting.
Little development for landing pages, forms, or custom integrations.
Multiple channels (SMS, push, email) & message templating.
And many marketing stuff like AB testing/fatigue, etc
Those just, obviously, can’t do almost any of the forty million serious things that Salesforce does, and real businesses do need.
Now I'm just glad I'm not responsible for this fire
Doing data engineering work with Salesforce data is an exercise on archeology, psychology and organizational politics.
Slop is basically the ontological and teleological philosophy behind Salesforce very existence. Despite the official discourse that the "No Software" meant no infrastructure, no toil with updates and configuration, the subtext as intended for executives was very clear: "No need for you to be blocked by those pricks from engineering and their stupid, bureaucratic and gatekeeping rules".
"No software" was a call-to-arms to a certain subset of managers that were radicalized by Nicholas Carr's 2023 HBR article "IT Doesn't matter". It doesn't matter that Carr was a journalist and a writer with a masters in English that has never ever run even a small bodega, or has never managed an IT department. Anti-intellectualism and the abundance of capital brought in by the petrodollar that allowed the US government to run deficits year by year while exporting the ensuing inflationary effects to rest of world, would ensure that this message would ressonate and then even be amplified during the years of ZIRP and the Baillouts. Play fast and loose, first come, first served, a rising tide rises all boats and all that jazz. Wall Street favors bold, and the heck with the long term! This quarter will only live once!
Frankly, this is just poetic justice: Kill by slop, be killed by slop.
And I think you're confusing crud with slop.
Why is every company's Salesforce team absolute bottom of the barrel developers with super high churn, no responsibility, and little competency?
Something about the product and its positioning attracts catastrophe. That's what GP is talking about.
The lost business value and competitiveness caused by outsourcing IT overseas to unmotivated parties under Carr's premise is hard to put your finger on but I have seen the aftermath and it's pretty massive.
What business focused systems is this _not_ true for? Doubly so for such systems that encourage non-developer users to customize the database schema?
Not saying that's always a great choice, but they're far from the only ones to have made it, enterprise customers love to buy it, and Salesforce is actually one of the better ones I've seen to deal with from a technical perspective. There is _far, far_ worse out there. Most of this comes down to a business and design problem... your entire first paragraph reads as a broken process that no system is going to fix.
But if you’re having trouble with the standard rest or bulk apis, that’s 100% on you.
Salesforce API?
Sandboxes drift from production in both metadata and data. Refreshes change record ids.
API access is gated by edition. Professional edition has none by default. Salesforce sells MuleSoft as the fix to their own shit architecture.
Custom objects and fields mean that connecting successfully doesn’t establish what represents a customer, subscription or completed sale. Even standard objects can be used differently between organizations.
Metadata deployment is SOAP-based and slow. Dependencies between components break deploys and is massive pain to debug.
And notice that I said integration, not API integration. There’s a shit storm of terrible way to integrate beyond just simple API. Have fun with Bulk 1 or 2, Composite, Streaming, Platform Events, Change Data Capture, Tooling, GraphQL. An my most hated item, anything web based.
I seriously want to talk to someone that have a good time integrating against Salesforce. Maybe I will have my mind blown as to how easy it could be, but I suspect I will have the same experience I have every time I critical scrutinize and such situation: it’s just as terrible as it seems and the person claiming it’s easy is hands down producing close to nothing of value.
I find it odd you would expect sandboxes to not drift and record ids to not change on refreshes. People are working on sandboxes so they will naturally diverge from production, same for record ids.
Refreshes mean that data will be wiped out so you have to preserve the data you want first, then restore it after the refresh. Isn't that the same with any other database where you dump a copy of production on an instance?
I don't see a difference between custom objects in SF and any plain SQL database table that needs to be changed to accommodate a new requirement.
Dependencies between components break deployments - isn't that universally true?
What are other platforms where such things don't happen?
At least salesforce is consistent with their design language
If you use salesforce you know what all of that stuff means. Just click on one, it’s not rocket surgery.