Hacker News Reader

Hacker News

Comment by mergy | Hacker News Reader

Full thread

mergy

Unplanned outage timing is never good but this is really not good.

https://www.salesforce.com/dreamforce/

Sept 15-17

View on HN

Say what you want about the product and leadership... they do throw a good conference.

Probably not a coincidence

Yup. Could be that everyone was rushing to get all their products and demos ready leading up to it.

AWS has a history of this before re:Invents (its annual big conference) in my experience.

They've gotten a little better in recent years.

Most places at this scale have code freezes in place well before conferences. The most likely issues are some launch couldn't handle the scale or periodic deployments have been saving them from some sort of long-standing leak bug, and pausing going into Dreamforce meant some service hasn't been restarted in a week. Historically, Salesforce sharded by customer, so that goes against both of these, unless it's in a routing layer.

I'm an ex-Salesforce, and yes, at the time I left, there was a huge change freeze surrounding Dreamforce. Unless a demo of an announced feature was coming in really at the buzzer, change velocity would have been low since a few weeks ago. I worked in a sub-cloud, so I can't even speculate as to the reason for the failure.

Something I wonder about is whether SRE responses were delayed due to having to be emergency-change-approved because Dreamforce was on. I don't recall a global outage ever occurring during a change freeze when I worked there, so /shrug.

This is what happens when more than half the company is away attending the Salesforce cult-indoctrination stuff while spending all their bandwidth making customers/partners feel good.... The stuff that matters to keep the lights on gets overlooked.

Arguably, making customers and partners feel good is the more important part of the business

They wont feel good if the product they pay for doesnt work

From what I can tell, the business model is "it doesn't work, but you can pay folks exorbitant fees to 'customize' it for you"...

Also see: Oracle

arguably, this is what sales cares about and the half-measures taken to tackle what must be the Mt. Everest of tech debt at Salesforce is what leads to large, systemically degraded customer trust in products that keep shipping bugs

I don’t think engineering and SRE of the organizing company are ever invited to those events. They’re mainly for marketing and sales (which includes solution architects).

this event is only for customers. its not a company event.

I'm an ex-Salesforce engineer and I can attest that very few line-level engineers are ever invited to attend Dreamforce. This would have been an ordinary day at the office for the SRE team.

Reply on news.ycombinator.com