Hacker News Reader

Comment by jiscariot | Hacker News Reader

I work in a role where half of our stuff is in IBM's cloud (for decisions made before my tenure). On a day in Mid-November, we started receiving alerts from our UAT and PROD environments. Logged in, all our stuff was gone. Opened a ticket and did some digging and found in our audit logs an IBM SRE had deleted all our stuff. They then told us there was no way to recover and we'd have to rebuild from scratch.

They had apparently been doing some "cleanup" and somehow our site number got on to a list. All our servers, attached volumes, subnets, load-balancers were deleted.

My boss and I spent the next 30 hours applying terraform and rebuilding anything not automated.

Would not recommend IBM Cloud. We are moving off in next 6 months.

View on HN

so I worked at IBM cloud, and can confirm this. They bought a cloud service, that by itself, and if left alone would been great. Soft(....something...) was the company... then IBM came and changed everything. I remember the SVP / President of ibm cloud showing up... talking 3 hours about how her son is so great and that we should 'follow' his example... after hour 3, man goes and interrupts and asks a question that was "please get to the point of talking about your son".

Few weeks later, massive layoffs, that triggered the warn act. So now I know how IBM works, see first hand meeting, after meeting being totally worthless. I would say stay far away from anything the touch. Just be glad you could move off their platform and not stuck using their platform with their CICD ... else you would be in living hell.

I’ve sat in an all hands like that before. Literally 50k$ an hour being wasted listening to some guy we’ll only ever see once talk about his sons 18th birthday.

Weird experience.

Same here- if they start talking about their kids, it’s over for someone, possibly lots of someones.

I think it's a power move. Nobody got up and said, "Fuck you, I have better places to be." So they win. You probably thought it was about the business or something.

I have 100% opened up a laptop and tuned out a meeting like that. Never go to a corporate meeting without a laptop.

I used to do that too. Then I got accused of not being a team player or whatever. OK, fine.

Turns out sitting on my ass and listening to something useless pays the same.

It's about power, just like mandating going back to the office.

There are lots of reasons other than power to want people on-site. This is a strawman.

Mandating it is a power move. You can give folks the option to come back to the office if they so chose, and let others work from home if they want to.

This implies the company has set itself up to do remote work well. Not every company has, not even through the pandemic. Many are still struggling to get back to the performance they were once at.

It isn’t always a power move, even if HN wants it to be; and yes, I realize that’s an unpopular opinion here.

Yeah, I surround myself with people who have their stuff together but don't feel compelled to go around flexing about it.

To be fair, it was the general manager’s bosses boss. No body was surrounded by him and by the time we were due another visit by this position, he had been promoted on.

Owner of the company spent half the meeting talking about how they take their grandkids anywhere in the world they want to go when they turn 16...I look at the audience paying rapt attention and I'm thinking 'this is a little odd'

Then he mentioned the 'merger of equals' and I thought 'this isn't good news'

Narrator: It wasn't.

Also been on one of those. One of the employees raised the fact salaries had been on a freeze for the last two years. Answer from VP doing the all-hands meeting: "My wife also wanted me to buy a new boat this year but I could not"

Started looking for a new job that same minute...

Softlayer is the name you're thinking of. They were pretty decent 10 or so years ago, but never touched them after the IBM acquisition.

Everyone wanted to be on Softlayer for a time. It was THE hosting provider. I don’t know if they where any good, but the got great press coverage.

Two years ago I had to help a customer debug some weird nginx behaviour, resulting in their traffic spiking at ten times the expected rate. The IBM/Softlayer VPN required that I used Internet Explorer, but it still failed to work. We spend three month with IBM and IBM Cloud consultants to make it NOT work.

IBM destroy everything it buys. They have POWER, their mainframes and associated software left. How that keeps them afloat is a mystery.

Softlayer was solid at providing colo and dedicated servers. But they were never really architected with the cloud in mind. The pivot to cloud came later on. I always wonder why IBM didn't buy a provider like Linode.

Likely because at the time (am unsure now) Linode did not own any actual data centers, they were all collocated.

Softlayer had the idea of API-driven bare-metal server hosting (as opposed to ticket-driven or phone-call-driven) early on, which was a big differentiator for a while. But AWS came along with an even more extreme version of API-driven hosting and they never caught up.

As an example, most network or server changes you made through the API resulted in an automated email saying your sales representative would be in touch about your order, followed a minute later by an automated email saying the change was done at $0 charge.

It has really gone to crap since then, we were softlayer customers before the acquisition and everything is beyond brittle now.

Isn't this the typical corporate acquisition process?

* Bring in Deloitte, McKenzie, or other consulting group.

* Gut the acquired company to reduce costs.

* Fold newly acquired divisions into existing mediocre division.

* Service gets run into the ground without the original people who made it great.

* Eventually the service degrades and customers leave, business line stops generating profits

* Company goes through the process again.

Gives me the unpleasant thought of how much value is destroyed and lost in these mindless corporate acquisitions. Incentives are broadly misaligned with what we should want as a society, instead investors want a payday as well as founders, and the acquiring corporation wants a fresh coat of paint and has access to the finance that can make it happen and only hurts 10 years later. In the end promising human efforts are destroyed because the rewards for doing so are too great.

OTOH the average startup is garbage held together with duct tape that naturally falls apart without the founding team who created it. By the time of acquisition it’s usually reached a point of technical debt bankruptcy. This goes hand-in-hand with an over-heated sales team pushing hockey-stick growth that will crash back down when those brand-new customers churn at the next renewal - because the product, while nice-looking, is unmaintainable garbage.

Nice payday for the founding team and the ticking time bomb is paid for out of bigcorp’s wallet. They go on to great new things and the eventual demise of their garbage pile will be blamed on bigcorp.

Not to say that large companies don’t destroy value - they absolutely do, frequently - but that the main error they make is not being able to appraise which startups are smoke and mirrors and which are legit. The rest of us are not so great at it either.

> This goes hand-in-hand with an over-heated sales team pushing hockey-stick growth

This seems like just another argument for limiting startup acquires. Perhaps if a big exit wasn't the goal, the company would focus on more long term viability.

Anyway, I don't think they big companies care as much about whether they destroy value, as long as they destroy a potential competitor.

I see competition often being less about the startup and more about which other competing big tech company could buy the startup to consolidate an existing market strategy. I’d more charitably call it “revenue protection” to preemptively acquire them.

Startup acquisitions, in the absence of astoundingly deep due-diligence should probably be placed in portfolios where a 5:1 failure rate can be tolerated. I’m not sure how such an acquisition would be compensated.

It's not just the company internally that want (needs?) those big exits, the whole VC architecture is built around shot gunning out money for the occasional huge pay off.

After being on the inside of many of these, can confirm 100%, this post could not be more accurate:-)

I have seen that with General Electric.

It's not easy to know when is the right time to jump your old trusted provider. At the beginning of the acquisition not much happens. Things degrade slowly because employees take the burden of doing the job for two people. But nobody can be subjected to the stress for too many years.

Big corporations create nothing, only abuse the good faith of employees and the cost of moving providers of small companies. I have seen that happening in tech. When small innovative companies grew they got enough economic power to not have to innovate anymore. Purchase small good companies and drain them is the new business model.

And it's a shame, there was a time that liked IBM, and others.

I stopped using Softlayer after IBM took over. I still get emails about the daily "incidents" in "IBM Cloud", as well as monthly billing notices for my $0 bill. I don't know my "IBM Id" or "Softlayer Id" to log in any more as it's been almost a decade, so I can't unsubscribe from any of it.

Sounds like a violation of the U.S. CAN SPAM Act.

Relationship messages (like incident reports and bills) aren't typically covered by the CAN SPAM Act, but that law's never stopped anyone anyway. I don't have enough fingers to count the number of daily commercial emails from otherwise respectable US businesses that don't include a mailing address or don't honor opt-out requests. Cold sales mails from tech startups are a big offender...

That seems negligent enough for your company to sue them

Good luck suing ibm..

It may take a while, but you’ll win.

At what cost?

My guesstimate of the legal fees to try the case would be in the region of $100k.

In civilised lands the loser has to reimburse the winner's legal costs, so at no cost. If you don't live in civilised lands, well you have other problems.

I'm not sure that is as civilized as you think it is. Loser pays discourages any small company or individual from suing because the cost is too great if they lose.

But the thread here was talking about a slum dunk case of contract breach.

Even AWS contracts are vague enough to not make this a slam dunk. They pretty much explicitly say if you don’t store stuff in multiple regions you are going to take outages/data loss.

Does your civilized legal system also force the inevitable loser to front your legal fees while the case is ongoing? Or is it possible the plaintiff with deeper pockets can just stretch things out until the legal system has bled you dry and you must withdraw?

pyrrhic victory is not good for business

For material breach of contract and negligence? Easy peasy.

For breach of contract and negligence, IBM were successfully sued, and then banned from all future projects, by the Queensland government in 2013. [0] Which sets you up with a nice precedent and set of documents to see their angle of attack.

> IBM will not be allowed to enter any new contracts with the State Government until it improves its governance and contracting practices.

With that ban _still live today_, it astonishes me that any corporation would trust the organisation to actually carry through with their obligations. You have to really, really, royally screw up for a government body to consider you anathema.

The old adage of "No one was ever fired for hiring IBM" is no longer true or reasonable.

[0] https://statements.qld.gov.au/statements/72961

> For breach of contract and negligence, IBM were successfully sued, and then banned from all future projects, by the Queensland government in 2013. [0] Which sets you up with a nice precedent and set of documents to see their angle of attack.

I mean, yes, but also think about how long a government can afford to have their lawyers pursue a case like that. If you don't have those kind of resources, it's a lot riskier.

They also have a very significant interest in pursuing it, in that they often have to follow procurement rules that prevent them from excluding vendors without good reason. If you think a vendor will be a problem in the future, getting a legal judgement in place may sometimes be necessary to save a lot of grief that a private company can avoid by just privately and quietly blacklisting the vendor in question.

Reminds me of the IBM lawyer scene in Halt & Catch Fire

We're a pretty small org, and not sure we have the organizational heft or resources to do so. That being said, the $1k discount they gave us for Nov based on the broken SLA (resources down) was kind of a slap in the face. We had our minimal core services up in 16hrs, but recorded about 100-120 internal engineering hours for config, testing, and other fallout +1 month.

When we opened the initial ticket, the IBM engineer kept saying "you should ask X why they deleted your stuff". Eventually after attaching the LinkedIn page for the IBM SRE in the audit logs, they realized something was screwed up on their side.

Reply on news.ycombinator.com