The death of Rackspace’s ‘fanatical support’ | Hacker News Reader

The death of Rackspace’s ‘fanatical support’

sanantonioreport.org

164 pointsalt227141 comments

My company used Rackspace back in 2013. It was my first gig where I was "managing servers" and "deploying to production". The rest of the team was as inexperienced as I was.

We paid extra for the support services that entitled us to unlimited 24/7 human support. Whenever I was stuck on an issue with deploying, scaling, crashing etc. my boss would say "try calling support." Initially you'd get a first-line operator but I learned quickly to "escalate to a sysadmin".

The sysadmin was eager to help and would log into your VPS and do anything you'd ask, plus advise and implement on stuff they thought you'd need! Debug your running app and code? Sure. Fix broken packages or get something to compile? Of course. Custom scaling and deployment scripts in bash? Gladly. Some reps were more helpful than others and I got to know them on a first-name basis.

In hindsight it seems totally absurd to have a phone support running bash on our VPS but it was super helpful at the time. I learned a lot and they saved my bacon a few times. I can only imagine the liability issues.

I’ve had a similar pleasure in the early 2000s. It may seem absurd, but from a business perspective it makes sense: so many support questions fall into a gray area of “this is really not our business because the customer doesn’t know X, but without solving it they’ll be unhappy”. It makes sense to just say “fuck it, we’ll help them with anything, as long as they’re paying”.

It’s a similar approach as we take at my current employer. If you’re using our database products, even if you’re stuck doing basic python stuff, don’t know how docker works, or an absolute noob at AWS, we’ll help you. The time we spend is cheaper than losing business, and enterprises love it.

We do something similar on the non-tech side. We offer a saas product but most of our questions are more around how the person should be doing their job rather than how to use our software so we have a small team of experts that can answer those questions, they can also help with content creating on the marketing side too which is handy. We constantly hear from our customers that this level of service is one of the key things they really value from us.

+1 Build the customer relationship.

I probably know some of the guys who helped you out and there's a couple great ones still there. The legal team came up with the phrase "reasonable endeavor" (they didn't like best effort), and they wrote a disclaimer form they'd have customers fill out that what they were about to was beyond official support. As long as you signed that, support was usually pretty happy to "give it the old college try" as one told me. And I'll let you in on another little secret - a lot of those sysadmins were learning along with you. One of my very early projects was building out a precision time protocol service for a customer who asked for it - which ended up being a good learning experience for me.

I'm pretty often surprised by how good AWS premium support is. Every time I've asked them a question they seem to have gone to quite the lengths to gather information even when the issue isn't really theirs.

We had Rackspace and used them for our managed servers at a prior job. We had a machine with 18-24 cores and something like 768GB of RAM that handled all of our VMs, including our SAS server. It had a bunch of native SSDs and it was connected to a 20TB JBOD of SSDs and a 100TB JBOD of slower spinning disks. They recommended this setup to my predecessor as it was WAY less money than a similar SAN configuration.

So. One day the two JBOD enclosures just start failing. We lose EVERYTHING. Ok, that sucks, but Rackspace handles all of our backups for us through our SLA. Let's get that restored while they figure out the JBOD issue.

Two things come out of this:

1. Their 'fanatical support' were 'backing us up' but they never actually tested the solution and...we lost everything. There were no backups.

2. They sold us the JBODs less than 18 months prior but they no longer supported them and thus we'd have to invest in a SAN--which was a crazy investment more.

---

We had already been planning a transfer to a new local data center over the course of a year's time. We called up the new company and told them we needed to move over to their shit in 7 days. They had us up in 2.

I'll never consider Rackspace for anything.

About which year did that happen? (The data lots etc)

We were using Rackspace too, at a former workplace, for a while (no crash though).

I'm guessing late 2015. We went to the interim provider while they did a slow planned migration to 100% GCP which was completed after I left in late 2018.

Yikes. A backup solution that was sold, but not configured (at all! not to mentioned non-tested backups), was a very frequent occurrence whenever I inherited clients (in the 2000-2010s, before AWS took over the world, or simple VM snapshots at most other hosts).

(I'm not defending anyone here, I think it's how horrible many hosting companies where was back then, or still are, whenever they have sales teams that only aim to up-sell stuff and do not bother getting it properly implemented)

Yep. I spent similar years at an MSP, and there were multiple occasions where standard backups were not provisione, despite the client paying for them.

I was with a company back in 2007-08 who was using Rackspace. I'll never forget the day I was sitting at a command prompt looking at htop trying to figure out what was straining the server so much...when I got a message in the terminal from Rackspace telling me exactly what was happening and what we needed to change to fix it.

Support person just chatting with me right there in the terminal, without me even having to call for help. It was incredible and I'm really sad to see them struggling now. That turned me into a fan.

When they picked up Slicehost, I was initially really happy about it but the benefits never really seemed to materialize. Then Digital Ocean appeared as essentially "the new Slicehost" not too much later and progress continued from there.

Just never adapted as a viable cloud option. For a while there, before GCP and Azure were prominent a lot of conversations were "Amazon or Rackspace".

That seems pretty sketchy from a data access perspective.

At the time, I thought it was pretty cool.

IIRC part of the pitch and the cost was that they were literally supplementing your IT department with the same types of agreements you'd have for an outsourced IT provider.

You will be surprised how much digitalocean has access in your cloud infra. We had a support ticket, the support person could see all our pods, namespace and kubernetes objects, disk mounts etc.

How is that a surprise? They quite literally have physical access to your VPS (even if not easily).

All the things you mentioned are simple metadata available via query.

If things went really sideways on one of my servers, and I could not manage to get back in (even with their 3-4 alternative methods), I fully expect DigitalOcean to be able to log in and restore access, and charge me for the privilege.

Question is how would you know the support engineer who has access to your stuff and can log you in is not a bad actor.

Seems like it's the default that employees have privileged access like this unless it is explicitly designed out of the system. It's hard to do that even in domains much simpler than a cloud.

Managed services, managed access

Unsolicited direct inspection of customer content is what’s surprising.

I know we don’t have homeomorphic encryption, but typically policies (requiring explicit user request / Consent) would be in place even if they can’t be sufficiently enforced with current technology.

The whole thing reminds me of that Azure LinkedIn thing (where someone was contacted on LinkedIn by a Microsoft PM based on their azure activity).

It was pretty normal in that space. Good MSPs proactively monitored their customers servers for issues. I certainly would have sent a wall message if I saw the customer was on the machine.

Understandably so! There is a relationship that I find difficult to explain here.

For a managed service provider, we were there mainly for the operating system. If told about particular services, those too.

To tend to that, we had to maintain more access than what common convention today tells us should be comfortable.

It's really sold as "an extension (or replacement) for your IT department"

We would do everything we could to avoid going even remotely near the data, but it does depend on a bit of 'trust us'

There were varying degrees to it, too. Some customers would get notified if we even pulled their IP addresses, others only if we logged in, some if we escalated privileges at all.

Like the peer comment from TheHappyOddish mentions, sending a wall message to folks logged in was a sort of common courtesy; chances are we're both in here for the same reason.

One thing to remember is that norms changed over time. In the 90s and 2000s it was pretty widely accepted that a sysadmin could access customer data pretty easily and that trust was basically factored into the relationship. Over time and various scandals our expectations have changed but there's definitely a generational divide here.

Not to detract from the paradigm shift, it is very true... but this is also more of an 'upsell' now.

Providers generally don't want this access, but they'll accept it - and your money

It's a compliance requirement in certain fields, too, so someone asking for it is a reliable way to tell that they're probably less cost-discriminatory.

Fair point!

I should have realized this was relevant and included it... that's been eating a fair bit of our time lately.

Have they changed? I assume Amazon, Google and Microsoft all could enter any "instance" (VPS) they choose or simply clone the disk.

I used to be one of those phone answering sysadmins for a specific customer revenue. I think that job gave me phone anxiety. Every single different call was some new destroyed system by someone who had no idea what they were doing, hacked systems, unplanned migrations, whiteboarding-in-your-brain architecture discussions, etc. The other webhosts I worked at would never put an admin on the phone with a customer directly unless they were seriously enterprise, like a portion of the datacenter.

One time the CTO of a major company called me directly and I had to migrate a SQL database that was the backend of their public website while having him breathe down my neck. Lots of calls like that.

We gave great support but at cost. That was tiring and we did it in shifts. I hated the days I had to do that.

FWIW for people with different support experiences, support was based on revenue. The different tiers (smb/enterprise, etc) assigned you more senior admins, etc. It wasn't a massive pool of admins or anything.

I quit eventually because I hated answering the phones. Great company while I was there, though. I recommended it to everyone who could deal with the phone calls and San Antonio.

Thank you for your service.

I visited them during the peak of their success and they had a giant stainless steel slide in the middle of their facility that had been given to them by the German government.

It was steep and twisty, and I inquired about the liability of having it. With a grin, a staff member told me that several employees had broken bones using the slide and that there was an active lawsuit in progress with a former employee due to an injury.

I asked if they were going to get rid of the slide, and the response was, “No way, it’s a symbol of who we are. We move fast and break shit!”

I liked that version of Rackspace a lot. The office was incredible (still haven’t seen anything like it), and the service was great. Sad to hear where things landed.

Would a "use this slide at your own risk" sign be enough to prevent any lawsuits?

This is so a US thing. In my country an adult is just expected to be able to look at a slide, judge it as dangerous, have te freedom to do it anyway, receive medial help if needed, feel the shame, experienced a learning moment. Get laughed at, Laugh about it.

In the US there is always a lawsuit and it's someone's fault. It always feels like a not-so-nice place to be. Then again it makes for great lawyer and medical series. Lots of emotions.

I'm in the US and for years wrote software for the medical industry. I was contracted by a European company to write an app, and of course filled it with disclaimers and extra clicks asking "Are you sure?" When I demoed it, they asked what all that crap was for. I said something like, "Umm, liability, duh." Their response: "Oh, we don't sue people for honest mistakes here."

I just want to mention this in case someone reads your comment and does something inadvisable: "European" is an insufficiently general term. Some countries in Europe are more litigious than others.

Furthermore, while confirmation prompts can certainly be taken to an extreme (and they probably don't do much good if people get inured to just clicking "Yes"), they absolutely make sense for some operations.

That's wild! I'm in Canada, which I generally consider to be much less litigious than the US and A, but we still cover our bases for liability all the time.

>This is so a US thing.

It's also rather exaggerated. People get injured all the time doing all sorts of activities without lawyers getting involved. And actual findings of negligence have a fairly high bar although companies do sometimes make payments to make suits go away especially if they did something not by the book.

How much does it cost to repair a broken leg in your country? In the US it will probably be 5 figures, so getting it paid is a pretty big concern.

And if you can't work due to your injury and are one of the many Americans who does not have paid time off, the best case for you is using FMLA to at least retain your job so you can work - but your absence won't be paid. Hopefully your emergency fund is higher than the American median of $2000.

No idea I never saw any medical bill, I just pay my 100€ a month. I also pay about 50% income tax and 21% on just about anything I buy.

>In the US there is always a lawsuit and it's someone's fault

You can pretty much sue anyone for anything for sure, but there are big differences between criminal and civil law. A lot of cases come down to the "reasonable person standard" and what level of security should be inherently provided. Choosing to slide down a big crazy slide while standing up and hurting yourself probably fails the reasonable person standard, but choosing to slide down a big crazy slide while sitting and becoming injured because a piece of the slide breaks probably passes the reasonable person standard

In the UK, you might not get the 'at own risk' sign, not because people aren't going to sue you, but because posting an 'at own risk' sign doesn't do a lot to change your liability. Whatever signs you post, it's still your responsibility to do everything you reasonably can to make sure it doesn't injure or kill people.

It's not clear how much these sorts of signs help. You're still liable if you were grossly negligent in some way.

One place I've been told liability waivers are useful is with activities that can potentially be dangerous. It can make it harder to make a credible a claim that you didn't know you could have a fall while hiking harder. Though, honestly in most cases no one reads these things and, again, gross negligence on the part of a guide or other leader isn't absolved anyway.

It's usually way over exaggerated in the US. The "I'm suing for $50000000 because I was the stupid one" is rare, which is why it hits the news.

While it's true we have lots of lawyers, you need them for everything from making wills to starting a business. They're not all criminal/civil and I'd be willing to bet that a lot have never seen the inside of a courtroom, they're just there to make sure all the T's are crossed and i's dotted properly in legal contracts. My company is rather small, but has four lawyers just to interpret EU to NA trademark/copyright/contracts/etc...

When it comes to "who's actually more litigious", the US doesn't actually come close to Germany; Sweden; Israel; or Austria.

"Here is a list of the top 5 most litigious countries by capita: 1. Germany: 123.2/1,000 2. Sweden: 111.2/1,000 3. Israel: 96.8/1,000 4. Austria: 95.9/1,000 5. U.S.: 74.5/1,000. The Top 10 also includes the UK (64.4); Denmark (62.5); Hungary (52.4); Portugal (40.7); and France (40.3)."

https://eaccny.com/news/member-news/dont-let-these-10-legal-...

Sometimes a lawsuit is justified. There was a water slide in KCK that decapitated a kid in 2016. [1]

[1] https://en.wikipedia.org/wiki/Verr%C3%BCckt#Fatal_incident

> In the US there is always a lawsuit and it's someone's fault

When I broke my arm, the insurance company explicitly asked “Can we sue someone for this?”

No damn it, do your job! I pay insurance so insurance can pay for shit like this.

I think sometimes (often?) in the US it is not the injured individual pursuing the lawsuit, but rather their private insurance company trying to recoup losses.

A photo is captioned "A Rackspace employee alleges her then-19-month-old son fractured his left tibia after his foot got caught on the inside of the slide while riding it on March 1, 2013."

Even a slide that looks twisty, fast and adventerous must have something wrong with it if your foot can get caught on the way down. Properly manufactured slides don't have things you can catch your foot on going down.

Visual inspection from the outside won't reveal that. It looks like something a reasonable person will assume is a properly manufactured slide, and make their personal safety assessment based on how twisty and steep it looks.

Perhaps "we know some people catch their foot on something on the way down which you can't see just by looking from the outside, and someone broke their leg because of it, so use at your own risk but be warned of this non-obvious risk" would be more appropriate?

> that had been given to them by the German government

Really? that does sounds unlikely to me. Are you sure they didn't say "We got it from Germany" meaning it was imported from Germany rather than given to them by the German government?

[edit] infact I wonder if the slide wasn't by German artist Carsten Höller: https://en.wikipedia.org/wiki/Carsten_H%C3%B6ller#Slides

According to [1] the slide was built by Wiegand Sports [2] who design and make slides, from small children's slides to 'spectacular event slides' - including indoor slides for corporate offices, malls, museums etc.

So this is a slide made by an established manufacturer.

[1] https://www.expressnews.com/business/local/article/Resolutio... [2] https://www.wiegandslide.com/en/products/slides.html

Looks like Carsten Höller slides are Wiegand slides: https://www.wiegandslide.com/en/products/slides/event-slides...

I guess the problem is that people see a slide and think playground/children's toy and that they are inherently safe, but the size of them can lead to trouble - there was one in the Tate Modern Gallery in London a few years ago that had to have safety personnel at top and bottom.

Oh man, I can feel the summer time 3rd degree burns just by looking at those photos.

My town used to have a huge slide built into a hillside. It was "controversial" in that it was considered dangerous. After general playground renovations, the plastic substitute was subject to repeat vandalism and it's been removed entirely.

I interned there around the same time. The office was an old mall. Apparently, there had been a shooting in/near the mall, which led to people not going and the mall going under. Rackspace bought the mall and converted more and more to office space as they needed it. If I recall correctly the slide was from the old food court balcony area down into the main floor. I agree, it was a very cool office.

I too visited them during the peak of their success, on one of their big job interview days. While a large group of us were waiting (for hours) we were invited to watch a series of videos of their employees "having fun" doing various contests and adventures. They were especially proud of the video of their new "Chief Brand Officer", who decided to have the company logo tattooed on their arm - like, yeah, a real & permanent tattoo.

I left.

> Remember the old Rackspace and its famous commitment to “Fanatical Support?” Highly trained people handled the company’s incoming telephone and online inquiries, priding themselves on quick responses and fast solutions.

That spends money to create a good reputation, which is an asset that some MBA type will later be tempted to burn to create $$$ for the short-term gain of shareholders. My understanding is private equity has a very well-developed process to do that.

Aaaand Rackspace was acquired by private equity in 2016:

https://techcrunch.com/2016/08/26/rackspace-to-go-private-af...

> Don’t expect Rackspace to ever fully recover from Friday’s meltdown. The 1.2 million-square-foot headquarters in Windcrest is now for sale. The remnants of the company will soon decamp to less than 10% of that space, in a smaller office building near Stone Oak. A company stock that soared to about $80 a share in early 2013 fell from about $5 a share Dec. 2, when the meltdown hit the wires, to $3.23 as of Friday’s market close.

If anything the title is an understatement - they might just as well have used "The Death of Rackspace" instead...

Rackspace have many divisions and revenue streams, Exchange being a small part of their overall business. They have many thousands of customers still using their other products that are completely unaware of whats going on in Exchange right now. I mean, it didnt even make HN front page!

Rackspace will not die from this, they will plod along as a reseller for other clouds for many years to come.

Rackspace will not die from this, they will plod along as a reseller for other clouds for many years to come.

Equinix will pick them up for pennies on the dollar.

> I mean, it didnt even make HN front page!

There was an article on the frontpage about the outage when it was first happening.

Edit: This is the one I saw on the first page https://news.ycombinator.com/item?id=33839941

Rackspace does have other revenue streams, and this will be a wakeup call for their remaining customers because there's no reason to expect that a gross management failure like this will be isolated to one unit.

This did make the HN front page – that's where I learned of it – but it wasn't really a surprise to most of us. This is what private equity does and it's what was predicted back in 2016. I didn't see anyone with high expectations for the deal, but I did hear about several people migrating away as soon as the Apollo news broke.

Er, well this is the second time it’s hit the front page so

>Highly trained people handled the company’s incoming telephone and online inquiries

Not sure I can agree with this. Almost a decade ago we had some shared hosting accounts at Rackspace for some very legacy clients (ones that even accounting had forgotten about and not billed them for years). We had an issue with accessing one of the databases to export for handover, and so I got in touch with their online support. They gave me full admin access to their database server which had several hundred other clients on there. I could also see the historical metrics for the server which were interesting. I told them immediately but it took them almost a day to revoke my access.

rackspace never had a "shared hosting" product outside of a very short lived acquisition of mosso from like 2007-2008....

It might have been called something else like cloud hosting, but it was definitely equivalent to a shared hosting model. We didn't manage the web servers, no root access etc and the database server was shared between 300+ of their customers.

cloud sites. The product was bought out by liquid web about 10 years ago, if I remember correctly.

I used Rackspace back in 2001-2002. Made the switch after hosting my JSP web app on servlets.net and experiencing an extended outage with no support. The difference in support was nothing short of miraculous.

I’m inclined to think Apollo failed to understand the investments necessary to make such a business work, and figured they could squeeze costs out of the business without losing value. Funny how these finance types always seem to find themselves suffering from “Black Swan” events every few years.

Actually I think the financial types correctly identified that Rackspace's days of growth were over, and what remained is to allow it to wind down over a couple of decades, getting what profit they could along the way. There's no point investing in something who's days are numbered, so they didn't. They cut costs and jacked up the bottom line. They probably didn't want or plan for anything major like losing emails, but they probably did anticipate a degradation in quality of service and gradual customer attrition.

I'm not sure this is good or bad necessarily - it's just part of a business cycle.

It should be a wake-up call for folks that continuing to do business with a company that is no longer interested in growing and increasing top-line is risky. Companies trying to grow top-line will be very interested in over-delivering and keeping customers happy even if they lose on short-term bottom-line. The opposite is also true.

I worked at Rackspace from 2010 to 2014, and left on very good terms. I know many people who stayed well after I did. Private Equity killed Rackspace. They had a great product, a great team, a healthy war chest, and a good business with good margins.

They were purchased by Private Equity which took a $5B company and turned it into a $1B company, after laying off all the US-based staff, sending all the support jobs to India, destroying the brand, ruining their most innovative product offers, and loading the company up with unnecessary debt.

I saw the writing on the wall, which is why I left just a few scant months prior to the buy-out. I still wish it hadn't happened. Rackspace (of that era) was the best place I ever worked in my entire career, and I still remain close friends with many of my former coworkers. I grew immensely as a technologist and a person through my time at Rackspace, and I learned to love customers and love supporting customers in a way I never thought possible having done my time earlier in my life in retail. Rackspace was something really unique and it's sad to see what's happened to it due to greed and stupidity from the shareholders and the PE firm that bought it.

One of the coolest things that happened to me was being on third-shift near the end of my shift the same morning that Lanham appeared on a talk show as he did sometimes. As he always would, he'd get them to live-dial Rackspace Support from the number on the website. At the time, Rackspace did not use phone trees or first-line, everyone was in the queue to take calls off the main number and transfer if needed. I picked up the phone on the 2nd ring (he always said, and it was true, you'd get a /real person/ with /real knowledge/ in less than 3 rings) and suddenly was talking to the CEO on air. Was super cool and a good way to the end the shift.

To go from that level of support, to get an extremely senior sys-admin (I was one of several people at Rackspace that was a "double L3" during my time there, reaching seniority in both Windows & Linux systems administration), within 3 rings, who can just solve your problem, even at 2AM, to what they have now getting stuck in phone-tree hell before being transferred to a time-waster with a script in India who'll eventually hang up on you is truly horribly sad. Private Equity has ruined SO MUCH value in the American economy, and it isn't a new phenomenon.

Former customer here, when I migrated off of Rackspace in July, they couldn't manage much of the infrastructure, firewalls, switches and netapps were all managed by scripts, automations, and clueless engineers.

I don't blamw the engineers, I blame management

In a functioning network, firewalls, switches, and NetApps are managed by scripts, automations and clueful engineers.

Once the clueful are replaced by the clueless, or are burnt out, that's the end.

That's one of the first lessons old schoolers learnt about web hosting when the internet boom began and busted - The oldest web hosts are the most reliable, but always keep an eye for change of ownership. Rackspace, Pair, Slicehost, Linode etc. were some of the really popular and good ones in the small and medium segment (Rackspace, ofcourse targeted enterprises) during this era but now ALL of them have been bought by someone else for their brand value. And yet, the new buyers didn't understand or seem concerned about the values behind the brand that made it so respected, and so all these brands have suffered under new management (I guess it is too early to speak about Linode yet, as Akamai just bought them).

I went with Slicehost to Rax (as a customer) in that acquisition, overall it was as good as could be expected. Since Akamai isn't a PE firm (I think? :) there is hope for Linode

Did anyone else's spidey sense tingle when they read this in the article: "All the founders are now engaged in building other companies while often investing in other promising startups. All except Yoo were early supporters of the Rivard Report."

Note the author of the article: Robert Rivard. The Rivard Report is now the San Antonio Report. https://sanantonioreport.org/one-year-later-more-than-a-name...

To be clear: there's no questioning recent events, those are well-documented. The story of the company, its founding, and fate since Apollo bought it in 2016, though. That I'd want to hear another source.

> Customers learned that the hard way in the early morning hours last week Friday when their email accounts went dark,

The company I work for was directly affected by this, I wondered how come I didn't see it mention here on HN (or maybe I missed it). Either way, we moved to MS Exchange and for the time being it's all for the better.

It was posted on HN about 20 times over the last week, but never made it to the front page. See here for some threads about it:

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

It definitely made the front page or I wouldn't have known about it

The writing was on the wall when a private equity company bought Rackspace. My company was already planning to switch to AWS but that was extra incentive.

Yet another example of why you should run for the hills (as a customer or employee) if a private equity firm buys a business.

No joke, starting in the Linux administration field I felt cursed. My first three employers were acquired by investment groups and quickly run into the ground

First Hostgator, then Rackspace, the third... not sure I want to say

In one of my previous companies we used Rackspace to manage our AWS account. In the early years support was quite brilliant, however, in later years we only kept paying them to keep external investors happy. We could claim that a skilled company was keeping our AWS accounts secure. However, in reality they were useless, and just offered us AWS Well Architected reviews that we could have done in house for a fraction of the costs. In the end it was cheaper to build up an SRE team and pay for AWS Premium Support.

Apollo is the private equity giant that bought the company in 2016 in a $4.3 billion deal.

The latest market cap is less than $800 million on sale of more than $3 billion / year. Still losing money.

Officially, Rackspace is ~30% sold short. Someone is already making tax-free billions shorting it to death.

Why would it be tax free? Until the position is closed out it'd be an unrealized gain but there's nothing special about shorting. Gains are not taxed until they're actually realized by closing out the position.

By leaving the position open forever and taking a loan against the security.

I'm not sure I understand - to sell short you already need to borrow securities, can you actually take loans against a short position that becomes worth something? That's interesting.

You can Fail-to-deliver and never locate the stock that you are supposed to borrow. Or you can short ETF with this specific company in basket while going long on anything else in this ETF.

Everything you own, even your own debt, can be used as a collateral by creating and selling swaps.

I work in markets. No, you cannot just fail to deliver. I'm also not sure how the ETF thing would work. If it's 1% of the ETF you're going to hedge out the position using a giant notional. It doesn't work.

Official SEC document regarding regulation SHO describes both illegal and legal cases when you can "just" fail to deliver [0]. Market makers which also happen to have hedge-fund branches are having the most flexibility in this.

[0] https://www.sec.gov/investor/pubs/regsho.htm

So it may not be illegal in all cases (although it is in many cases), but how does failing to deliver close out a short position?

    By leaving the position open forever and taking a loan against the security.

Where would they get the money to pay installments on the loan? What you're claiming is really just shuffling around which dollar gets taxed.

Maybe I misremembered something and cannot find a source now, but I think there was some way to avoid paying tax on short sales when company goes bankrupt and gets delisted.

EDIT: See sibling comment.

EDIT 2: Am I reading this right? almost 1,800,000 shares failed to deliver just in one day of Sep 22nd? [0]

[0] https://fintel.io/ss/us/rxt

If the company actually goes bankrupt and their stock is delisted from exchanges, covering your position becomes much harder since liquidity is greatly reduced.

I think the ideal scenario for a short seller is if the stock loses 99%, stays listed, they cover, then it gets delisted.

I might have that wrong though.

You have that wrong; bankruptcy is the ideal scenario for somebody with a short position. The brokerage writes it off. The shares are worthless, so why go after somebody for owing you $0?

I did some research and found that I was right.

No-one's going to go after you. The problem is that you have to keep paying the fee to borrow shares, and you can never repay that loan because there are no shares available anywhere.

See this article for an example: https://www.bloomberg.com/opinion/articles/2018-04-11/-go-to...

> He shorted some stocks that he thought were frauds, and the SEC agreed that they were frauds and halted them, and then ... things got worse for him. The shares were worthless, but they didn't trade at zero or $0.01 or whatever: They didn't trade at all, so he couldn't buy them back to deliver to his stock lenders.

Usually short sellers do manage to cover at some point before the stock completely stops trading though.

Not that I'm saying Apollo's strategy was to pump, dump, short and then crash the company, but if you could generate the trading volume to support the play, you could make money at every point in that trade.

What was the plan? AFAICT they were in a tight spot strategically at the time, and still are. You have AWS as the behemoth, GCP as the pretender, and a whole bunch of other vendors like DO and Hetzner, making things competitive to put it mildly.

Interesting how the whole Web Hosting industry changed. When RackSpace used to be well known. Now it is basically AWS / GCP / Azure or Linode / DO / Vultr or Hetzner / OVH.

It's not changed that much, it's just thata new infrastructure as a service industry has grown up.

If you went to any of the infrastructure companies you have listed and said that you had a few Wordpress sites that needed hosting but needed someone to help you get them migrate them over and make sure everything was set up properly, then they would not be able to help. With most of them you wouldn't even get through on the phone. There's still plenty of hosting companies about who will do that as part of the service with no problem.

Rackspace had such a strong reputation during the 2000s, particularly for dedicated servers.

I wonder if anyone in their exec team ever brought up the fact that they got their ass kicked by DigitalOcean, and similar platforms when it comes to affordable and simple hosting.

They bought and then shuttered Slicehost in an attempt to occupy that market segment.

DataPipe (another company where tech support was the #1 priority) was also acquired by Rackspace. However I don't know if it was acquired by the original owners or the current org.

That deal was announced in September 2017 and closed in November, so Rackspace was already in the clutches of Apollo by then.

Onica too, which I don't know a whole lot about but my impression is that they were the top tier of cloud consultants.

slicehost wasn't shuttered so much as rebranded. rackspace didn't have a viable vm service prior to slicehost.

I feel AWS ate Rackspace's lunch.

Rackspace were never part of the affordable hosting segment.

I think DigitalOcean ate the shared PHP hosters' lunch.

Rackspace was never the most affordable; their differentiator was service. Digital Ocean had very little to do with their decline.

The biggest factor was the shift in corporate mindset from “we need to operate our own servers” to “we need to orchestrate services running on someone else’s servers.” Probably AWS is most to blame for that.

Rackspace tried to make the jump across that chasm several times but they were trapped on the wrong side of the Innovators Dilemma. All their existing customers expected (expensive) full phone support. And new customers who wanted automated cloud systems with minimal support did not look to Rackspace for that.

Eventually they leaned entirely toward service and started reselling and supporting all the major cloud platforms. I would bet that is the majority of their revenue today. Any hosting they do should be considered a legacy business.

> Digital Ocean had very little to do with their decline.

From my personal experience, Digital Ocean did pull away a lot of indie devs and small businesses away from Rackspace. Those folks did not need most of the features of AWS. They just needed simple cloud servers and maybe some load balancers.

My own small business used Rackspace in 2010 to 2013 ($5k per month op cost). When DO came to our radar, it was dead simple to switch to and cut the cost almost by half.

Another thing I remember about DO was their abundance of community articles (HowTo's and server setup tips, etc...) that were well-maintained and always kept up-to-date with latest linux distros.

Digital Ocean was just too late and too small to carry much blame for Rackspace’s decline in hosting. Even as recently as 2021, DO revenue was $120 million vs almost $3 billion at Rackspace.

Play around with various hosting company names in Google Trends if you want to see how interest changed over time.

I’m not saying DO is bad, far from it. But it’s more accurate to say that they benefited from trends that were well underway before they had significant market presence.

I had been a Rackspace customer a few times at various roles. I never experienced this "fanatical support" on anything that mattered.

After several outages where we received pathetic root-cause reports initially, and had to push them hard to provide actual data, and clear commitments to shore up their systems, I would describe them as "fanatical liars".

It is funny, just the other day my wife and I were discussing cloud stuff and I said "I wonder whatever happened to Rackspace". Sounds like they've continued their path of sub-par strategy since I last dealt with them a decade ago.

In my experience, the story with companies like this, with images like this, is that's it's almost always lying marketing/incorrect customer belief long, long, long before it's widely recognized. It's not a question of them degrading; it's a question of enough people realizing it, or them screwing up enough to get wider coverage. I was not happy with Rackspace's service something like 9-10 years ago, and the story I see here today is not "Rackspace bad," but "Enough people finally realize Rackspace bad." I've had similar experiences with the usual suspects: ProtonMail, Namecheap, Gandi, a few more I can't remember right now. Whatever the popular perception of quality they enjoy, you can be assured their actual quality is far, far less.

And you can see they are always trying to control this message.

Rackspace was the best company I ever worked for as a software engineer. Started off as a contractor -- even then they treated me with respect and I felt no different than an employee. But when I became an employee -- WOW! The best onboarding I've ever had. It was also the best team I ever worked with. Only team I ever worked with that actually felt like a team and not a bunch of frenemies trying to step on eachother's necks to climb up the ladder. I still think Rackspace is a Good company with Good people. It's a shame to see the mismanagement of the work of such a great group of people.

Used Rackspace heavily around 2012ish to 2015ish but eventually migrated most of out stuff to AWS. As of 2020ish or so we had a handful of VMs we'd not migrated yet and were migrating them one at a time.

Whenever we completed the migration, and wanted to delete the VM we were forced to make a phone call to a real person, who would try to talk us out of deleting the VM! We were prevented from doing it from the console, and chat support would not do it either, we were forced to call in. Complete bullshit and would never consider Rackspace for anything ever again after that experience.

I ran in to this Sunday removing my last VM from their cloud. I've used CLI/Ansible/etc to manage these forever but this time I was in the web console making sure very last byte was gone. Notably the APIs don't have any of these restrictions which is why I hadn't noticed earlier. "openstack --os-cloud rax server delete webfarm03" did the trick...

They were definitely not cheap, but looking back, $249 for a low-end server plus the expertise you got was well worth it. They had some people that really knew their stuff back then.

But by 2010, 2011, it started to fade. They physically lost one of our servers and it took 2 hours to power cycle our stack (the firewall they provided is actually what died).

We wanted to double our RAM from 16GB to 32GB. It would have doubled our monthly cost.

Then they purchased Slicehost and that integration went terribly.

Their hosted Exchange support was bad from the start. Useless agents using chat, nobody knew how to read logs, it was always down...

Right before they went public, they started to outsource a lot of their support. It looked like lipstick on a pig.

There was never any support outsourced while I was there from 2010-2014. They originally went public in 2008. That is just not true. Everyone in support sat physically in an office other than a few very senior folks that worked from home. Most were in San Antonio, some were in Austin, some in London, a few folks for LATAM were in Mexico City and Costa Rica, and folks for the BENELUX team were in Amsterdam. We also had a small office attached to the datacenter in Hong Kong and Sydney, Australia for support in APJC. Most support was US-based, regardless of customer location, and we did a shifted model rather than follow the sun.

There was never any point until after they were acquired by Private Equity in 2015 that Rackspace outsourced any part of its support pipeline, including specialty support (databases, sharepoint, exchange, etc).

I stand corrected. I guess when it went private, it got outsourced.

This is the first I heard of any of this. Were there no stories on HN about the whole ransom debacle? Bizarre.

There were multiple last week:

https://news.ycombinator.com/item?id=33851863

https://news.ycombinator.com/item?id=33839941

https://news.ycombinator.com/item?id=33884714

I think this just goes to show how few people use Rackspace hosted exchange. Most people migrated to O365 years ago. Rackspaces exchange servers were still running Exchange 2013 until the end, and their engineers were deperately trying to push people to migrate to Microsoft long before this happened. I think this is a blessing in disguise for them really.

There were around 20 submissions to HN over the last 2 weeks, but never made the front page. Shows how few of us are still using Rackspace Hosted Exchange!

I think hosted exchange is also more "enterprisey", and legacy. I'm sure the install base is still immense, but I wouldn't expect tons of discussions about pretty cookie cutter enterprise MS IT infra here.

> were still running Exchange 2013 until the end

I don't remember well but MS killed some things in the later versions on which some hosted management solutions relied.

We still host some dedicated servers at Rackspace and while the technical side has continued to be solid their phone support is awful now. I used to call in and have a specialist on the phone very quickly, now I get handed off to an obviously off-shore support person who I think is running a script before being escalated to a proper specialist... when available. We've been with them for almost 20 years but I think in the next couple that's going to change.

Companies and the products / services they produce have a lifecycle. Some are very long, tech companies tend to be shorter. Rackspace, I suspect, is past the harvesting stage and now in the maintenance / slow burn stage.

How long this goes on for is anyone's guess. They may turn into the Oracle of VPS providers and somehow keep going on for decades even though what they sell isn't really of great value anymore.

I think it's untrue to say the whole company is dead especially with the head office reduction as an example. Moving support to offshore is common in tech now so I guess they are just doing the same as everyone else.

Also note that companies that have a large data breach are often the most secure companies you can buy from after they have taken corrective measures. They know the pain of not being secure and are willing to spend money securing. That brand new tech start-up probably doesn't really care about security at the moment as they think they have higher priorities.

I was only every a very very small rackspace customer, but my recollection was that what Rackspace were actually fanatical about was telling you how fanatical they were. It always rubbed me the wrong way, regardless of the quality of support I received, so now, 10+ years on all I actually remember about them is that they were vaguely irritating. We ended up moving to Rimuhosting, and while I think the quality of support was probably equivalent, to my limited sample size, Rimuhosting was much less in your face while providing it, so the actual experience was far more pleasant.

Used to use rackspace all the time, it was my 'go to' platform of choice a decade ago.

AWS poached all the actual talent, and rackspace is left with the 2nd and 3rd stringers at best.

Now I wouldn't trust them to host a static page for me.

I remember them and that infamous podcaster who joined them from Microsoft and then managed to help bury Google Glass by posting a photo wearing it in the shower.

Ahh the era when Techcrunch reigned.

My first big project at my current job was to move all customers from hosted exchange 2013 to office 365. It was a difficult time for customers and the company, but the rackspace drama confirmed that was the right thing to do.

Funnily we have customer project to move them from several IMAP & HEX providers to O365 - the customer went radio silent for ages, but instantly called us to finally get it moving.

The company died the second they sold to a PE firm.

They've been piss poor for at least a decade. They were losing even in the managed dedicated server arena by 2010 which was their main product afaik... they were late to the cloud, their managed email was disaster (both exchange and non-exchange version), the support was anything but fanatical...

-- is it true the digital ocean support also bad now?? - who has good support? --

Serious question. What support would digital ocean , an unmanaged vps provider , really need to offer besides keeping their network, control panel and kvm nodes up? Assuming their infra is running normally which is a bare minimum there should be no reason to contact support.

-- they have large product set these days --

I've used DigitalOcean, Linode, Vultr.

All have had OK ticket-based support.

There are horror stories out there for each one.

I've always found Digital Ocean to have such excellent documentation and usability that I've never had to contact support. Except a couple times to increase my number of reserved IPs which they did in less than 30 minutes or so

I chose Rackspace at two different start-ups in the 201X's. I think one of them is still using them, but planning a migration to Azure.

Yeah, their 'fanatical' service was really good once upon a time.

I don't suppose that level of support was ever scalable.

it's too bad a once great company like rackspace had such a sad, dramatic fall from grace.

i was a customer of webmail.us, which was bought by rackspace in 2007 or so. i waited for things to go downhill, but the integration was seamless, and continued to hum along, with only a couple of issues, for several more years.

before rackspace began their nose dive, i made the jump to fastmail.

When your core differentiator is fanatical support maybe you shouldn’t turn it over to “Fanatical Offshoring”

rackspace died a long time ago. Im surprised anyone even uses them now.

Reply on news.ycombinator.com