Hacker News Reader

Comment by flukus | Hacker News Reader

The screwed over themselves as soon as the decided to rely on third party SaaS solutions for core infrastructure.

How many more examples do we need for this to sink in?

You're getting downvoted but if people look past your inflammatory tone there is a good argument to be made.

Back when it was more traditional to buy a software licences and self-host rather than pay a rolling subscription to an XaaS, a service going dark just meant that you were shut out of updates but not the software itself. Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.

> Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.

I think people often underestimate the value of being able to manage their own destiny. They key point here is "manage" though: you have to actually do it - it's an active process.

If, on the other hand, you simply stick your head in the sand and ignore the situation it will eventually get to the point where it becomes extremely difficult and expensive to deal with due to a variety of factors such as no clear migration path, loss of institutional knowledge, and so on. I name no names, but worked for one UK company in particular with form here.

I'm not making an argument against SaaS (we use a few of them), but it's important to understand the trade-offs you're making in terms of ceding control.

Sure, but a service like Smyte requires significant expertise to make work well and run. Pretending you can snap your fingers and implement a Sift Science or Smyte is ludicrous.

Further, some of the biggest value they will provide is being able to see transactions between vendors. So eg calculating an IP address reputation score by seeing bursts of fraudulent purchases. That is impossible to do in house only.

> Pretending you can snap your fingers and implement a Sift Science or Smyte is ludicrous.

But that's not what either I or the GP were talking about: we were talking about hosting a product that you'd purchased from another vendor on your own infrastructure.

Choosing SaaS products that have self hosted licenses or open source alternatives is part of managing your own destiny and making the right trade off. If the former exists, this discussion is moot because there will almost certainly be a commercially supported migration path (or someone didn't do their research and chose the wrong vendor). If it's the latter you're back to having to invest capital into creating an in-house system while learning how to use, scale, and troubleshoot open source software. If neither... god help you.

But this sort of systems is particularly suited to being hosted by a third party as you benefit from a network effect via the other customers.

I'm not going to argue that these sort of problems aren't ideally suited for SaaS however it is still worth remembering that systems like this have existed before SaaS took off. eg you would host the API yourself but cron a db update every n hours or days. So if the service went offline you still had a local database and API.

The obvious downside to that is you don't have real time updates (as well as the usual other benefits that SaaS brings), which is why SaaS suites this kind of business well. But it can be done and was sometimes done this way "back in the old days".

Having the db freely available makes it a lot easier for a bad actor to figure out how to avoid getting in it, though. So situations like this are suited to having a centralized owner of the data who can perform rate limiting, restrict to its paying customers, etc.

Distributing the database to customers is still workable despite this drawback - antivirus systems are a common example.

> Having the db freely available makes it a lot easier for a bad actor to figure out how to avoid getting in it

I don't agree with that. Having the database only allows bad actors to see if they've been added to the database - not the business logic of how they ended up in that database.

You could argue that knowing you've been added to the database is still an advantage, which is true. But those same bad actors could still test the system with SaaS solutions too. eg when one acquires stolen bank cards it is common to use those details to make small payments for things like food delivery to test if those card have been blocked or not.

Having a fast, cheap and detailed feedback loop is useful for testing for the bad guys, like in any programming process. For example, in the credit card case, you can check thousands of cards without needing to find a merchant who will take your thousands of small, frequently-declined transactions. Or you can check the details for cards you haven't fully compromised (maybe the guy selling you stolen bank cards only gives you a portion of the details until you've paid him).

Oh absolutely. Like many things security related, it's not always about stopping them but rather slowing them down enough so that it doesn't become cost effective.

That's preferable for the customer but it's obvious why a vendor wouldn't be keen.

> That's preferable for the customer

I wouldn't say it's preferable for the customer. It's just one of many options.

> but it's obvious why a vendor wouldn't be keen

Yes, SaaS obviously has benefits for vendors too but what's most preferable to them is making money. If every customer refused to do business with SaaS then vendors wouldn't prefer providing SaaS solutions (yes, I know I said "solutions solutions" :P). The reason SaaS is so prolific is because it offers advantages to customers as well.

>>yes, I know I said "solutions solutions" :P

When has SaaS stood for anything besides Software as a Service? Because software as a solution could describe almost any business model?

Yeah you're right. And weirdly I did know that so I haven't a clue why I wrote that comment. I can only assume it was a multitasking fail. Thanks for the correction though :)

It's not really a good argument. It's an ok (if stultifyingly predictable) argument when it's made about free services. When you enter long-term business relationships, pay for services, sign contracts, it's quite reasonable to expect to receive what you paid for. Not a lot of useful stuff would get done or made at all if we all turned into preppers.

You claim it's not a good argument yet stuff like this does actually happen. Sometimes it's genuinely because a business closes up shop, sometimes it is because it is bought out. Sometimes the business is still running but they just decide to deprecate a service or API without grandfathering existing customers / offering sufficient time to update customers code or even switch to a new provider.

Granted there are more cases of when SaaS et al works than when it does not. And I do agree with you that it's no ok to shaft paid customers in this way. But that doesn't mean that it doesn't still happen. This is why I've have to consider the self-hosted verses XaaS debate myself writing Business Continuity / Disaster Recovery / et al plans.

Of course it happens. Bad things and unexpected things happen. It is better to be prepared for them than not, etc. But saying completely predictable and trite things whenever some unusual bad thing happens is not 'a good argument'. It's barely even an argument.

The argument of self-hosting vs SaaS is still a valid one regardless of how predictable you might consider it. However the argument of arguing about arguments is definitely trite. But perhaps this is a good place to end things before it gets too meta ;)

That's not a very inflammatory language in the previous post

"You're getting downvoted but if people look past your inflammatory tone there is a good argument to be made."

I really dislike this idea that it's our responsibility to ignore inflammatory tones and trolling, instead of it being the responsibility of the poster to communicate clearer.

You're over analysing what I posted. I wasn't suggesting it's people's "responsibility" to look past trolling. I just felt there was a valid point buried in there that was worth reiterating in a non-inflammatory tone.

I assume that in the real world you grow your own food, generate your own electricity, dig up your own wells for water, and knit your own clothes? Companies have to rely on other companies for critical services, just like we do in the real world.

Food and clothing aren't services they are once off purchases that you own, these are more like buying the software and self hosting it than SaaS.

Electricity and water are heavily regulated and often government owned, not run by Silicon Valley cowboys.

I don't want to make assumptions about your metabolism, but I find that I actually need to eat food on a fairly regular basis, to the point that I would not characterize it as a "once off" purchase.

When I need more food I'll make another once off purchase. If the place that had lunch today has boarded up it's windows tomorrow I'll just eat somewhere else. If I bought that food from a supermarket it doesn't disappear from my pantry when the supermarket closes down.

My food supply is a self organising network that can route around any single point of failure. This is about as far removed from SaaS as you can get.

If the company I buy food from decides to ghost all their customers with 30 minutes notice, I can walk a block in the opposite direction and buy food from a completely different company. And I can walk a block further and buy food from yet another company. Any of these companies vanishing into thin air would only cost me a 10 minutes of my time at most.

> Electricity and water are heavily regulated and often government owned, not run by Silicon Valley cowboys.

Yet, you still get Puerto Rico (electricity grid recovery failures of a few levels), and Flint (no clean water for you, because no).

Multiple sourcing is the key. All of the above products can be bought from multiple places, this API could be "bought" from a single vendor.

Any usage kf SaaS would require a solid contract in place and a plan B (and C, and D, etc) to reach even a fraction the reliability of e.g electricity or food.

Managing risk comes into play here. Would you spend time integrating 2 services, negotiating 2 contacts, supporting 2 providers who may change features and approaches over time, splitting a fraction of traffic to make sure both still work, limiting yourself to common feature set... Or would that cost you more to implement them a few days of (potential) complete downtime.

Choosing the potential downtime is not a failure, if you considered your options.

You have to admit that the Smyte's customer list that's floated around, here, isn't all just some startups that can't afford to research their backups. And maybe they did, actually. Did anyone even hear if Zendesk was even affected by this happening?

Most of this has a latency that's orders of magnitude different. For electricity, those are microseconds (and for anything longer you do have UPS and generators and failover, right?), for the other things, it's days. "Half an hour" is just about the worst timeframe for a deadline.

Own a generator, have a stock pile of food and water, have a well I can run with the generator. Have plenty of clothing that can last many years. It isn't about being able to be self sufficient for forever, but being able to be self sufficient for long enough to weather a disaster (last time hurricane took down power for a few weeks, I only had minor inconveniences and was able to focus on helping others who weren't nearly as well prepared). Yes, it costs money and not all people can afford it, but for businesses, it should be something they do plan for.

Well, I guess every company on AWS is screwed, huh?

Eventually yes. Fortunately there is enough demand for actual machines, but if that ever goes away and all of our compute power is owned by a handful of companies then we'll be well on our way to a scary dystopian future.

At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.

> At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.

And if that happens people (I'm looking at YOU, software and devops engineers, with your damned sketchy business cases, desperate to move projects onto AWS mostly so you can bolster your CVs) will only have themselves to blame.

Fortunately I suspect there are enough of us jaded and sceptical types around to avoid it.

Truly competent Ops folks are setting up Terraform/Terragrunt modules for multiple providers and designing with cloud agnosticism in mind.

If they aren't, either they're incompetent, the CTO is, or you don't really have a company yet.

What part of terraform is cloud agnostic?

Uh, the whole thing? Do you need a link to the documentation or something?

I'm as big of a Hashicorp fanboy as you will ever find for on prem implementations -- I've used Consul, Nomad, and Vault. But that's simply not true. Here is an example of a Terraform template straight from their getting started page.

https://github.com/terraform-providers/terraform-provider-aw...

There's nothing preventing you from using other providers as well. You just have to understand how to achieve the same results when you build your modules.

So you have to rewrite your modules for each platform so you're still tying yourself to one vendor and that still doesn't alleviate risks from migration or the large amount of regression testing -- just in case Amazon goes out of business or you can might be able to find an equivalent slightly cheaper?

If you are a small company, the difference isn't worth the risk and the expense. If you are a large company you can probable negotiate with Amazon.

That doesn't even take into account all of the dependencies that your developers have taken on AWS services....

So you are saying if I take my AWS infrastructure, which I launch with my terraform config, I can simply use that same config to launch the same infra on GCP or DO? Didn't think so. It's only slightly more agnostic than AWS CloudFormation, in that I can at least use some of the same conventions when I completely rewrite my terraform files.

I think you are confusing "agnostic" with "compatible with lots of different providers", but will require a total rewrite if you want to switch providers.

That's also what's attractive about Kubernetes - it's nearly a provider-independent cloud API.

And then you get yet another level of abstraction. It's about like developers who want to wrap all of their data access in a repository pattern just in case they want to change databases.

Two things about that -- hardly anyone ever changes infrastructure just to save a little money and their code has to be so generic that they can't take advantage of all of the features of the platform.

The chances are far greater that your company will go out of business way before Amazon stops offering AWS. Theory is great that Terraform allows you to seamlessly move infrastructure between cloud providers, but the amount of risk and regression testing it takes to do so would make it a non starter for most companies.

The idea I'm floating isn't not to use AWS, it's to not be locked into it. Don't use the things that only they offer.

This isn't just about cloud provider lock-in, but something you need to do to have a good DR strategy.

Back on that Christmas Day that Linode got DDOSed, my company was 100% on their infrastructure. I only had to work a couple of hours that day because I had worked and planned in advance to launch our infrastructure on another host. I saved my company from a multi-million dollar loss that day.

So if you're using AWS and instead of using their services, you're just hosting your own equivalents on EC2 instances, what's the point? The entire reason behind using AWS is the "undifferentiated heavy lifting" - allowing them to take most of the burden off of you except where your company can add value.

If all you're doing is hosting a bunch of VPS's, you're really not saving yourself too much over just using a bunch of colocated servers. You can also get much cheaper, less reliable, less geographic diverse hosting somewhere else.

But I would have my head handed to me if I even suggested to a company to base their entire infrastructure on Linode instead of AWS, Azure, or even GCP.

There isn't one way or reason to use cloud infrastructure. Maybe for smaller companies the "undifferentiated heavy lifting" narrative makes sense and I've used it as such in the past.

Managing tens of millions of dollars in physical and cloud infrastructure at multiple providers/dcs for a publicly traded company like I am now? No, that's not the reason to use cloud infrastructure at all. Cloud in this case is more about capacity planning and avoiding managing hardware inventory than anything.

You realize that this is exactly what Netflix is doing? They are using a lot of AWS services and have completely tied their fate to AWS.

That's not entirely true now but has been in the past. They've actually tied their fates to both AWS and GCE in different parts of their infrastructure. The obvious example of this is the one or two global outages they've had due to misconfiguration in GCE this year.

I know their video encoding platform is heavily tied to AWS because the teams that built it optimized for cost (which for them makes a certain amount of sense) and they massively parallelize the encoding work. I talked to some of their engineers at re:Invent last year and it was kind of interesting. They aren't using a lot of AWS-specific stuff though, just AWS EC2 reserved instances.

I don't think that just because they're Netflix that everything they're doing is correct. This can't possibly be true.

Employees wouldn't have to keep one eye on their CV and plan for the next job if employers treated employees better and gave them long term security & oppertunities.

I think you might be underestimating AWS migrations initiated at the executive level.

I dare say the AWS sales pitch is even more effective at that level: "Increase your agility, reduce your operational overhead, scale out with the push of a button. Be like Netflix and Apple."

Most executives aren't well-equipped to probe the sharp corners that aren't in the pitch.

Not even eventually. Right now. How many companies out there absolutely depend on all the value add services that AWS provides that are hard to replicate?

Does your business depend on Amazon Transcribe (or any other AWS-only service)? Do you have engineers competent to turn around an equivalent product for you in a short release window? No? (Don't lie to yourself. It's no.) Congratulations, you are now Amazon's bitch.

Internally we have physical servers, externally we us linode deployed via ansible, we don't rely on any linode specific features (api'snor anything).

I (as the sole programmer) made that decision early on.

AWS is a none starter because of the gradual vendor lockin (also nature of our business means growth/scalability is very easily predicted).

I think people will start feeling the pain when Amazon isn’t printing money anymore and starts looking at winding down less popular/profitable services.

I think we're already there, and it isn't much of a dystopia. There are only 3 big cloud providers, and many major companies rely on them for everything.

But their business is built on providing this service. At what point would Amazon, Google or Microsoft say "we don't want that $xB in revenue, just turn it off. NOW!

Is that what you're suggesting is the future?

At that point government would step in and nationalise that part of the company :-)

More seriously I can see cloud going the way of telecoms in that there would be a provider of last resort if a telco goes bust another operator takes over to provide service.

The more likely risk is not them turning it off, but increasing prices.

Of course, it's a competitive market, so the other two providers would eat their lunch.

My coworker wrote a service that scales extremely well (he pretty much maintains, monitors and operates it by himself, across dozens of servers).

He has written multiple deployment options. Kubernetes, Docker swarm, bare VMs, etc. He has tested all of them and written installation tools that can deploy to any of them.

For exactly the reason you imply, he doesn't trust any of these options to always be available. I think this mentality has a lot to do with why his service scales so well.

And while he was doing that, how many features could he have added that would increase revenue for the company?

And while he was doing that, how many features could he have added that would increase revenue for the company?

There's more to building a long-term, sustainable business than optimizing revenue at any cost. And part of that "more" is exactly the work you to do insure against future risks.

What’s more likely to happen first - Amazon will stop offering an AWS service that you depend on at a price you are willing to pay or that the system you are building will become obsolete and when the time comes to replace it, you can choose different infrastructure? I would even say that your company going out of business is a bigger risk than depending on AWS.

The old saying that no one ever got fired for choosing IBM could be applied equally to AWS. The companies that chose IBM in the 70s and 80s can still buy new compatible systems. The companies that chose Stratus or VAX, not so much.

You're kind of operating on hindsight bias here.

Let’s take today’s equivalent - what’s the likelihood of Amazon not offering AWS or even Microsoft walking away from Azure and pulling the rug out from under developers?

Even businesses that chose Microsoft technology in the 90s when they were the clear leader have had a relatively painless upgrade and support path. They abandoned VB6 15 years ago, but you can still wrap your VB6 code as a COM object and interact with C#.

I’m saying that it is the safer bet to go with the largest provider with the largest customer base. In the 80s that would be IBM, in the 90s, MS over SUN (and the royal claustrof%%% that Java has become under Oracle) or Borland.

What is the "largest customer base" relative to? In the early 2000s, most Fortune 100s used Solaris [1]- presumably this trend started in the 90s, so I don't know why you think that a manager during that time period would choose Microsoft over Sun except for specific use cases (e.g., smaller hosting providers).

Let's look at yesterday's equivalent to AWS- Sun Grid. Sun was viewed back during the first dotcom era in much the same way that AWS is today. All that it would take to completely bork it would be an economic catastrophe that mirrors the first dotcom bubble. Obviously, Sun Grid is no more.

[1] https://news.netcraft.com/archives/2004/11/16/solaris_remain...

Solaris didn’t disappear overnight like Smyte did. When Sun was acquired by Oracle they just didn’t abandon thier customers - that would have been too value destroying.

You can still buy Solaris based systems today but you would have plenty of time to migrate. But seeing the writing on the wall you would have plenty of time to migrate.

I think you're committing something of a "fallacy of the excluded middle" here. Those aren't the only two things that can happen. AWS could, for example, continue offering the service but not to you. We've already heard stories of AWS accounts being closed for various reasons, and it's hardly a stretch to imagine a CloudFront / Daily Stormer kind of scenario where Amazon decides to punt you for taking an unpopular political position, etc.

I would even say that your company going out of business is a bigger risk than depending on AWS.

I would say that I disagree, in at least some cases.

I think there’s a balance you can strike between things. But it’s not always a single variable. Learning those things was probably valuable to him in other ways though. If he eventually grows his service and has to hire someone to manage ops he’ll be much more capable of evaluating them.

Because his service scales so well, it is bringing in a lot of revenue for the company (sorry, don't know the exact amount), at a very low cost. He is a very productive developer.

Whatever his salary is as an engineer, he is definitely underpaid.

Since I have mostly avoided doing side projects outside of work to learn something new [1], I understand spending extra time at work to scratch an itch and doing a low priority project as long as it doesn’t interfere with revenue generating - or cost reducing - initiatives.

[1] I would much rather work on a work related project using a new to me technology and if it requires longer hours to complete it, I’m find with putting in the extra time. It seems like more of a win, I get to learn a new technology, get to see it actually be used by others, and it will be helpful during either review time or worse case a resume builder.

Well, I guess every company on AWS is screwed, huh?

Screwed? Maybe, maybe not. At risk? Abso-fucking-lutely. There are plenty of stories out there of companies who had their AWS accounts shut-down for some random reason, plenty of stories of AWS outages, etc. Even if Amazon does't pull a "Smyte", there's plenty of risk associated with being on (or "solely on") AWS.

By and large, anytime you are completely dependent on someone else's platform, you are increasing risk by putting your destiny in someone else's hands.

Does this argument apply if there was a signed contract with years remaining?

It really depends on the circumstances but often a contract makes no difference. If the service provider disappears (bankruptcy, acquisition, natural disaster) and there is no one to sue then the contract is irrelevant. If it's core functionality and you go out of business then being able to sue but having no money to makes the contract irrelevant. If the provider can absorb the cost of a lawsuit because you're their only customer and it's cheaper to drop you then a contract could be irrelevant. If the provider want's to launch a competing service and breaching the contract is just a cost of doing business then the contract is irrelevant.

Contracts can be valuable but often they are over valued.

Contracts don’t disappear on acquisition, of course. The new owner takes on the responsibilities of the acquired company, as well as its assets.

Companies also are not physical items that can disappear due to natural disasters.

The only ways they can disappear is by closing down, and that does mean honoring contracts, and by going bankrupt and that means using all leftover assets to satisfy all the parties whose contracts you’re breaking. And no, you can’t just transfer all assets away and leave a shell to go bankrupt because that is fraud and the judge can nullify the transfers.

Evaluating the business viability and solvency of any important suppliers is something that you should be doing yourself even if you have a years-long contract.

Also, such risks can be insured.

Also, such risks can be mitigated in various ways - I recall having a contract where the source code of any product version delivered to us would be held in escrow at a third party, and in case the supplier went out of business or a bunch of other conditions, we'd get their code and legal rights to use/modify/maintain it ourselves or through some contractor.

A contract is between you and some entity - if you want to protect yourself against that entity "dying", then that can be done by a contract with someone else (e.g. insurer, escrow, etc) who can make you whole even if that entity is dead.

The existance of a contract does not prevent actions like this. A contract will get you compensated later, but if the provider cuts you out, nothing can prevent it. The outage will be there wether it is legal or not.

I am not 100% familiar with exactly what Smyte does but if it uses heuristics to determine if a comment is spam/harassment, then thats not really core infrastructure.

Your company isnt going to die without it (its not like Hootsuite losing access to the Twitter API or Heroku cut off from aWS) But it might be more effective with it.

Just like all those people that screwed them over by depending on 3rd party CPUs instead of creating their own?

Reply on news.ycombinator.com