Twitter ‘smytes’ customers
techcrunch.com
techcrunch.com
I'm pretty accustomed to executives making dreadful decisions without the approval/acceptance of other stakeholders within said person's firm. I'd rather know who made this error and not interact with that specific person's department rather than stop doing business (e.g. large ad-buys) with Twitter as a whole.
I don't want this to be a witch-hunt so much as I want the person to just come forward and own the decision, because unless they have an exceptionally good reason for it, it comes off as absurdly high-risk to both Twitter-the-business as well as to all the clients who've likely written serious penalties into their contracts for events like this, which again brings that business risk back full-circle to... Twitter.
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
It's talking about how we all talk to each other. I think it's main purpose is not to lessen criticisms for corporations so much as to keep the discussion constructive.
The parent comment actually seemed spot on to me. The rule is to assume good faith of your interlocutors. The parent failed to assume good faith of a billion dollar corporation. I think it's reasonable to push back against being asked to tone down the latter, while honoring the rule for the former.
That is a metaphor made with an industrial grade cement mixer.
It could also hamper their ability to make further acquisitions. In future, a startup being linked with a possible Twitter acquisition is going to cause its customers to panic and start moving elsewhere. That'll make startups more reluctant to engage, and may turn some off entirely, if they're not comfortable with the prospect of totally screwing over the customers who've supported them during their early stages.
I think potential for reputation damage, on an activity they don't do very often, and where they will justify it, will be weighed up against the financial benefit now, and disregarded.
I think it's possible to overstate the relevance and importance of people typing furiously on internet forums about things like this. Twitter is what it is - most people aren't using APIs, third party clients etc and will never notice this.
This isn't about people typing furiously on internet forums. This is about business risk, as weighed up by people whose job it is to care. Acquisitions take time and negotiation. Just because a startup is talking to Twitter, doesn't mean the acquisition will definitely happen. But if customers get wind of it, then they may start looking to move elsewhere, anyway, instead of waking up to find services they rely on have been shutdown without warning. If you're the customer of a startup Twitter is negotiating to acquire, then you're negligent not to reduce your reliance on them. And if you're the CEO of that startup, then you have to consider the risk of customer flight (and your own comfort with the thought of screwing them over) vs. the likelihood of the acquisition happening.
Why would anyone else?
Let's not forget that those customers are also likely candidates to advertise on Twitter
But you're actually right -- if the acquisition hasn't closed yet, they're independent and independently responsible.
Twitter could have stepped in and halted things, but that would have required Smyte to have acknowledged breaching the contract and forfeiting $$.
This is all guess of what seems to me more likely than any Twitter exec pulling any triggers like this. Of course, it's still a screwup by Twitter to have not been tuned in and aware that fingers would point to them.
But that's a very different kind of mistake than "I have an idea, let's hit the power button at 6:30. Team: you have 30 minutes to let all our customers know."
My hunch as an outsider is that Smyte wasn't GDPR compliant. Their leadership knew it, they knew they couldn't easily become so (for instance, they may have been using Kafka in a way where compaction wouldn't help, and didn't want to build an encryption-based monstrosity [0]), realized that they wouldn't increase in value as a business due to that risk, took an acquihire for cheap in order to give their employees a decent landing and give a return to their investors, and couldn't tell anyone about these plans in advance because it might jeopardize the transaction.
EDIT: They were indeed using Kafka per [1], and due to the strict latency requirements on their business, that may have ruled out the type of encryption scheme in [0].
[0] https://danlebrero.com/2018/04/11/kafka-gdpr-event-sourcing/
GDPR does seem like the most likely culprit here, if they've been planning an aquihire for a while it wouldn't have been worth implementing GDPR and Twitter likely made it a condition that the service is shut down before the acquisition completes
I'm quite surprised none of the cloud vendors were interested/could offer more than Twitter for this team though, seems like a logical service to add.
But that doesn't come with any actual guarantees does it?
You can have a contract where you specify indemnification of damage in case the business ceases operations, goes bankrupt, get acquired, etc...
While you can have a contract that indemnifies that isn’t going to help your going concerns when the other party just pulls the plug.
In time, the situation can change. The product can get sold. Cash can be spent. The guarantor can no longer make good on its guarantees.
You're back to square one where the guarantees are as worthless as the original service.
You need 3rd party backing (insurance) in such situations, but that costs money. This money is a cost which makes competing against unbacked entities tougher.
In most cases, you can not have 100% foolproof guarantees of anything. The closest I can think of is governments standing behind their banking institutions. Even there though, governments have defaulted on their guarantees.
The world is not a stable, perfect, and cut-and-dry place as many would like to believe. It is dynamic, and ultimately backed by trust.
The problem has always been present especially when larger slower moving companies buy from smaller, riskier, companies.
In the days of software, code escrow was possible to mitigate some of this kind of risk. That's still got it's costs but can be an effective hedge against a supplier going bust.
I've been on both the customer and service end of such agreements.
http://www.ironmountain.com/information-management/software-...
Yet... that you know of.
Nothing specific against Sift Science whatsoever in my comment, but many, many times this sentiment has been conveyed by companies and it rings hollow IMO. There are a lot of different ways a company can change or disappear at some unforeseen point in the future, and claims that "you can trust us to be here forever" do not carry much weight for the large group of experienced users, devs, management, etc who have been burned multiple times.
Clearly, you need to always be prepared for your partner to go away, but you still have to work with other companies.
Sure, that's the intention but when someone comes around waving a checkbook, they can in turn buy you and shut you down, sell your customers' info, etc. Sure, its not YOU allowing that to happen but by ceding control you essentially allowed that to happen.
Half an hour of warning, to screw over all of these?
They broke npm's user sign-ups, and publishing of packages, with half an hour of warning.
I can't imagine the havoc over at Zendesk either.
That is not 'winding down'. That's ghosting.
Per the article, they did at least have long-term contracts locked in with Smyte. That's not the same as control over the feature, obviously, but plenty of companies do business on the strength of contracts instead of vertically integrating their entire product.
Obviously we don't have many details yet, but I think there's a real difference between simply using a third party API that works for the moment, and buying a service from a third party. If Twitter/Smyte simply broke contract with all their existing customers, that's very much their responsibility.
Unless you're advocating against any kind of *aaS, this is kind of a silly argument.
A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.
Obviously when you make your product/company depend upon these things, you're taking a risk that they could go away, but you protect yourself against that by having robust contracts.
Outages are different, but for a significant outage you expect to see some kind of RFO and explanation of how they're going to mitigate it.
Deliberately withdrawing service for all customers with 30 minutes notice? That's entirely the fault of whomever is in charge at Smyte.
Even 30 days, while it'd probably ruin some existing plans, would at least allow people to have a chance to migrate without things just breaking.
For the cases I was involved in, there were options of buying the service but hosting it internally with some update pipeline. So even if the updates were suddenly turned off, you have a far longer time to swap to a different system. Generally deploying these aren't to much time, but what is really the issue to me isn't that the options that were chosen were the ones which were chosen, but that the process of choosing them did not evaluate the risks of creating such a strong external dependency at all. Its one thing if it is a risk taken with full knowledge of the potential costs and benefits, it is another when people do it because they think there is no possible downside.
No. Delegation is the cornerstone of civilization. Trying to build everything yourself is a terrible idea.
* No website or app should ever call out to any third party API
* Every single piece of functionality a website or app has should be developed in-house
* The functionality third party APIs provide would often take "a few days" to reproduce in-house
Definitely. Your website and app should only call your APIs. Your API can then call the third party API from the server. It's a lot easier to change something on the server than having to update the app.
It's not really addressing the whole issue of depending on third party APIs if your API is calling on them, but you are technically correct, which I've heard is the best kind of correct.
Twitter owns Smyte. When you buy a company, you acquire all of its responsibilities. There is no longer a Smyte to point the finger at. It's all Twitter.
I’m actually surprised that it was done in this manner, as many startups have a “business continuity” section in whatever contracts are drawn up with customers, detailing the specific steps and timeframe of retiring a particular service. I would have thought this is standard practice for a SaaS company.
Twitter owns this now and the way to ensure it's their reputation that is affected over the long term is by correctly calling this a "Twitter" failure.
Tbh that has always frustrated me though. I should be able to pipe those requests and just add an X-Forwarded-For header.
Is there any indirection that would avoid having to walk the traffic over my own network?
Obviously you then pay the bandwidth, but it’s likely negligible compared to your app traffic. As a bonus you’ll probably circumvent adblockers.
If you want to avoid traffic forwarding, but keep flexibility over the endpoint, you can override the init of the sdk to first query your server for which endpoint to use. That way if the third party service goes down, you just need to change the config on the server.
Would Smyte's rating have cut it for people?
The problem is, if all suppliers use the same contract clauses you may not have a choice.
and how would their credit rating be affected if they pay all their bills but just reduce the number of bills gradually over time?
2. If it's that critical, you have redundancy (an alternative provider, or a naive implementation that maybe causes more false positives but that will cover you during an outage)
And no, I wouldn't have an alternative provider. It would not make sense to, if I'm paying a company to provide a service. Why should I pay another to do nothing?
I've been a senior developer in an organization and known that a solution isn't perfectly robust. Clearly I want to fix it because I (like most other developers) like building robust solutions, but I have six weeks of tasks to finish in two weeks, and that isn't one of them. What do I do?
You might say "do it anyways" and, if so, that's a junior developer attitude. I'd strongly urge you not to do that as shipping big projects is really more about choreography than engineering.
Or, you might say "fight for permission" and that's not a bad idea, as long as you accept that you'll likely lose.
You'll likely lose for business reasons. On some level, we all know that a SAAS on a critical path is a bad idea, but we do math. Our SLAs are not 100%, but we can accept a few hours of downtime a year. And, the probability of a service you've seemingly vetted being acquired and shutting down this quickly is low. Is the probability of a shut down like this high enough to justify adding and testing redundancy?
The math usually works out in favour of bug fixes and new features > redundancy.
It is extremely irresponsible to forward sensitive user data and site interactions to a third party, even if it is in the name of spam/abuse/scam filtering. That its implementation brings down production websites in the case of service failure only adds insult to injury, originally caused by plain-awful product design.
If there's one thing that you should be doing in-house when you run a large online community of privacy-conscious users, it's the kind of service that Smyte provided. If sound business reasoning had prevailed, npm would have suffered no downtime.
I don't agree that it's irresponsible to forward sensitive data to third parties. Rather, I believe that intelligent companies have intelligent policies around sharing data with third parties. Some third parties provide an amazing service that would be extremely expensive for companies to mimic in-house. And, you cannot convince me that the probability of a well vetted service shutting down as fast as Smyte did is high enough to justify bringing this in-house when you have other things to work on.
I wonder how much Twitter is going to pay on top of the acquisition cost of what I can only imagine constitutes breach of contract with all of these companies. (If it doesn't, why even bother having a contract with a duration?)
Acquirers usually prefer asset deals for this reason as it allows them to leave unpleasant liabilities behind. There's generally some sort of shell left that goes through wind down but it has few/no assets attached. In this case you can try to go after the original investors including founders or other shareholders (the IRS may do this if there tax issues) but since there's nobody home you are less likely to get much out of a favorable judgement.
Moreover recovering damages in this kind of case is time-consuming, doubtful, and too late to fix anything. (Like closing the door after the horse is gone.)
What's left is reputation--the names of the founders of Smyte are listed at https://www.crunchbase.com/organization/smyte. I would have some pretty hard questions if they ever showed up in a deal or looking for work.
p.s., According to the Techcrunch page "Smyte stops bad actors on marketplaces and social networks." You can't make this stuff up.
I expect one or more lawsuits to emerge, maybe as a class action. It might not come to trial and simply wash up as a settlement.
I am not now, nor have I ever been, a lawyer.
Self-service web services on the other hand generally come with take-it-or-leave-it conditions of use. When's the last time you redlined Github terms of use when starting a project? ;)
Apparently, they were a dominant player in this space, such that its shutdown impacted such a wide array of significant-sized companies, from ZenDesk to TaskRabbit to Meetup. Even more surprising is that they were part of YC (W15), and yet from what I can tell, have only had one significant mention on HN from 3 years ago (65 upvotes, 11 comments)
https://news.ycombinator.com/item?id=9758464
I would've guessed that Smyte would've been the kind of Silicon Valley business that was made for media hype (what with all the attention/criticism given to fake news and Google/FB/Twitter moderation). Then again, it does seem that content moderation -- important as it may be -- isn't a field as sexy as AI/deep-learning and other forms of automation.
There might be a carveout for consequential damages (which would be huge here), but this seems like it will trigger a number of sizable lawsuits against Smyte's new owner.
I guess the liability is covered by the usual "no liability outside provision of the service" clause - so you can't claim that you were damaged by their lack of service. But I can see a lawyer making a convincing case that the liability should rest with Smyte. IANAL, obviously, but it seems like the sort of thing they love to fight about ;)
gets popcorn this one will be interesting...
In the former, you can take the assets (including employee contracts) without the liabilities.
There's actually a reasonable quora answer (for once) on this, saving me from writing it up for you :)
https://www.quora.com/What-happens-to-legal-contracts-financ...
Don't know, not a lawyer.
At the end of the day, judges (and appellate panels) have to interpret the law and they don't function like automatons. They take into account the spirit of laws/contracts as well as the letter and most have enough common sense to get missed off at these naive tactics.
(All of this is a good reason i hate LLC's. They aren't necessary anymore to actually do risky innovation in 90%+ of cases. In a non-LLC, they shareholders would be liable, and then you'd still have someone to go after)
Usually, they would use the cash to wind down.
If nothing than just getting a pound of flesh from twitter.
Astoundingly unprofessional.
So... what was the Service Level Agreement? and the Terms Of Service?
People get all hyped by -aaS stuff, but services do go down, and if you aren't the one controlling when and how they will go up, then you better have a contract with the Service Provider on some terms you like.
Even then, prepare for when they go down, because they will sooner or later, for shorter or longer periods of time.
I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
This is always trotted out and it's, by this point, a completely content-free thing to say. Short of removing yourself from the economy and society in general to live off potatoes grown in your own dung, you're going to rely on some external services and the fact that, for the most part, the parties you do business with will behave responsibly. When they behave very irresponsibly, unpleasant things happen and people quite reasonably complain.
For starters, most smallish companies I know of use housing for their servers. So, they actually do own the hard metal servers and have full access to them. Both over SSH and on the console for debugging purposes. The housing provider generally replaces HDDs in case of disk failures. naturally, you're still dependent on electricity and internet, but even that can be mitigated by spreading your servers across several locations.
All of that is obviously possible with AWS, GCP or other VPS hosts, but doing it yourself is not that much harder if you've got an experienced ops person that wants to bother with all of that.
but i repeat... Its often not worth the time investment to get that system properly set up. Just renting VMs or going straight into dockerized microservices is an increasingly better alternative
Yes, any of those sounds like a good enough plan B. You don't need it to be perfect, just good enough to avoid getting stranded in a pinch. Then you can start shopping around for someone who does that one little thing much better than you do. That way it no longer is a show stopper and rather just an optimization problem.
* Malware detection?
* Customer support software?
* Website uptime monitoring?
* Accounting software?
* Email delivery?
I'm not going to argue that one is better than the other when it comes to SaaS vs self hosting. But people in here seem to have very short memories when arguing that self-hosting isn't at all viable. In fact I have personally supported all of the above in self hosted versions in the last couple of years.
I also have the experience of managing many of these services, and I therefore know just how much more productive you can be when you don't need to do it yourself.
Of course there are trade-offs; business is nothing if not a game of tradeoffs. We're just talking about what is optimal.
[1] Not to mention how much more stable and secure; Self-hosting can involve huge security and reliability risks.
I don't believe what is optimal can be generalised like that. "Optimal" is going to be specific to the business in question. Which is why I'm a firm believer of leaving all the options on the table.
> Self-hosting can involve huge security and reliability risks.
In fairness so can the cloud. I personally don't see difference as 'risk' but more 'responsibility'. Some of the places I've worked have been PCI DSS and Gambling Commission audited so we had that responsibility already. Self hosting a few other resources didn't really add much extra in terms of our security responsibilities. But the case would be very different of lots of other different types of businesses.
It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.
So the discussion is easily resolved: don't be absolutist :)
Even that strikes me as a mischaracterization, or, at best, taking just one statement out of context.
It was an assertion about the folly of relinquishing control, not being prepared for that lack of control, and thereby subsequently suffering a prod outage.
I don't think anyone is suggesting never to outsource, or even that the answer to "build vs. buy" is always "build".
The point is not having a critical dependency on a single vendor.
I don't think anyone is even saying "never", either.
The question is one of risk.
The risk of your tax accountant "going down" (or being crooked or something) is sufficiently low, and the cost of making that redundant is sufficiently high, that I don't think anybody does that. This could even apply to general accounting software, though one might argue that's not critical in the same way and/or more easily replaceable.
There's a sibling sub-thread that essentially implies, if not outright states that there's no choice but to "buy" in "build vs. buy". Although it may be true that building is no longer a practical choice due to time/cost, under certain circustances, it by no means eliminates the risk if there's only one vendor.
It's also a decision that can end up having tremendous costs at scale, if not re-evaluated, especially with even a soft version of vendor lock-in, like AWS.
you did not include "so much that if it goes down it would cause a prod outage" in your quote, which is why the strawman claim was put forth.
This is yet another strawman. The original commenter found it "silly", which, I agree, was a poor choice of wording.
Perhaps "overly risky" would have been better. I don't know. I just take the most charitable reading, per the guidelines.
It also comes after an exhortation to have a contingency plan, so it's at least implied that the "silliness" isn't inherent merely to the dependence but to the dependence without such a plan.
> in many, it's a completely sensible tradeoff
Is the tradeoff actually considered, though? Or is there just an automatic "we're not a nuclear reactor" decision process?
> is 100% uninteresting
Were that true, I doubt there would be replies. This is not one of those traditionally emotional/political issues.
> Including or not including 'in prod'
I (again, charitably) read "in prod" as a metaphor for "business critical". I'm sure we can all come up with examples, even in Internet companies, where one does not necessarily mean the other (or vice-versa). In the instant case, it seems to apply, so it made sense as shorthand.
> It's just lazy grousing.
I would agree if there were no built-in suggestion on how to avoid the problem at all, but there were.
Just because the point has been made before doesn't make it any less valid, until it has been refuted.
Only if 'strawman' means 'thing I disagree with'. The comment says such a dependence is bad and how you have to prepare for it with SLAs or whatever. This isn't universally true at all.
I doubt there would be replies.
A lot of really boring, trite things generate piles of replies. That's why they should be avoided.
Just because the point has been made before doesn't make it any less valid
The stated goal of the forum is not 'an exhaustive, if repetitive collection of generically valid things'. Remember that time you wanted to print something and the thing just wouldn't print? That's bad and annoying. It's valid that it's bad and annoying. We probably don't need to talk about it in the general sense every time it happens, though.
Your local restaurant going out of business is no reason for you to starve to death, you just go to another one, or to a grocery store, or indeed grow your own food if you're in some post-apocalyptic world or got no money... but somehow a service provider going out of business is reason enough for your production to stop? No, just no.
Only if you pretend there are no companies that do in fact responsibly nail this stuff down. There's plenty. Your argument seems to be that nobody does it because it's hard.
There are pretty solid SLA's for that. We provide some services as kind-of-saas but the SLAs are watertight.
Usually in those kind of sla's you can't sell the company (or move assets) without guarantee of delivery of the contract. In some cases they (the contract holders) even own you if you screw up and go under.
I'm curious; what would you have these companies do in such a scenario? If 100% uptime is impossible then what you seem to be suggesting is that people should never use third party services or if they do use them they must not be in a critical path.
Are you suggesting everyone should do everything in house?
If you have to use non-free software, buy software, not services.
If you have to buy services, always have a well-tested fallback path for when that service is not available.
This was in order of preference. If you can, use free software that you can fix and support yourself if need be. If not, buy the software and run it yourself, so you can at least control its operation. The last option is to by SaaS, and if you do, you need to have contingency plans in case it goes away. You should take this risk seriously, and factor it in to your decision when buying SaaS, and invest the engineering resources to make sure you aren't overly dependent on it.
What I'm suggesting is that people should use external services for their lower costs, better performance, or extra features, not to 100% depend on any of them.
There is a popular C# package called Polly that has all sorts of fault tolerant strategies.
https://github.com/App-vNext/Polly/wiki/Transient-fault-hand...
In this case, use a Fallback strategy that responds to an API failure with some type of generic success response.
I postulate more than few roles were halted at that time.
I'm calling it now: B2B startups will soon have to sign poison pill contracts that specifically detail what happens if they're bought out. This might ruin the "buy them for their people and discard the business" method of hiring - or at least it will if the customers have any sense.
I had assumed Twitter was going to be paying out a bunch of breach-of-contract penalties, because this is basically what contracts are for. Is there a standard way around this?
Also I wonder how many customers are staying quiet, because "our fraud protection provider just stopped working" isn't something they want to reveal.
(Disclaimer: I work for a smyte competitor).
I'm confused; how do you know these companies did not do that? In my experience even small companies will typically abstract away a third party service so that if it does fail it fails in an expected way.
What you seem to be suggesting is that all of the companies failed in expected ways. I didn't see that in any of the articles. Did you have a source? I'm curious if it hit some harder than others.
That's an outage...
>manual verification of new users
So is that, unless you have tons of customer support reps sitting around doing nothing when shit hits the fan.
Seems unrealistic especially since the majority of times when these services go down it's likely just a short time so they probably have abstracted around them to avoid unexpected failure and just have routine, down time failures.
Every tech company big enough probably sees the same writing on the wall the rest of us do that it's time to press their advantages. That means shutting everyone else out.
We probably all have mission-critical parts of our infrastructure that aren't core competencies. We probably can't afford for them to go away tomorrow, but now it's time to expect and plan for it.
Get ready to ride the consolidation wave.
The goodwill itself has to be worth a TINY bit.
Companies like Twitter whose core business is to shit on their users and customers don’t care about goodwill.
Imagine if Google decided that tomorrow at 6AM PT they were pulling the plug on YouTube embedding, or GMail / GDrive integrations. The court's going to say, "They put in the contract that they can change the terms of service at any time, and they're not liable for outages or lack of service. And you signed it. Sucks for you."
Then again, I'm just anti-corp in general and this gives me better reason to avoid Twitter.
This will be how it turns out for a few big players with deep enough pockets. Surely though this is hurting somebody's business that can't afford the lawsuit -- potentially to the point of putting them out of business. That's good for Twitter (and especially Twitter, as precarious as they are).
We've all joked for years about how FAANG pay their engineers so much mostly to keep other companies from being able to hire great engineers. We've gotten the big three slapped for colluding against their employees on wages. Why do you think they would engage in some anti-competitive business practices and not others?
Not a lawyer, just interested.
1) If your business depends on Internet access you shouldn’t pay for a residential plan. You need an SLA.
2) Natural disasters happen. Not having emergency drinking water is very naïve.
3) If your business depends on electricity you should also get business rate with SLA and maybe have a backup generator or two. Heck, I have a UPS just to not lose the 15 minutes of work that’s not automatically backed up.
4) Yes, get a weapon (preferably a gun) and at least know enough first-aid to be able to stop a bleeding. Keep an aspirin on hand in case of a heart attack. Know your emergency exits and evacuation plan. Make sure you have the mindset to leave all your possessions behind in a burning inferno.
All of the above used to be common sense before technology made us think we are immortal and entertainment distracted us from real world.
I think the issue here, other than earthquakes being potentially very deadly, which makes this comparison a little absurd, is that they are in a sense unavoidable.
On the other hand you can avoid using SaaS. For example, there’s enough market demand for Github to provide on-premises secure hosting. Clearly then, if SaaS as a whole is seen as fundamentally unreliable it will create more market demand for similar on-premises, or open source, or otherwise “buy not rent” model services, which I see as the motivation for OP’s comment.
What I've seen was a select group of companies and their users who've been inconvenienced by the abrupt shutdown of a service; and whose unpleasant experience should be used to teach every user - this can happen to you. I would not be at all happy if, say, the servers for medical tracking software used in hospitals around the world went down the same way; or if suddenly air traffic control software that every airport uses goes offline.
This particular situation doesn't cause potential physical harm to a human being. It's not actually an emergency situation. That, to me, is a major difference.
You don't spend a lot of time on Facebook, do you?
I bet their term of services mentioned that they can cease operations on no notice.
No company lawyer I know would willingly put their signature on such a contract unless the other side is using a free tier. If someone actually signed 3 year contract with such a clause, congrats, you managed to throw a lot of money into a black hole.
https://techcrunch.com/2017/11/01/google-will-pull-its-qpx-e...
I'm not sure how good apple is, but when they buy software available on Windows that version usually gets killed. (See Logic music production, killed 3 months after purchasing, but presumably the old installs still worked.).
Even Halo was demoed on mac before MS bought it (The mac version never came out)
https://www.reddit.com/r/SuicideWatch/comments/5jin1q/warnin...
https://www.reddit.com/r/SuicideWatch/comments/5x60f0/update...
I'm not sure they know what interop means.
Something something, a good craftsman doesn't blame his tools.
Which is especially weird for Twitter. Their official iOS client was a third-party API client they bought. Third-parties created @replies, #hashtags, etc. If anyone should recognize the important of third-parties using APIs, it should be Twitter... yet they've been probably the most abusive of the large APIs.
Twitter will never change how they do business, replacement is the best strategy.
What an absolute debacle. This doesn't bode well for the entire Twitter platform.
I went through an acquisition of an unsuccessful company. My employer created a shell which is what was acquired. The old corporation wasn't acquired and existed as an independent entity to wind down contracts / deal with the acquisition hold-back, etc.
If eg smyte was out of money, this is plausible. And all of a sudden no-one was there to run the old corp.
crunchbase says they raised a $4m A in Mar 2017, but linkedin says they had 23 employees. They could burn that in 1.25 years with 23 employees and have hit a cash crunch.
I have no knowledge, just speculation.
$4m over 1.25 years (and don't forget comp costs the company 1.3) == $2.5m comp to employees per year. That is not much for 22 employees.
math: (4/1.25)/1.3
Otherwise it would be a good way of making debts disappear.
Also, it's quite plausible that Smyte had negative value before the acquisition - that it's going to be an "empty sack" not because valuable assets were transferred elsewhere (which can be reversed in courts) but because its debts already exceeded any value it had, that it was an empty sack in the first place and no debts can be paid in full in any case.
* If the assets were bought, they'd be the property of the original company, and the money would go to the company, which would therefore have money to be sued for.
* If the directors took that money out of the company, full in the knowledge that they had legal liabilities as the result of terminating service, at least in the case the UK (and I can't speak for US law), they'd be guilty of a number of crimes, as they are responsible for acting in the best interests of the business.
There are startup acquisitions which result in lots of money being paid to investors, and there are acqui-hires which amount to a pre-bankruptcy firesale to get part of the money back, but resulting in unpaid debt and bankruptcy anyway. I don't know about Smyte's financials, but it may well be that their situation is like the latter case.
I don't see any reason to suppose that there some big bag of money (even post-acquisition) that's somehow illegally taken out of the company. It may well also be that the money paid to the company itself for acquisition is insignificant, and the rest is in form of some "hiring bonus" to make the transfer of employees work, so not something that the debtors can claim.
Any side payment to the directors and staff outside of the acquisition - such as a "hiring bonus" as you suggest, which lead to the sale of assets at a lower price than would otherwise be the case - I'm pretty certain (again, in UK law, and I'm certainly not a lawyer), that this would be seen by the courts as a bribe, ie. money paid to perform their duty to the company improperly.
See item 1.2 from the Bribery Act 2010: https://www.legislation.gov.uk/ukpga/2010/23/crossheading/ge...
Obviously not exactly relevant in this case, but I'm guessing there's a US analogue.
In my failing startup that was acquired, all eng got a carve out. It was basically a bonus to get us to remain employees after it became obvious the company was failing and was going to be sold.
I'd imagine the legality goes like this: without that bonus, I certainly would have left. And there would have been nothing to be sold. So it was long-term better for investors.
So you will see payouts and bonuses to executives to get them to stay with the sinking ship in hopes of eeking out more money before the collapse, but if you were to just pay a bonus to an executive on their way out, you would have a very hard time justifying that and could be sued for breach of your fiduciary duty.
This type of uncertainty does lead to a situation where potential customers are wary of using a startup service — I can’t build a business off your service if I don’t have confidence your company will want me as a customer in 3 years.
Ultimately I think we’re witnessing the final consolidation of the software industry. I expect the cloud/software industry of the next 30 years to start to look much like telecom over the last 30 years.
Probably worse for consumers short term, but ultimately the plumbing isn’t all that interesting and it will hopefully enable the next round of innovation.
> Because the reverse triangular merger retains the seller entity and its business contracts, the reverse triangular merger is used more often than the triangular merger.
I was going to say "I wish that I could do that!" But then realized that bankruptcy comes pretty close.
But if it is all within the states, it could be just that.
> If eg smyte was out of money, this is plausible. And all of a sudden no-one was there to run the old corp.
Sound more like a tech/knowledge acquisition to me.
4M$ for 23 employees only works if you have proper financial management though.
The problem, it seems, Twitter got the timing __all wrong.__
Twitter is large enough to plan these details. I think it's fair to assume they are at fault until shown otherwise.
If you buy the assets of the company then usually you do not buy the contracts unless they are specified as sold.
In both cases if there are change-of-control clauses those could trigger with various penalties or rights becoming unlocked.
And if you're thinking "well they could just personally reach out to their customers if they're fired because it's the right thing to do" -- if the last-minute nature of everything is any indication, current/former employees were probably ordered not to say anything.
I wonder if that triggers any clawbacks in the purchase price, as Smyte was running a substandard service.
I find it hard to believe being acquired magically relieves you of your contractual obligations... could this be cause for litigation?
I guess the real issue is that smyte's contract probably said something like (from GCP's terms of service):
> 13.1 Limitation on Indirect Liability. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, NEITHER PARTY, NOR GOOGLE’S SUPPLIERS, WILL BE LIABLE UNDER THIS AGREEMENT FOR LOST REVENUES OR INDIRECT, SPECIAL, INCIDENTAL, CONSEQUENTIAL, EXEMPLARY, OR PUNITIVE DAMAGES, EVEN IF THE PARTY KNEW OR SHOULD HAVE KNOWN THAT SUCH DAMAGES WERE POSSIBLE AND EVEN IF DIRECT DAMAGES DO NOT SATISFY A REMEDY.
> 13.2 Limitation on Amount of Liability. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, NEITHER PARTY, NOR GOOGLE’S SUPPLIERS, MAY BE HELD LIABLE UNDER THIS AGREEMENT FOR MORE THAN THE AMOUNT PAID BY CUSTOMER TO GOOGLE UNDER THIS AGREEMENT DURING THE TWELVE MONTHS PRIOR TO THE EVENT GIVING RISE TO LIABILITY.
Given there are actual damages coming out of this to paying customers perhaps we’ll finally see them taken to task.
Either way, really poor from Smyte. No reason to immediately turn off access
This problem creeps up with external mfa/2 factor auth APIs that go down, bringing down the main login. Some choose to skip mfa if down, some don’t
This seems like a clear breach of contract. I can’t imagine how anybody thought it was a good idea. Maybe it was a mistake, although that wouldn’t be a whole lot better.
PR will come up with all sorts of reasons said acquisition will benefit both company's customers but I've yet to see it.
At least in Austria we have a law that exactly tries to prevent such things (§38 et seq. UGB): when you buy a company you enter into all of their existing (non highly personal) contracts and obligations, unless you specifically don't want that, in which case you must tell the other side and give them some time.
Genius. /s
This has happened so many times that it can really affect how small companies are preceived as too risky too deal with by larger customers. sigh
Say what you like about Facebook but it does at least seem as though somebody is calling the shots in there.
Twitter and Google wins by having fewer competitors, but the customer loses doubly so. If Facebook acquires every fifth social network, followed by immediate shutdown, users will eventually stop using them, hence less competition as a whole to FB.
Twitter is a pretty big part of any modern online company’s strategy, whether they like it or not.
In your own example
"Twitter likely doesn't want to allow people to be able to test against their spam shields by spamming other people"
Yes, the attacker may find a new pattern that can bypass the shield, but as soon as that pattern is added to the machine learning recognition, they are then protected against it on their own system.
"spam detection is just pattern recognition"
The more patterns you can analyse, the better your recognition?
Friend worked on a large, multi-institutional risk-modelling project ... until the largest institution decided it had enough data on its own, and didn't need competitors who were competitive in this regard, and pulled the plug.
Law of large numbers (statisttics) is powerful. Even for complex (multvariate) models.
It sounds as if there may well be other considerations at play, but I'm answering your question specifically.
Don't give them your data, don't sell your ads on twitter, don't develop for Twitter.
There are alternatives.
Replacing Twitter is simple. Have a proper support channel (Intercom, etc) so your customers don’t need Twitter to get support from you, and use your existing marketing tools (email, push notifications, etc) to keep in touch with them. Done.
"Winding down" literally means that before shutting down, there is an incremental process with the explicit purpose to smoothly move into the transition. There wasn't anything like that, not even a little.
What's especially rich is that Engineering VP Mike Montano nonsensically repeats this phrase after the fact: "we made the difficult decision to wind things down right away". That's plain bullshitting.
He should have said, "we made the difficult decision to shut things down right away". But that would have sounded super irresponsible! But it was super irresponsible, and describing it differently is just dishonest.
And that wasn't accidental, this entire situation is exactly about the difference between "winding down" and "shutting down".
Why do we allow people to communicate in this way? (publicly and/or from a position of leadership). Twisting words in plain sight. There is no good motivation behind this, it represents messed up priorities between appearance/saving-face and responsibility.
I don't know anything else about this Mike Montano. Maybe he's a very good manager when not talking like this. But leadership comes with responsibility and when you twist words like that, it's only because you're trying to wriggle out from underneath the responsibility.
1. Trust small, third-party, closed-source SaaS vendors at your own peril.
2. Do not spend/waste your time integrating with random third party services of questionable/unknown sustainability.
Does the whole page hang? Do UI elements disappear ot overlap others? Do JS error messages surface in the UI? Can the user continue with core functionality or at least receive an explanatory message?
I usually hate it when people blame the victims for being so stupid whenever something bad happens. But some of the victims in this case are corporations with lawayers.
So either this really isn't a mission critical service and they just took a calculated risk, or these lawyers haven't done the job they were hired to do, or Twitter is in breach of contract.
I would say we should use this as a sign to migrate away but we've had enough warnings already. I guess Twitter is too big to fail?
But if a standard exists, that is, we've split the API from the primary implementation of the API, then we as a re-implementer no longer need to replicate bugs and all; we just need to do as the standard says, and anyone relying on cpython bugs is, well, at their own fault. Compatibility claims are no longer dependent on implementation-specific details.
And of course, it doesn't matter until a second implementation begins to appear; there's no reason for cpython to follow (or have) the standard unless it actually wishes to be compatible with jpython. At the same time, it makes a lot less sense for a second implementation to appear as well, as they have to put a lot of extra time and effort into matching cpython's implementation details (as far as they matter; a difficult determination itself), and this also makes it more difficult to produce an alternatively-optimized implementation (focusing on say, memory-usage instead of speed or whatever). So it's also a chicken and an egg problem.
So ideally everything would be an official standard, and we'd put little to no effort into simply matching whatever the popular implementation is at the moment, and we could criticize the mainline implementation for failing to adhere to the standard.
In this particular scenario, to replace smyte everywhere, transparently, you'd both have to adhere to the API and whatever details incidentally exposed by the API, because you have no protocol to actually reference (the protocol is whatever smyte was doing). But thats also particularly difficult here, since you can't even test it against an implementation of smyte anymore..
Kind of like how "cookie" is not web specific but unless mentioned in an explicitly nonweb context, it means browser cookies.
Perhaps they were in violation of either the GDPR or their customer data agreements.
[I've removed the founders' names, even though I strongly disagree with the characterization that I was somehow acting out of rage. My intent was rather consumer advocacy. I've had a service cut off in a similar way, and I view such behavior as being unethical. Fine if you disagree, but please don't characterize my emotional state that led to this post (which itself is in violation of HN's admonition to "Assume good faith")]
People justifiably have certain expectations when they pay for a serious b2b SaaS that they don't have for a free social media service.
I'm also a bit irked with having to prune all the time. I follow someone or something that I expect to be about something and really it's just a lot of noise about other things...
Again, if it turns out that Twitter lied to them about how the service would be wound down, I'm sure that will come out in due time. If I were one of the founders (and I know at least one of them has been often on HN), then I'd be on this thread making some explanations right now.
Finally, the name of the founders isn't some kind of secret. This is public information. The founders have been featured in many media pieces. They sold their company to a public company.
All: please keep the internet rage reflex off HN. Same for the online shaming culture.
https://news.ycombinator.com/newsguidelines.html
(Edit: I didn't mean that you were feeling rage, but that by posting this way you were stirring it up. In any case, thanks for removing the names.)
I think there is an issue of personal integrity here, even legally.
When I worked for a large F50, our CEO's name had to be on every email we sent out from our German staff - though I'm not sure of the specific German regulation, 'his name is on it' because it's a matter of integrity to the organization.
The founders are generally 'Officers of the Company' - and this implies both a legal/fiduciary and moral obligation in America and most jurisdictions of relevance.
I don't believe that the 'limited personal liability' concept abnegates the responsibilities of the Officers of the Company in this specific regard.
I also don't believe this is a matter of arbitrary doxing if the Officers of the Company are making decisions which are considered foul play in terms of reasonable contractual obligations.
Unless there is an issue of national security or likewise, there's simply no excuse for ending service without reasonable notice.
This is not a political, or socially nuanced discussion such as those involving harassment etc. - this is a matter of commercial civility (and legality).
In American law: "In limited circumstances, such as the sale of the small business to a new owner, the business judgment rule does not apply, and it becomes the burden of CEOs and company directors to demonstrate that their actions were in the company's best interests." [1]
(FYI - the 'business judgement rule' protects Officers from liability.)
I guess I understand we don't want to be doxing folks here ... but I do think it's reasonable that individuals should be held publicly accountable for their commercial actions - for example, someone does an ICO and then - legally - walks away with the cash ... we should rightfully be wary of investing in their next ICO or company for that matter.
The inability for executives to be held accountable for actions of the company is a core problem in present day capitalism, in my opinion.
[1] http://smallbusiness.chron.com/legal-relationship-between-sh...
I know they're an YC company, so I don't want to ruffle anyone here, but I do think a class action lawsuit against Twitter is warranted, even if it has a shaky legal foundation.
I believe that Twitter's actions - and the response - will be duly noted among M&A camps and the last thing we want is this to become standard practice. This is just 'bad acting' and it will come back to bite everyone here in startup-land.
I risk my erstwhile happy HN account by getting into a fuss with pvg, but he indicated that we want to 'wait to find out what's happening' - I would argue that in light of Twitter's 'total and immediate blackout' the onus is upon them to provide forthright information, and that absent that, we can assume 'bad acting' given a total blackout without any information.
Maybe we can use this as a lesson and the powers-that-be can provide some leadership, and possibly push towards a 'best commercial practice' like '120 days minimum notice' or something like that as a standard.
This is exactly the community to do it in, as I frankly don't see where else anyone has any broad legitimate authority among early stage startups such as it exists here.
Yes, we are aware that Smyte is YC W15. Thanks for the reminder that you protect your own from base accountability at high cost to others.
Since it sounds like you follow these things closely, I'm surprised you'd be so cynical. If there's literally a single case where someone did that where we haven't chided them, I'd like to know about it. In fact if there are any personal attacks of any kind on HN that we haven't moderated, I'd like to know. The likeliest explanation is that we just didn't see it. In the present case a user (not connected to YC as far as I know) emailed us to complain about the comment, I looked at it, and replied—same as it ever was.
I predict some back-pedalling later today and access switched back on and properly 'wound down'.
If not, customers with contracts should have every right to angrily sue.
1. Smyte had a security flaw that could potentially (or already did) affect twitter customers.
2. Leaving Smyte service on could do alot of damage, so turning it off would be the sensible thing to do.
3. Telling the world about this security flaw would create a really bad backlash.
https://www.buzzfeed.com/alexkantrowitz/how-twitter-made-the...
even as those eulogies were being published, things started changing. Twitter began beating earnings expectations. Star ex-employees trickled back in, finding a new, more positive internal culture than the toxic one they’d left. Advertisers came back too, as did users. The company finally began addressing its trolling problem. And its stock, once unappealing to analysts like Nathanson at $14, is now trading above $46.
It’s still somewhat taboo to say it, but it’s no longer possible to deny it: Twitter is making an unexpected, somewhat miraculous comeback. It is the first major consumer social company to lose users and start growing again in a meaningful way.
I don't know if Trump is really causing Twitter usage to increase, but it has certainly made people who don't use Twitter more aware of it.
If you compare the awareness/user ratio on the various social media platforms, I bet Twitter's is pretty high, which means they have potential for significant growth if they can improve the UX and the "how do I use Twitter? Why would anyone use it?" problem.
I remember following Scott Adams, whose work I enjoyed. Before he started talking about Trump, he had some 40k followers.
Since Trump, his follower count has ballooned to 250k and he almost exclusively talks about Trump now
The real surprise is that they've failed to effectively monetize their huge, huge audience.
Yesterday I wanted to look at some joke account (by correctly spelled name), Google yields nothing from the domain, Twitter wants me to sign up to search there ...
(Not a habitual user, or even reader, as you may guess.)
However that situation would usually never be described as an acquisition by the buyer. (The seller might put out an announcement that says “We’re proud to be joining FooGleZon, it’s been an amazing journey!”)
In Smyte’s case, Twitter PR does use the word “acquire”: https://blog.twitter.com/official/en_us/topics/company/2018/...
That sounds like something else than an “amazing journey” non-acquisition. But IANAL, just speculating out loud.
Hmm, is the new owner not bound by that contract?
Create a product, gather funding.
Attract customers.
Get acquired, founders make bank.
New owners trash the product, stranding customers.
Tell me again how this is good system not for only making money but actually providing value to customers.
I would argue that most new companies (at least started by and solely run by technical founders) fail exactly because of that.
Product-oriented people tend to think that the success of a business is due to its product(s), when the reality is businesses succeed due to their ability to sell their products. Hence, the most important thing is actually being able to properly find and target an audience, which should be the first step of starting a business.
The second step of starting a business should be to test marketing channels to make sure the selected audience can be effectively/efficiently reached. Only after this has been properly done should a product be ideated, based on the knowledge acquired about the target audience and the marketing channel(s) that work.
Just my 2 cents to the HN community after having learned this lesson the hard way, failing multiple times and starting several businesses.
What's more, you could be product focused and still produce bad products. I just left a startup that seemed to have really bad decision making in areas like "what feature should we build first/second/third", and major areas needing design and careful thought were instead treated as an afterthought. But the tech was solid and lots of work was getting done, so... good enough?
I'm a tech worker who doesn't live and work in the Sili Valley reality distortion field (SVRDF). From outside the SVRDF it's hard for most people to tell one company from another. They all look like SV. That means they all look a little untrustworthy and a little dangerous.
Now, sure, this Smyte had an insider's brand, sort of like "key gaffer" in the movie industry. But their shenanigans, and Twitter's, just made it harder for all the rest of us to sell to prospects outside the SVRDF.
Y Combinator, will you consider doing some time on basic business ethics with your incubator residents? (I'm talking about stuff like "don't screw your customers, suppliers, or shareholders: keep your promises and honor your contracts.")
Does the rapacious acquisition game of Silicon Valley and seemingly increased rate of consolidation of startups to established giants feel weirdly like he days of Ma Bell to anyone else?
It seems like more and more There's tales of these acqui-hires with promises that the aquirer will present some kind of "polished" alternative but under the larger brand and I've found quite frequently the promised offer-if ever shipped at all-is far less superior and far more self serving to the brand than what was previously offer by the acquired when they had more on the line by operating as a startup.
Or am I just a cynic?
Do you have any data to support increased acquisition rate?
Welcome to being wrong on this if some good soul does have the data on this, though. It's why I asked, maybe someone has research that would prove me wrong and I'll gladly take being corrected on the matter.
EDIT: just remembered WhatsApp. This happened recently: https://www.cnet.com/news/whatsapp-founders-may-have-left-1-... and it doesn’t seem like them selling did anything positive for their product or customers, only their bank accounts. Seems like they couldn’t even wait for FB to open up the golden handcuffs.
Anyone have examples where the acquired company goes on to be really amazing, on a trajectory that they wouldn’t have had access to without the big injection of funding from the acquiring company? I can’t name any, but that doesn’t mean they’re not out there.
Instagram is a fairly high-profile example.
This is such a constant thing that right now, I'm very reluctant to use startup-made SaaS for anything that isn't throwaway. The expected lifetime of a service is just extremely low, and the cost of integration (lock-in due to platform idiosyncrasies, and your data being taken hostage) is too high.
You couldn't develop telephony product at all unless you were Bell.
So, anyone got a non-paywalled version of this?
Is anyone else as tired of this schism of material gain over all other considerations? If the tech industry wants to end up with the same ill repute as Wall Street this is certainly the right way to go about it.
"Tech company takes money and screws over customers" is a headline that writes itself.
How many more examples do we need for this to sink in?
Electricity and water are heavily regulated and often government owned, not run by Silicon Valley cowboys.
My food supply is a self organising network that can route around any single point of failure. This is about as far removed from SaaS as you can get.
Yet, you still get Puerto Rico (electricity grid recovery failures of a few levels), and Flint (no clean water for you, because no).
Any usage kf SaaS would require a solid contract in place and a plan B (and C, and D, etc) to reach even a fraction the reliability of e.g electricity or food.
Choosing the potential downtime is not a failure, if you considered your options.
At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.
And if that happens people (I'm looking at YOU, software and devops engineers, with your damned sketchy business cases, desperate to move projects onto AWS mostly so you can bolster your CVs) will only have themselves to blame.
Fortunately I suspect there are enough of us jaded and sceptical types around to avoid it.
If they aren't, either they're incompetent, the CTO is, or you don't really have a company yet.
https://github.com/terraform-providers/terraform-provider-aw...
If you are a small company, the difference isn't worth the risk and the expense. If you are a large company you can probable negotiate with Amazon.
That doesn't even take into account all of the dependencies that your developers have taken on AWS services....
I think you are confusing "agnostic" with "compatible with lots of different providers", but will require a total rewrite if you want to switch providers.
Two things about that -- hardly anyone ever changes infrastructure just to save a little money and their code has to be so generic that they can't take advantage of all of the features of the platform.
The chances are far greater that your company will go out of business way before Amazon stops offering AWS. Theory is great that Terraform allows you to seamlessly move infrastructure between cloud providers, but the amount of risk and regression testing it takes to do so would make it a non starter for most companies.
This isn't just about cloud provider lock-in, but something you need to do to have a good DR strategy.
Back on that Christmas Day that Linode got DDOSed, my company was 100% on their infrastructure. I only had to work a couple of hours that day because I had worked and planned in advance to launch our infrastructure on another host. I saved my company from a multi-million dollar loss that day.
If all you're doing is hosting a bunch of VPS's, you're really not saving yourself too much over just using a bunch of colocated servers. You can also get much cheaper, less reliable, less geographic diverse hosting somewhere else.
But I would have my head handed to me if I even suggested to a company to base their entire infrastructure on Linode instead of AWS, Azure, or even GCP.
Managing tens of millions of dollars in physical and cloud infrastructure at multiple providers/dcs for a publicly traded company like I am now? No, that's not the reason to use cloud infrastructure at all. Cloud in this case is more about capacity planning and avoiding managing hardware inventory than anything.
I know their video encoding platform is heavily tied to AWS because the teams that built it optimized for cost (which for them makes a certain amount of sense) and they massively parallelize the encoding work. I talked to some of their engineers at re:Invent last year and it was kind of interesting. They aren't using a lot of AWS-specific stuff though, just AWS EC2 reserved instances.
I don't think that just because they're Netflix that everything they're doing is correct. This can't possibly be true.
I dare say the AWS sales pitch is even more effective at that level: "Increase your agility, reduce your operational overhead, scale out with the push of a button. Be like Netflix and Apple."
Most executives aren't well-equipped to probe the sharp corners that aren't in the pitch.
Does your business depend on Amazon Transcribe (or any other AWS-only service)? Do you have engineers competent to turn around an equivalent product for you in a short release window? No? (Don't lie to yourself. It's no.) Congratulations, you are now Amazon's bitch.
I (as the sole programmer) made that decision early on.
AWS is a none starter because of the gradual vendor lockin (also nature of our business means growth/scalability is very easily predicted).
But their business is built on providing this service. At what point would Amazon, Google or Microsoft say "we don't want that $xB in revenue, just turn it off. NOW!
Is that what you're suggesting is the future?
More seriously I can see cloud going the way of telecoms in that there would be a provider of last resort if a telco goes bust another operator takes over to provide service.
Of course, it's a competitive market, so the other two providers would eat their lunch.
He has written multiple deployment options. Kubernetes, Docker swarm, bare VMs, etc. He has tested all of them and written installation tools that can deploy to any of them.
For exactly the reason you imply, he doesn't trust any of these options to always be available. I think this mentality has a lot to do with why his service scales so well.
There's more to building a long-term, sustainable business than optimizing revenue at any cost. And part of that "more" is exactly the work you to do insure against future risks.
The old saying that no one ever got fired for choosing IBM could be applied equally to AWS. The companies that chose IBM in the 70s and 80s can still buy new compatible systems. The companies that chose Stratus or VAX, not so much.
Even businesses that chose Microsoft technology in the 90s when they were the clear leader have had a relatively painless upgrade and support path. They abandoned VB6 15 years ago, but you can still wrap your VB6 code as a COM object and interact with C#.
I’m saying that it is the safer bet to go with the largest provider with the largest customer base. In the 80s that would be IBM, in the 90s, MS over SUN (and the royal claustrof%%% that Java has become under Oracle) or Borland.
Let's look at yesterday's equivalent to AWS- Sun Grid. Sun was viewed back during the first dotcom era in much the same way that AWS is today. All that it would take to completely bork it would be an economic catastrophe that mirrors the first dotcom bubble. Obviously, Sun Grid is no more.
[1] https://news.netcraft.com/archives/2004/11/16/solaris_remain...
You can still buy Solaris based systems today but you would have plenty of time to migrate. But seeing the writing on the wall you would have plenty of time to migrate.
I think you're committing something of a "fallacy of the excluded middle" here. Those aren't the only two things that can happen. AWS could, for example, continue offering the service but not to you. We've already heard stories of AWS accounts being closed for various reasons, and it's hardly a stretch to imagine a CloudFront / Daily Stormer kind of scenario where Amazon decides to punt you for taking an unpopular political position, etc.
I would even say that your company going out of business is a bigger risk than depending on AWS.
I would say that I disagree, in at least some cases.
Whatever his salary is as an engineer, he is definitely underpaid.
[1] I would much rather work on a work related project using a new to me technology and if it requires longer hours to complete it, I’m find with putting in the extra time. It seems like more of a win, I get to learn a new technology, get to see it actually be used by others, and it will be helpful during either review time or worse case a resume builder.
Screwed? Maybe, maybe not. At risk? Abso-fucking-lutely. There are plenty of stories out there of companies who had their AWS accounts shut-down for some random reason, plenty of stories of AWS outages, etc. Even if Amazon does't pull a "Smyte", there's plenty of risk associated with being on (or "solely on") AWS.
By and large, anytime you are completely dependent on someone else's platform, you are increasing risk by putting your destiny in someone else's hands.
Back when it was more traditional to buy a software licences and self-host rather than pay a rolling subscription to an XaaS, a service going dark just meant that you were shut out of updates but not the software itself. Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.
I think people often underestimate the value of being able to manage their own destiny. They key point here is "manage" though: you have to actually do it - it's an active process.
If, on the other hand, you simply stick your head in the sand and ignore the situation it will eventually get to the point where it becomes extremely difficult and expensive to deal with due to a variety of factors such as no clear migration path, loss of institutional knowledge, and so on. I name no names, but worked for one UK company in particular with form here.
I'm not making an argument against SaaS (we use a few of them), but it's important to understand the trade-offs you're making in terms of ceding control.
Further, some of the biggest value they will provide is being able to see transactions between vendors. So eg calculating an IP address reputation score by seeing bursts of fraudulent purchases. That is impossible to do in house only.
But that's not what either I or the GP were talking about: we were talking about hosting a product that you'd purchased from another vendor on your own infrastructure.
The obvious downside to that is you don't have real time updates (as well as the usual other benefits that SaaS brings), which is why SaaS suites this kind of business well. But it can be done and was sometimes done this way "back in the old days".
Distributing the database to customers is still workable despite this drawback - antivirus systems are a common example.
I don't agree with that. Having the database only allows bad actors to see if they've been added to the database - not the business logic of how they ended up in that database.
You could argue that knowing you've been added to the database is still an advantage, which is true. But those same bad actors could still test the system with SaaS solutions too. eg when one acquires stolen bank cards it is common to use those details to make small payments for things like food delivery to test if those card have been blocked or not.
I wouldn't say it's preferable for the customer. It's just one of many options.
> but it's obvious why a vendor wouldn't be keen
Yes, SaaS obviously has benefits for vendors too but what's most preferable to them is making money. If every customer refused to do business with SaaS then vendors wouldn't prefer providing SaaS solutions (yes, I know I said "solutions solutions" :P). The reason SaaS is so prolific is because it offers advantages to customers as well.
When has SaaS stood for anything besides Software as a Service? Because software as a solution could describe almost any business model?
Granted there are more cases of when SaaS et al works than when it does not. And I do agree with you that it's no ok to shaft paid customers in this way. But that doesn't mean that it doesn't still happen. This is why I've have to consider the self-hosted verses XaaS debate myself writing Business Continuity / Disaster Recovery / et al plans.
I really dislike this idea that it's our responsibility to ignore inflammatory tones and trolling, instead of it being the responsibility of the poster to communicate clearer.
Contracts can be valuable but often they are over valued.
Companies also are not physical items that can disappear due to natural disasters.
The only ways they can disappear is by closing down, and that does mean honoring contracts, and by going bankrupt and that means using all leftover assets to satisfy all the parties whose contracts you’re breaking. And no, you can’t just transfer all assets away and leave a shell to go bankrupt because that is fraud and the judge can nullify the transfers.
Also, such risks can be insured.
Also, such risks can be mitigated in various ways - I recall having a contract where the source code of any product version delivered to us would be held in escrow at a third party, and in case the supplier went out of business or a bunch of other conditions, we'd get their code and legal rights to use/modify/maintain it ourselves or through some contractor.
A contract is between you and some entity - if you want to protect yourself against that entity "dying", then that can be done by a contract with someone else (e.g. insurer, escrow, etc) who can make you whole even if that entity is dead.
Your company isnt going to die without it (its not like Hootsuite losing access to the Twitter API or Heroku cut off from aWS) But it might be more effective with it.
This is BS.
Sadly, no.
The majority of services have complex functions often requiring key data and updates, and it's just not possible, let alone feasible to 'self host' most things.
But it's a neat idea ... maybe there's a solution there ...
If this was not a painful enough lesson, the next time will be. :)
You're going to host your own version of Google maps?
Host your own version of 'background check' or 'credit check'?
Host your own DB of global financial transactions?
Host your own instance of weather data? Geospatial data?
It's not possible in most situations, and not feasible in most others, unfortunately.
I've never worked on a service which involves the other things you mentioned, but I've worked on some map-based services, and we went with self-hosted OpenStreetMap instances for exactly this reason :)
A temporary production outage due to a rare situation is far preferable to being at the perpetual mercy of hackers constantly pwning whatever half-baked homegrown security system you forced your engineers to implement with inadequate support & resources. Real businesses don't operate on "I guess" solutions and they don't just fork over huge chunks of cash willy-nilly without doing at least some cost-benefit analysis and making sure they're protected in case of contract breach.
It largely involves multiple teams of full time employees working on different aspects of fraud prevention (some of them are even Competent Developers!) a few dedicated offices, some teams embedded in regional development offices, a bunch of bespoke software and it's ongoing development and maintenance.
Thats in addition to the core dev teams being aware of and mitigating any possible vectors within the platform itself.
At any kind of scale it's rarely as easy as 'hiring a few competent developers'.
If my site messes up then dodgy comments might get posted (unless they're blocked by other systems). If you misclassify an "is-fraud" check you either have a customer lose money, or a potential customer be denied service. Neither of those are great.
I suspect actually doing anti-fraud testing could be done simply with a few heuristics, knowing the kind of attacks I see on other sites I run, but the only way to be sure is to get a lot of data-volume and real-life results to compare against.
Sure, it sucks that people lost an API they were using, that much we know, but hundreds of votes for a “bad” commercial transaction on which we have no details?
The way I see it, a number of Smyte users just received a free billing period of API access?
Did customers pre-pay for Smyte service and not receive it?
Everybody got 30mins before lights out. It broke stuff.
> According to reports from those affected, Smyte disabled access to its API with very little warning to clients, and without giving them time to prepare. Customers got a phone call, and then – boom – the service was gone. Clients had multi-year contracts in some cases.