Twitter Cuts Off DataSift To Step Up Its Own Big Data Business
techcrunch.com
techcrunch.com
2010 - After switching to a game model and doing quite well - neck and neck with Zynga. We were walked into a Facebook meeting room and told a bizarre story how Mark was upset about friend feed forms. They explained that we could only share (by means of post feed forms) our games with people that already had our games. Pleading did nothing, they ended that model. Zynaga is still flailing around.
2012 - I changed companies and focused on mobile advertising. Another decent path with 12 employees or so. We got a phone call . [EDIT REMOVED REST OF STORY]
2015 - I received another phone call from a large company (They deal in search) . I wont go into this one.
* Phone calls from large companies are extremely bad. That means they don't want things in writing.
So sure, have fun with working with large companies. Just remember one thing: they hate you.
Let's call the company that called us [redacted]
Here you have worked and toiled away for years trying to get your big break. Finally it comes and you get a phone call or invited out to the valley to demo/talk about selling your blood, sweat and tears. You then realize your just getting stomped on and they may have just stolen all your hard work. You think ...huh this is how it goes for the little guy innovator against this huge tech company you held in such high regard.
The tech world is not a extremely friendly place to even a ton of Caucasian guys(it's all about who you know/who is behind you), yet alone to any person who isn't!
Nonprofits and governments are not driven by profit as a main motive and are in their own category.
It is 2015. You can download a bunch of apps from the play store that automatically record your phone calls.
Phone calls are as much "in writing" as emails, perhaps more so as they are much more difficult to fake.
Remember, Facebook et al. is the furthest thing from your friend, whether you're a facebook user or another thing in its ecosystem. Put your game face on when approaching or being approached by these companies. Document everything, because they will most likely end up f'ing you over.
edit: recording phone calls isn't just really good to protect yourself Facebook, it's also good for a whole array of other reasons. It captures synchronous information to keep note-taking easy, gives you a chance to look into everything a client requested/complained about and when when they brought it up, etc. tl;dr: get a proper phone system with recording abilities installed asap
In any case, this discussion about phone calls is a side-show. If Twitter is going to cut you off from their API, you can't avoid that by not picking up their calls.
Further, recording is fundamental to how digital devices work. It is unavoidable.
Your cell phone is not an analog device. The only question is: how long does your device keep a recording of the signal before deleting it?
Modern devices already make a recording of the 'call' (it is fundamental to the working of the device).
Consent is explicit in choosing to use a device that is well-known to make recordings.
There is no expectation of privacy due to the explicit consent mentioned above and wide-spread allegations and revelations of US intelligence agencies and their affiliates and subsidiaries copying and storing Internet communications, as well as being able to 'listen in' on communications as they occur.
Nobody's claiming that a phone doesn't create, transmit, and receive "recordings" in order to act like a phone. The pertinent question is whether/when you are allowed to make a recording of a phone call, for yourself, for whatever reason. You're being downvoted because you're missing the point.
>632. (a) Every person who, intentionally and without the consent of all parties to a confidential communication, by means of any electronic amplifying or recording device, eavesdrops upon or records the confidential communication, whether the communication is carried on among the parties in the presence of one another or by means of a telegraph, telephone, or other device, except a radio, shall be punished by a fine not exceeding two thousand five hundred dollars ($2,500), or imprisonment in the county jail not exceeding one year, or in the state prison, or by both that fine and imprisonment.
The one that explicitly exempts radio devices? The type of devices that modern cellular phones are?
And people are arguing about 'expectations of privacy' and 'explicit consent', when the actual statute uses 'confidential communication'? My phone calls are not confidential and they occur over radio devices.
HN's behavior here is rather shameful.
You are talking about "explicit consent" and "expectation of privacy" and so on that sound legalistic but have no meaning in this context, while ignoring the fact that there are laws, rulings, and precedent already on this issue.
Of which, none you can cite?
>California's wiretapping law is a "two-party consent" law. California makes it a crime to record or eavesdrop on any confidential communication, including a private conversation or telephone call, without the consent of all parties to the conversation. See Cal. Penal Code § 632. The statute applies to "confidential communications" -- i.e., conversations in which one of the parties has an objectively reasonable expectation that no one is listening in or overhearing the conversation. See Flanagan v. Flanagan, 41 P.3d 575, 576-77, 578-82 (Cal. 2002). A California appellate court has ruled that this statute applies to the use of hidden video cameras to record conversations as well. See California v. Gibbons, 215 Cal. App. 3d 1204 (Cal Ct. App. 1989).
Emphasis mine.
It then follows with:
>If you are recording someone without their knowledge in a public or semi-public place like a street or restaurant, the person whom you're recording may or may not have "an objectively reasonable expectation that no one is listening in or overhearing the conversation," and the reasonableness of the expectation would depend on the particular factual circumstances. Therefore, you cannot necessarily assume that you are in the clear simply because you are in a public place.
Again, emphasis mine.
Your first link appears to confirm my claims and specifically contradicts what you claimed in your previous post.
Next.
I am not a lawyer, but my understanding is that recording phone calls without the other party's consent is outlawed explicitly by California law. So criteria like expectation of privacy don't come into play here.
And while innocent bystanders do get whacked from time-to-time, I find that more often than not when big companies change their TOS in ways that impact other companies, there's actually a pretty decent reason behind the changes.
Facebook feeds got extremely spammy around the Zynga days, they are much better now (tolerable, at least, though still far from perfect). Not sure what your google story is, but I've heard lots of startup folks complain about changes google has made to their algorithms or TOS policies that hurt them only to visit their sites or look at their apps and see that they were actually pretty spammy and really deserved the deranking.
Maybe switch fields - that's a good way to break out of a destructive pattern.
Building an app? OK, better hope it doesn't violate Apple's TOS. Building a consumer website? OK, better hope Google's next update doesn't affect you negatively. Building a Buffer competitor? OK, better hope Twitter doesn't ban you from their API
Here's a previous comment: https://news.ycombinator.com/item?id=9262245
That was about FaceBook, but exactly the same holds here.
Quoting:
Certainly I'd be incorporating an exit strategy,
and my game plan would be to leverage off FB to
start with, to grow as fast as possible, develop
my own community and eco-system, and then be able
to shed FB and replace it with something else.This is just another thought-terminating cliche not unlike "if it is free, you are a product" and I can't help but respond.
Large technological companies create ecosystems where smaller players can coexist with "hosts" in a mutually beneficial relationship. Ecosystem companies are not able to predict the future of the ecosystem, its evolution for all intents and purposes is mostly unpredictable. Sometimes the relationship works out, sometimes it doesn't.
Startups have 70-90% failure* rate, the possibility of the host cutting of some parts of the ecosystem to integrate their food chains into the corporate borg is factored in the business and investments strategies of the smaller companies.
You should not trust another company for your business model, but building your tech business in the ecosystem of a large fish is an absolutely valid and quite possibly lucrative strategy.
*definitions vary
EDIT success/failure
Don't you mean failure rate?
> You should not trust another company for your business model, but building your tech business in the ecosystem of a large fish is an absolutely valid and quite possibly lucrative strategy
Yes, as long as if you have a Plan B if/when your relationship is cut (a valid Plan B is to close shop and try to do something else with the expertise gained)
Sure, thx
>Plan B
Certainly. Datasift has 25 various information sources. Twitter is really important, but doesn't seem to be critical for their platform.
I've worked for small companies in the past that had dozens of partners, but most partnerships don't really do anything except give you brand logo to mutually share on each other's web pages.
It's like anything else, partners provide revenue to you following a power-law distribution, with a long-tail that provides nothing and a very select few that make up the lion's share.
I agree entirely that there is value, sometimes significant, in exploiting the ecosystem created by a larger fish. You'll see that in my comment, wherein it says:
... leverage off XXXX to
start with ... be able
to shed [it] and replace
it with something else.
Consider, though. If you become successful - which is surely your aim - then you will come to their attention, and they will do whatever they like.
So I repeat: Don't trust them. Use them, yes, but don't trust them.
I agree, however, that blind trust is always a bad path.
Obviously, I'm going to trust an electricity company to provide my business with electricity. I'm even going to trust laptop manufacturers to continue to provide laptops I can code on.
In both those cases, the thing I'm depending on is a commodity, where there is also either strong competition or regulation. There isn't much risk in depending on them.
The stage to be wary of is when a whole industry is still at product stage and people are trying to move them into being platforms, or using them to Innovate/Leverage/Commoditise (ILC, see Simon Wardley) to play ecosystem games.
i.e. Where the business models and how the industry will be vertically and horizontally sliced are not yet settled.
Those are the ones you should be wary of dependence on.
Not all that long ago I did tech dd on a company that had exactly one very large company as their main unwilling partner. They claimed that they were 'too big to fail', guess how that ended?
Startups are risky, startups that are growing in the ecosystem of one large megacorp are even riskier but they are prime acquisition targets. You decrease success chance for higher yields. This is like business 101.
The advice "do not base your business model on another company" evaluates to "do not increase your risks" which would be an absolutely absurd advice.
If one qualifies the original statement with several conditions for different contexts, it may become a solid advice. As it stands this model of understanding business strategies is useless.
But basing your business model on being allowed access to other companies' data fails far more frequently than that it succeeds, and it fails far more frequently than the already pretty bad chances that you have as a start-up to begin with.
See the thread subject that you're commenting in for how Gnip's competitor fared.
Let me try to explain it a bit better: if you use documented interfaces and have an established supplier role with a larger company that's fine. But if you are taking data from some API and you're re-purposing that data in a way that the company supplying you with that data may not agree with then you may one day find your access terminated.
You probably should not include RedHat in a list like that.
Never, ever trust another company
for your business model.
What's wrong is the lack
of contract or agreement.
Is that the case?
Most people would sooner die than
think; in fact, they do so.
-- Bertrand Russell
If someone chooses to terminate their thoughts based on something someone says, without considering why they might be saying it, and why there may be value to be extracted from it, then maybe the problem is not with the expression, but with them.
Think of it this way. You may consider it a given that there are times and places where you can and should trust other businesses. This statement about never, ever trusting a another business therefore seems to be at odds with your experience. Do you ignore it, or do you use it as an opportunity to learn why someone might say it?
I honestly don't understand the point you're trying to make, so let me parody your stance and you can explain where I'm wrong.
* I create and provide a service
* Said service has an interface
* You spot a service you can provide based on my interface
* You build a business using that interface
* You are successful
* I notice you're successful and build a clone
* I launch my clone and make the interface unavailable to you
* Lots of your customers switch to using my replacement
* You go out of business.
Is that all OK in your opinion? If not, what's wrong?
But of course, MBAs are for suckers who know nothing and should just cease to exist. At least that's what I keep hearing on HN.
Never, ever *trust* a company: always have
formal contracts. If you can't get formal
contracts, don't work with them or rely on
them.
Never, ever trust another company
for your business model.
Am I missing something?
It really does seem to me that we have pretty much exactly the same viewpoint - there is a risk analysis to be done, and risk mitigation is critical. My risk analysis says that if someone else is in a position to screw you over completely, don't proceed on blind faith. Instead, have contracts or agreements in place. In other words:
Don't trust them.
Certainly I'd be incorporating an exit strategy,
and my game plan would be to leverage off FB to
start with, to grow as fast as possible, develop
my own community and eco-system, and then be able
to shed FB and replace it with something else.
You can fly under the radar for a while but if Twitter/Facebook/Google/etc decide your area is lucrative, they will go after it, whether that will kill one of their "ecosystem players" or not.
There is no partnership involved, they provide a service as a huge "blob" that is making them money and they don't care about 1 out of 1 million of their regular users. Your small company cannot call itself a partner, you've as much leverage as any user when it comes down to deciding to sacrifice you in the name of something else more lucrative.
I'd say explore the benefits to bootstrap but have a strategy to depend less and less on them or your time will come. And if you really want to tie your business survival to one of these ecosystems, get yours and theirs responsibility down in writing (and even that won't help much because a breach of contract for them will mean nothing in terms of dollars, while for you it'll be your existence).
Can't really argue with that. Maybe not useful advice, but it's self-evidently true.
If you build a great business based on something over
which you have no control, and which is such that the
ones who do have control actually may have an interest
in cloning what you do and terminating your facility
without warning, be sure to have a "Plan B".
Honestly, would you build a business based on a Twitter or Facebook facility, relying entirely on them not arbitrarily terminating your access?
What is your advice?
As such encourage and build openness by default into your products, as for sure users want that (how many people for example wanted only to email other AOL users, or view only other AOL websites?
https://en.wikipedia.org/wiki/Stac_Electronics
In the MS DOS days they were looking at building on-the-fly compression chips for hard drives. But it turned out that it could be done nearly as well in software, so they made a very popular product that basically doubled the space available on every hard drive.
Microsoft thought that was pretty swell, so they talked with Stac about licensing their technology, getting as far as inspecting their source code. But they decided to just build it themselves. Stac eventually won a lawsuit on patent claims, but it was neither easy nor cheap.
It makes sense from an MBA perspective: you can definitely boost your short-term numbers by fucking over your previous partners. You get a proven market and no significant competition. But what never shows up on the spreadsheets is the extent to which you've reduced long-term ecosystem innovation because savvier players learn you can't be trusted.
The example is obvious: Gnip vs Datasift.
It's a function of maturity (young startups have growing pains on what they want to be and how they monetize) which can definitely create collateral damage. Mature companies like Apple don't experience the same degree of angsnt.
It's no accident that Twitter, which has generally underperformed and disappointed more than its peers, is still screwing up and over its ecosystem far more than others.
Now, a few years later our largest competitor has been doing quite well with a product along the same style... but now, without warning, they've been cut off from API access and are facing requests for hundreds of refunds. Glad it wasn't us.
https://blog.scraperwiki.com/2014/08/the-story-of-getting-tw...
This should be a case study for anybody looking to get into a BD role.
A partnership is fundamentally coming from a position of weakness, and the relationship is a way for both partners to partially fill in whatever gap they believe they have in their strategy.
Sometimes partnerships go south, especially when the relationship is highly asymmetric.
In this case Twitter was the stronger of the two partners and they decided not to renew. That sucks, but that's life. Don't build your entire business around a single partnership, because that signals to your partner that all they have to do to get rid of you is replicate what you do and cut you off. Once you've gone through the blood sweat and tears to prove the market, you've just given your larger partner free market development.
DataSift's website lists a number of data sources [1], but Twitter comes up first. More importantly, I suspect these sources are slightly massaged to look like they have more sources coming in than are really useful for their business model. e.g. is Wikipedia really in the same space as Twitter? They also have a bunch of minor sources but they're listed like they all rank equally. This is fine, most businesses would do the same to make themselves look better -- it's just advertising.
Another part of business partnerships is that you have to expose your weakness to your partner. Twitter is going to know at some point that DataSift's business model is heavily built around their firehose. They might even have some sense about what volume of business all the rest of those sources makes up to DataSift's business. Twitter also, through their other partnerships, has a better sense of the size of the Twitter Analytics market. Even if DataSift's piece is 15% of the overall market, Twitter has other partners it can use to get a feel for the other 85%.
Partnerships are dangerous. Know what you're getting into before you get into them, and try to only do them with similar-sized partners or you'll find yourself getting screwed over/screwing somebody else over.
It's arguably not really a partnership if the relationship is highly asymmetric.
I understand the way big ecosystems and walled gardens work. I also understand that every large technology company that has come down the pike in the last 30 years has wanted to conquer the known universe. That does not mean I have to actively assist them in this activity.
Can't we all just "conquer the known universe" together? We need a framework for educating these companies that open solutions have greater reach than closed solutions even if they are less directly profitable. The people making decisions are often successful enough this should be possible.
I have no idea if DataSift and NTT were good partners, but not having anyone else access the Firehose suggests they think they can cover all significant use-cases. This almost always turns out to be wrong.
It also sends another chilling signal to say "we're a platform, but when we feel we want to roll over part of the ecosystem we'll do it".
It's only 500M items a day. At that level you've got plenty of options as far as hardware and software. (You could even record it into a normalass SQL DB to log each one if you wanted.) You may need some useful buffering for high spikes, perhaps some message queues with lots of RAM.
Twitter can handle this for people (like DataSift, providing historical lookback), or they can provide template to help them spin up. It's not gonna cost thousands of dollars a month in hardware.
When the big company you relies on buys your competitor, you should be very, very scared.
Then, by cutting off others' access to the platform, the platform owner can make their own service a monopoly and capture a significant proportion (if not all) of the others' customers and revenues.
Phase 3: Profit!
And getting 80% of the revenue for that (according to the article).
How does this benefit Twitter? How does this benefit startups wanting to use Twitter? The data is still available, just not through resellers.
So DataSift wasn't just reselling their data - they, along with Gnip, had a partnership that allowed them to get access to the firehose. For many social media companies, the firehose is a necessity, so they bought contracts with Gnip or DataSift.
Then Twitter bought Gnip, and is cutting off DataSift's access to the firehose.
DataSift didn't have much of a value add (OK, historical data, common filters for Twitter and Reddit and others, etc.) But nothing that Twitter can't just in-house now that the market got nice and established.
So, Twitter terminates their "partnership" with DataSift, and all of DataSift's customers (i.e. the market that DataSift spent money and effort getting "nice and established", educating potential customers, helping them figure out how to make use of the data, etc.) who want to continue benefitting from those services will now migrate to Twitter/Gnip.
I bet Twitter/Gnip will come out with some tool to make it ultra-easy for DataSift's customers to migrate across.
It's like MVP by proxy.
Have read of multiple other such cases where the rug was pulled from under such startups that relied on a single point of failure (SPOT) - http://en.wikipedia.org/wiki/Single_point_of_failure
I told that founder the reason why I thought his model was not a good idea, after thinking for a while and then declining the offer.
He should have totally seen it coming and been prepared. When you don't think about the future and don't have the foresight to build competencies to take advantage of what you discover, you end up being blindsided. This is not about hating on Twitter or siding with them, it's pure competition.
I researched this last summer. Even before I was explicitly told all the original firehose contracts were five years long, I could infer it from attitudes and behaviours.
I'm excited that Datasift are launching interesting new Facebook features. They're a great company, with good technology.
I hope Twitter's "no data resale" attitude leads to them becoming irrelevant, as it is so destructive of much innovation.
They've not totally converged in terms of audience, but I think have surprising overlaps. Lots of Twitter data use cases you can get from Instagram, I bet.
So as far as data access goes, Twitter and FB/Instagram are pretty much on the same page. Except Twitter's docs are way more accurate and nicer, probably because it makes money for them and isn't a "throw it out there and see what happens" kinda thing.
I don't see how that's true at all. While Twitter now has well-developed image capabilities it's still not the core of what people use it for. Also Instagram has no equivalent of retweets.
Are you serious? As in, do you honestly believe that end users are gonna say "what, Twitter won't let people have access to 'our data'? Screw that I'm gonna go to ... <some competitor I've not heard of>?"
Twitter gained a lot early on from having all sorts of innovative things happen on top of their APIs. They clearly believe that's irrelevant now, as they have energetically screwed over all sorts of API users. I think the hope is that users will gradually say, "Oh, I love cool tool X for competitor Y; I'll switch."
I don't think it will happen next week, but it's a plausible path to long-term failure. Missing important waves of innovation is how a lot of tech companies die.
Twitters got how many thousands of employees? Surely if they organize well they can innovate inside. Or just keep an ear to the ground and hear about neat-sounding things and clone them. Or buy them if they have to.
What value add did DataSift bring? They were taking 80% of the revenue. If those DataSift customers were really using the Twitter firehose, there's no real replacement. That means Twitter just upped revenues on those customers by 400%.
Historically, though, platforms die all the time. AOL and Compuserve lost out because they couldn't innovate at anything like the speed of the open web.
I don't think they need third parties immediately. They needed them in the past, though; they were important to Twitter's early success. They'll need them again at some point, too.
The notion that companies can innovate because a) they want to innovate, and b) they have lots of people is... unproven. How many years has Yahoo been floundering now? What's the last innovative thing Microsoft did? Does IBM even exist anymore? Lots of large companies open innovation centers and intrapreneurship programs. Small companies still regularly kill large ones. That's because large companies are mainly optimized for turning the crank on their current money machine. Things that interfere with that rarely survive, including most innovation efforts.
We have been working with DataSift for about a year now though and it is pretty damn clunky. I'd say GNIP is probably even worse at the moment but maybe not for long?
As a user, if GNIP can capitalize on being part of Twitter, I gladly welcome the improvements this could bring. I'm not super confident though as now Twitter/GNIP will also be a monopoly in the space with the usual disincentives that come along with that. Also, our use case is strongly biased towards the "historic query" features and neither service seems as interested in that as they are in the real time streaming features.
Also, I had considered earlier developing a niche competitor solely focused on the historics space for companies with research-oriented use cases like ours. I suspected Twitter would be dropping this hammer though so glad I didn't waste my time!
Though I suspect companies not in the jurisdiction of the US or its "partners" will probably be more likely to get away with an approach like this if caught or public about it, while still providing value added services based on walled gardens like facebook and twitter.
Apple did this with their introduction of Gamecenter, killing OpenFeint, and a couple others (can't remember their utility).
DataSift was proud of the money they raised. The said it made them "more difficult to acquire than Gnip". I was amazed when I heard that. It sounds like they are happy they took on so much money and were moved passed by Gnip.
But it is worse, I think. For many services they require you to provide API keys. Rate limits are a problem? No! Create more API keys and give them us. Why are they not blocked? If any big service decides to block API key rotation ... game over? If we do it in-house, then we can use many proxies and pretend to be separate. Maybe.
Or maybe not. Maybe because of VC connection DataSift can rotate API keys and with impunity. Who knows?
But yes, to me it is very very risky business, but not too difficult technology. (Maybe that is why I have not had a big hit. Too little risk.)
So does HN think it is successful to raise money like them? Do the founders make more than $500K/year total from this venture?
At it's core, what twitter provides is no big deal. The ability to post 140 chars. They could have been quietly funding an open twitter protocol (it will happen eventually) but they chose to keep their heads in the sand.
This is a good lesson in business.
https://qbix.com/blog/index.php/2013/04/a-new-kind-of-platfo...
One where you won't be cut off if you're a developer. And one which organizations can host themselves.
Why did Wordpress succeed for decentralized blogs? Why did Bitcoin succeed for decentralized currency? Many organizations run it.
Are you really saying you want just a few monopolistic companies to control people's social experiences online?