Hacker News Reader

Comment by anigbrowl | Hacker News Reader

- announced at 10pm - basically a week to decide what to do with your dev project - No information to help you make that decision

This seems like more management by manchild. I strongly suspect that Musk realized his promises of cutting down bots etc. were failing and that they either couldn't or wouldn't develop abuse detection, so he's decided to solve the problem by making it cost money, with the (perhaps desired) side effect of crippling academic/analytic research.

As an amateur network science researcher, I'm pretty steamed. I enjoy doing my own network analyses using tools like Gephi and have monitored probably a hundred breaking news or trending issues, as well as amassing a great collection of academic papers by smarter folk than me. I don't run any kind of app or service that sits on top of Twitter, but those who do run legitimate services are now being held hostage because of the unchecked abuse. Ant the failure to deal with botspam is lamentable. Twitter has refused for years to implement even the simplest things like hashing tweets and looking for collisions, or considering the count of edges to the hashtag graph, or even looking at tweet frequency. Many of their abuse problems will continue unabated; the sellers in the market for bogus twitter accounts are likely delighted because they now have a great excuse to raise prices, even though most of their 'product' is produced by hand with cheap labor in poor countries.

View on HN

> basically a week to decide what to do with your dev project

Even less for some. It seems further apps are being shut down in the run up because they have "violated Twitter Rules and policies”.

Such as movetodon [0] - locked out today - a service which helps you connect to the fediverse accounts of twitter users you follow. And reports that debirdify is closed out too.

[0] https://mastodon.social/@Tibor/109800904950500383

All things considered, that’s not surprising that a company doesn’t want their free API/resources used to make moving to a competitor easier.

Very true, they don’t have to give access to their API for free. My question is, why don’t devs use Puppeteer to spin up a Chromium instance and access Twitter’s data that way?

Twitter could add anti-bot protection like Cloudflare’s CAPTCHA. I’ve seen that make a site unusable with Puppeteer. By “unusable” I mean that correctly solving the CAPTCHA either just gives you another CAPTCHA, or gives the “click if you are human” thing, and clicking that just goes to a CAPTCHA or another “click if you are human”.

It didn’t even require doing anything to/on the page with Puppeteer. Merely using it to open a browser window and then waiting, and using that browser window by hand during the wait to go visit the site that was using Cloudflare anti-bot protection ran into the problem.

IIRC, puppeteer/webdriver things generally work by injecting JS through extension, and their presence can be detected by looking for those objects/functions from JS.

Isn’t there a stealth plugin that can hide the injected JavaScript?

I actually face the same issue on Firefox with some websites except I’m not using Puppeteer. I thought it was a glitch with Cloudflare but guess not.

That’s part of the reason that “it’s to stop the bots” is BS. Academia and other similar uses will pay or just stop. Bots and other malicious uses will find work-arounds and just change where the game of whack-a-bot is being played.

I think anyone can request a download of archive of all their data, and removing that would violate at least GDPR. And then do whatever they want with this archive, including uploading to anywhere else. So while it disrupts the current process, it cannot really take the competitors out.

It's still anticompetitive. Can't wait to see Twitter get destroyed in court for this.

Oof. I did a last pass migration last night, and noticed that movetodon was broken then, but I assumed it was due to load, rather than... this.

End of an era; Twitter is dead. Whatever is left, it's not Twitter.

> End of an era; Twitter is dead. Whatever is left, it's not Twitter.

Am-I the only one surprised the site still hasn't had a major (multi-hour) outage at this point?

Considering the waves of resignations that hit the automated content filtering and bot mitigation teams, it's no wonder they are getting desperate.

Many people with large Twitter following who use Twitter as the commenting system for their website (think John Gruber) have complained that Twitter mentions ahve been reduced to nothing. It's a silent failure that no one at Twitter has mentionned or acknowledged.

> Am-I the only one surprised the site still hasn't had a major (multi-hour) outage at this point?

Twitter has had at least one multi-hour global outage since Musk took over. This is in addition to at least one outage in Australia, plus outages of other critical services. The API outage happened before the after-the-fact announcement, login and 2FA outages kept people out of the app, etc.

For what it's worth, I cannot access Twitter right now, and DownDetector suggests I'm not alone, so we might be having another outage starting just now.

Twitter still has like 1000 heavily dedicated (either because they love elon or getting fired means getting deported) which means they can fight a lot of fires and keep things going.

  > getting fired means getting deported

i wonder what percent of workers left are in that position?

Wait, didn’t it? I recall a prolonged period when no-one could log in (and a few prolonged periods where no-one using 2fa could log in)

I would honestly love to see a poll of Mastodon users just to see if they understand that administrators can read everything in an instance, even direct messages between users.

It's going to take a few scandals for this to sink in, but it's such a recipe for disaster.

On Twitter, you at least know that it's only three letter agencies reading your DMs. On Mastodon, it could be some Reddit moderator type that makes it their life mission to destroy you.

How is that any different from threaded news groups, forums, and reddit threads?

And because this must be true for every admin on every server on the internet... thank you dang (when you read this)!

If you're targeted or sharing sensitive info, it's best to use encryption.

API reads/writes don’t generate ad impressions. It’s really that simple. All the bot nonsense is rhetoric.

API calls just grabbing data may not generate impressions but people follow bots that utilize this API and (some) bot-generated content generates interactions: likes, shares, comments.

Twitter may be heading closer towards becoming a ghost town.

Yep. Tearing down the ecosystem to support the product.

Whether it’s Elon or someone else, this shows a total lack of understanding of what made Twitter ubiquitous.

What ecosystem? I might be dim or out of touch but I’ve never seen anything related to Twitter outside of the app itself.

It's not an app it's a website.

What's with these cell phone people calling everything APPS? Hi. They're WEBSITES. Just because you might use a stupid cell phone app to access the stupid website doesn't mean it's no longer a website.

Use the right words and stop recording video vertically. I hate all of you cell phone people.

> Use the right words and stop recording video vertically. I hate all of you cell phone people.

Wouldn't the right words be "smartphone people?"

No.

Notes

Also named cellular phone, cell phone, cellphone, handphone, hand phone or pocket phone, sometimes shortened to simply mobile, cell, or just phone.

Sorry, imho it’s best consumed via the mobile app. Indeed all these “web-apps” should just be taken out of the browser and leave it to hypermedia experience. They can be their own native app just as fine

Are you nuts? Yeah I want 10,000 icons on my desktop rather than just one browser.

Man this generation blows. No wonder you're fine with 10 different game publisher clients, 10 different movie publisher clients, and 500 different "apps" to access websites. Keep recording vertically dude. (GEEZUS)

I created a bunch of lists and basically just follow those, rarely engaging. In a way, I am thankful to Elon for helping me shed my compulsive Twitter usage. He unwittingly performed a public service.

I think they do. One person writes to the API to post content two other people will see + view ads against.

Though a lot of the API usage is for content scraping, analytics and surveillance ("we are monitoring the internet for you")

You're sounding very confident, so I'm guessing you have a source for that claim?

Besides, even if 90% of the API usage is for scraping and does not impact the bottomline, it's still a smart business decision to leave the API open if the remaining 10% are for posting content that brings in views and thus ad impressions that generate more money than the API costs to run.

It's the main usage described in the "Twitter Files" (internal docs of Twitter which were openly leaked on Twitter).

That shouldn't be possible, API request limits for 3rd party keys were greatly reduced in 2018 to less than what is available for 1st party clients. Extracted 1st party keys or Web scraping would be used for that purpose.

API writes absolutely generate ad impressions.

Your problem could be solved by providing read-only API for free - you don't need to write access to do academic research (unless you want to influence the discourse).

Providing a read-only API still costs Twitter money though. He's trying to make it profitable. Seems obvious that stuff like this would be cut.

> He's trying to make it profitable.

Probably shouldn't have landed it with $1bn/year debt payments then, eh.

This is like PE takeovers that sink companies that were previously treading water. It took on a new, ongoing expense that it couldn't afford. The layoffs, the auction, not paying leases, and this, all seem to have some cost cutting angle to them. I don't think it's necessarily because Musk is cheap or sees lots of fat to cut; I think he's desperate to lose less of his original investment.

I still don't understand why this behavior -- leveraged buyouts -- should not be illegal. Congresspeople are tools, I guess.

> Probably shouldn't have landed it with $1bn/year debt payments then, eh.

It's worth remembering the market setting was very different when he made the offer. It's also worth remembering the offer was refused at the time it was made. It was only after the tech market saw a downturn, turning a good offer into an incredible one, was the offer pursued. Around that time, it also looks like he attempted to renegotiate the offer, but that was also refused.

Poor timing of the offer (only knowable in retrospect!) and shareholder obligations more or less forced this situation into existence. Musk made a good offer with a terrible contract; time, shareholders, the board, the legal system, and the stock market turned that terrible contract into a huge payout for shareholders and a signed a death warrant for Twitter.

Musk made a mistake in the way he made the offer for Twitter--but let's not pretend that the legal system isn't equally responsible for forcing the current situation. Even if the previous leadership were open to negotiating a lower buyout (and thus lower debt burden), they would have just opened themselves up to endless shareholder lawsuits had they done so. Their hands were effectively tied once the rest of the tech stock market started going down.

Not that obvious, if I was in the same position I wouldn't cut API access, if I was thinking about profits. Maybe make it heavier cached ("slower" time before new tweets are available, less infrastructure costs) but wouldn't disable it fully. My thinking would be that if there is less API access, there is less people accessing tweets, one way or another, and less links back to Twitter.

If I woke up feeling evil (really money seeking) I might add some clause to the terms and conditions that you need to prominently display "Powered by Twitter" if you use the API and aggressively check all websites using Twitter API, one way or another.

But I guess my perspective would be considered more long term while Musks management of Twitter seems to be heavily focused on short term.

>My thinking would be that if there is less API access, there is less people accessing tweets, one way or another,

the real problem from a business standpoint is less people trying stuff out and figuring hey maybe I could actually build a business on this, time to move up to the paid tier.

Also no devs at companies have played with twitter api because devs not going to spend money to play, so never say in meeting actually we can solve this with Twitter's api, we just have pay for a licence and I can write the solution! So maybe business look for other ways to solve problems.

> Maybe make it heavier cached ("slower" time before new tweets are available, less infrastructure costs) but wouldn't disable it fully.

This. I had a brief flirtation with working with people using the Twitter API for research purposes. A very large majority of them didn't care about having the latest tweets coming off the firehose, but instead getting access to all the old data. At least at the time, the latter was much more difficult to do.

Things like using Twitter for real time sentiment analysis does require realtime data, but those applications are more likely to be for-profit, and thus could afford to pay.

Delivering old tweets is actually more expensive than real-time because nothing is cached. Streaming access is pretty much all served out of memory.

I used to lead the search and historical API team at Twitter.

Right, the 30 day search API is still 100 request/mo for the sandbox and 500/mo for the bottom paid tier (which I think is $150/mo). Would love to hear more from you about this experience, whether in blog form or commentary. I use the filtered stream API a lot and the search API a bit.

Oh I’m sure. My point was just that the use cases least likely to be able to afford paying are clustered on historical data. And those clustered on real-time data tend to involve $$ anyways

We gave academic researchers free access to the full archive search API to help cover that use case.

> He's trying to make it profitable.

Might have been smart to not make it unprofitable in the first place.

I think Twitter was never profitable to start with.

It was profitable from Q4 '17 on, with the exception of Q1-Q2 '20.

https://www.statista.com/statistics/299119/twitter-net-incom...

Twitter can be profitable. It made over billion dollars in both 2018 and 2019 on revenues of $3 billion. 2020 was big loss but can blame the pandemic. 2021 was small loss but would have been profitable except for lawsuit settlement.

Without Musk, Twitter would likely have been profitable in 2022. No extra debt, no fleeing advertisers, no massive cuts. Some layoffs like everyone else is doing would probably have been enough.

> Twitter would likely have been profitable in 2022.

Please back this up with an actual projection that says anything close to this. Twitter was not on track to ever be profitable before Musk bought it.

2020 was great for most online product revenue. Why is Twitter different?

Twitter was selling ads to companies slashing their ad budgets, not products to consumers with a lot more screen time and a lot less access to stores or to businesses newly dependent on the internet to coordinate their newly remote workforce.

Twitter was profitable in 2018 and 2019, took a deep dive in 2020, climbed way back up in 2021 though not quite to profitability.

However pretty much the entirety of their revenue stream was ads, and Musk started taking a big dump on that early 2022.

If Twitter pulled the layoffs Musk did, without having taken on his debt, it would have posted profits that quarter.

Doubtful. Ad revenue is wayyyyyy down due to the impersonation/brand safety crisis Elon triggered with his erratic product moves. Letting any jokester with $8 impersonate global brands erased hundreds of millions in revenue per quarter from Twitter's bottom line. The debt situation is just an additional financial mismanagement cherry on top.

Perhaps I wasn't clear, but this was before Elon bought it. Twitter easily could have posted profits that year if they did the firings he had ultimately done. I thought that temporality made it clear we are talking in a hypothetical but I guess I wasn't clear.

yeah wow, imagine a company providing a free product in exchange for the externalities it provides like more traffic and marketing, unheard of!!

Cutting things that bring people in is a false economy though. It's like a restaurant owner deciding the wait staff should have to pay to work there.

Having free users also costs money. Same argument can be applied. And best of luck to Twitter with that approach.

Is this the guy who promised to "thermonuclear name and shame" the advertisers who no longer want to do business with him? Oh, it is!

https://twitter.com/elonmusk/status/1588676939463946241

Agreed, but I don't think scam and spam bots are using the API anyway - rather puppeteering tools built on Selenium or any of many scraping libraries and toolchains.

Even if they were, who's more likely to pay for an API, the scammer making shit ton of money from tricking twitter users, or the hobby dev making a cool integration with Twitter that brings people to Twitter?

Twitter API had traditionally been configurable between read-only, read-write, read-write + DM. Most apps request either of the latter two, but you always could.

Does this change affect all access? Like, I have half a dozen bots that only write (post rss items).

At first I assumed this was only impacting read access (apps that mimic the twitter app), but now I’m not so sure.

There's no write-only API keys, all keys come with read permission. If you didn't encounter discussion of any of "Key", "Secret", "Token", "CK/CS" while configuring that, and are thinking of apt/yum/brew installable terminal packages and IFTTT and likes, that package or the IFTTT backend are the apps and the maintainers will have to do something about it.

My bots are set up through dev. I wasn't saying my bots don't have read/write access. Just saying that my bots _only_ actually write, and I was hoping I could perhaps still use them for that purpose without having to pay.

I believe things like @PossumEveryHour and some of the automated traffic/weather systems have announced they'll be shutting down.

Yeah, I'm seeing mass announcements of shutting down. I think some are holding out small hope that something will change in the next few days. I just checked and I have six active bots.

@isjesusreturned @badassnames @pinballmapcom etc.

Bummer.

I'll especially miss the Earthquake bots :(

Musks promises of cutting down on bots only mattered to him and to the degree he made a big deal about it. He could have declared victory whenever and been done with it. Who would have argued.

I think people would have argued with him. There are people that fawn on him one day and then complain that 'the algorithm has shadowbanned them again' the next, and he spends a surprising amount of time engaging with them. It's like some weird parody of an abusive relationship, with the world's erstwhile richest man and uber-CEO earnestly and repeatedly appeasing influencers like 'Catturd.'

To an extent I think this is political theater, as opposed to ineptitude, petulance, or poor financial decision-making. Twitter is an important piece of virtual territory because it's still the platform for breaking news and pretty much every media outlet publishes and promotes stories there, as well as most politicians, govt agencies and so on. Like it or not it is the most 'realtime' mass market platform.

That gives Twitter tremendous strategic importance, even though its social graph is smaller and more ephemeral than Facebook's. Size is not everything; reach matters too. That strategic importance makes it interesting to state and non-state actors, because capture of that territory is loosely equivalent to control of radio broadcasting capacity or printing presses in earlier eras. If you think of Twitter as a magically effective print and distribution store for pamphleteers with a huge secondary ecosystem around it, the shutdown of free API access is essentially a market takeover. Musk was spitballing yesterday and suggesting that from now on API access would cost $100/month and require ID to use.

(It's interesting to me that Google could have owned that space with their News product, although it has always been a one-way thing rather than socially driven. Their failure to own that space is a direct result of their aversion to user configurability and selection.)

You are correct people on the internet would have argued with them. I should have been clearer. His employees at Twitter, the bankers, the minority shareholders of Twitter and the advertisers would have all accepted his claim that "bots have been defeated enough".

I don't know that they would have, but regardless it seems like Elon really cares what the people online say. He's going out of his way to engaging in memes from his "fans".

I would have because all his posts are still full of bot spam and crypto scams... But, he certainly has no shortage of bots that upvote and call him king.

> - announced at 10pm - basically a week to decide what to do with your dev project

I believe it's an intentional and strategic move to have every API client to start paying right away.

"Free API is being abused badly right now by bot scammers & opinion manipulators. There’s no verification process or cost, so easy to spin up 100k bots to do bad things."

https://twitter.com/elonmusk/status/1621259936524300289

This is mostly untrue. While it’s possible to stay anonymous you need a phone number, and it can’t be from twilio or other services. It’s been this way for years. I strongly suspect the bots that are unverified use headless browsers. The problem won’t go away.

Agreed, while there are tons of spambots, they are likely scraping Web API and using some backdoor-ish thing to get around requirements than using the documented API.

This announcement and today's ban-wave that followed seemed like someone not accustomed to Twitter just knew there are bots, and thought it to be workable idea to sample global timeline and script ban duplicates, which of course is not workable at all and only maximize harm on real users.

It is trivial to buy 1000's of phone numbers with SMS all available through APIs. You don't even need to keep them. The grey web is filled with many of these services.

If I looked hard enough I could probably buy 10000 verified twitter accounts with API access right now, aged 1 year or older.

Yup. PVAs can be had for a couple of $, with the price going up if they're 'warmed' (with a history of innocuous activity, usually liking celebrity tweets, retweeting the occasional breaking news story or heartwarming puppy video etc).

Where can you buy actual mobile phone numbers? (Not VoIP numbers)

I would think mobile carriers keep that in their own house

I think I've encountered some nefarious discussions around obtaining SS7 bulk access endpoint and intercepting SMS?

Furthermore, every time I create an API service, I get immediately banned (before I do anything) over and over until I truly prove that my bot is in good faith. They are super aggressive, and they actively enforce their rules (in the past, I've tried to make bots that reply to folks - that don't follow me - and the bot lasts like four hours).

Twitter hasn't my phone number, just checked. Maybe is it because my account is old?

Yes. But even with an old account, if you sign that up freshly to the developer program they'll force you to provide a phone number, and can require you to go through the approval process for higher API access even if you don't want that.

My Twitter account also has no phone number, but it gives me an error about it if I try to sign up for API access.

Scammers and opinion manipulators aren't using the Twitter API developer program. They're just using the native apps' API keys and posing as real people.

Classic sock-puppet usage is done through the web UI.

Is this kind of thing still possible with Buffer? I don't keep current with twitter manipulation tech. And do Buffer, RoundTeam and Hootsuite use the API directly?

How A Twitter Fight Over Bernie Sanders Revealed A Network Of Fake Accounts

https://www.huffpost.com/entry/democratic-bot-network-sally-...

https://shareblueastroturf.netlify.app/

Buffer, Hootsuite, etc. use the API. The app/integration something was published via used to show up below tweets; Musk removed it. https://www.theverge.com/2022/11/15/23460186/elon-musk-twitt...

Why are people downvoting jmeister who is just quoting Musk. Don’t shoot the messenger.

Care to link some resources for amateur network science? Search keywords? What tools do you use to monitor the spread of news?

https://news.ycombinator.com/item?id=34321135

Lots of stuff in this thread from myself and multiple other people.

> This seems like more management by manchild. I strongly suspect that Musk realized his promises of cutting down bots etc. were failing

He probably started to realize that other people began to realize just how many Tesla bots there are.

Time to use selenium

> This seems like more management by manchild.

You have good points and can probably make them without resorting to insults.

I can, but made a choice to be opinionated here. The CEO of Twitter leverages clownishness for effect, titling himself (at present) as 'Mr Tweet' and uninhibitedly trolling others, often in crude or aggressive terms. While I could take a Ghandian approach of not reciprocating such negative behavior and instead maintain a diplomatic/academic air of prim neutrality, I think Musk's behavior is strategic in that he asserts social permission to 'act out' as the natural consequence of his wealth and power, whereas his critics are obliged to water down their opinions or suffer reputational attack.

While I certainly don't want to import the combative and shallow standards of Twitter 'debate' to HN, I do think we have a right tp be opinionated, and that my the term 'manchild' labels a specific sort of behavior rather than some petty or crude dig at his appearance or personal life.

Yeah, I think use of that term is warranted by the facts. No need to sugar-coat it.

Indeed, I'd wish that the HN community was more hostile to Musk's bulldozer approach, seeing what it did to both the company, and the engineers. The whole affair has been needlessly destructive -- obnoxiously so -- and his getting away with it harms the profession.

I think the person you’re responding to was trying to help you be more persuasive. You can ignore his advice, but it may mean that a lot of people just ignore you or think you’ve got nothing to say.

> leverages clownishness for effect, titling himself (at present) as 'Mr Tweet'.

OK. I thought that was funny personally.

> While I could take a Ghandian approach of not reciprocating such negative behavior

I didn't think it was comparable to the British treatment of India under the raj, but OK.

IMHO, the name isn’t necessary. Getting names called and deeds attributed is part of the plan, so take steps to not follow the plan if you aren’t trying to be supportive.

Pay.

But there is no need for Twitter to allow you to do that for free just because.

From a pure business perspective it's either Twitter dies or Elon pays the bills.

From a business perspective, you don't want to alienate developers who work to build your ecosystem.

A more sane way to do this would have been with a public tweet explaining why they were doing it (ad revenue), a 3 month lead time, and an API for helping apps get users to "upgrade" their account.

> From a business perspective, you don't want to alienate developers who work to build your ecosystem.

Musk is trying to monetize every last aspect of current Twitter possible while cutting costs to stave off bankruptcy that his acquisition made imminent long enough to pivot Twitter into a completely different business model centering on being a payment intermediary, apparently to relitigate his firing twice from X.com before it became PayPal.

He doesn't care about the ecosystem, because its an ecosystem for a completely different busoness than the one he wants Twitter to be. He cares that use of Twitter outside of the 1st party frontends bypasses monetization, so he wants it monetized or gone.

This is true, but I suspect he greatly underestimates how useful the ecosystem is to actually making Twitter relevant enough for a critical mass of people to want to use the first party frontends.

Which is odd, because Elon is someone that definitely will remember AOL, MSN messenger and MySpace being things...

> From a business perspective, you don't want to alienate developers who work to build your ecosystem.

This does not follow, in Twitter's case. If Twitter's API made them money, they wouldn't have shut it down, but it's the exact opposite, it costs them money while providing no monetary benefit. So in this case, from the business perspective, it is correct to start charging for it.

Now they could spend 3 months but with debt service payments of a billion dollars a year, I highly doubt that they have the patience to wait that long.

It provides no direct income.

But it provide monetary benefits by attracting and keeping users, which encourages ad spend.

Or at least it used to, until the big advertisers were driven away.

If someone is trying to run a business and doesn't understand how indirect income works they might want to consider something less challenging.

What this actually does is remove API access from small startups, solo developers, and other innovators, and reserves it for corporations.

The number of users kept via the API is not as high as people might think, so I would actually agree that if that's the case, then Musk correctly shut off the API for people who weren't bringing in more benefits than the costs they were incurring.

Frankly the guy designed rockets that landed for the first time ever so I think he probably has the brain capacity to make this calculation.

He might not be great at decisions where people and politics are involved, but this isn’t one of those decisions. But some people here on HN really want to believe he’s an idiot, which he’s not.

Can we please stop saying that the CEO of a company did anything but lead a company (not to downplay the difficulty of leading a company). Musk certainly did not design rockets that land themselves. I highly doubt that he could even write down (let alone solve) the main important equations when asked. It's funny how some execs are attributed with doing everything (Jobs and Musk are the main ones), while nobody would say Adani dug up coal from the ground.

Watching even 1 video of an everyday astronaut interview with Musk makes it completely obvious you are completely wrong. And so do tonnes of testimonials on quora from spacex employees.

Musk is the chief engineer of spacex rockets. He's not just a CEO.

Do a little research before making comments like this.

Twitter API is what publishers use to bring content onto Twitter. That content is a reason why other users visit Twitter. With that content gone, Twitter is a lot less attractive to many.

Will be interesting to observe ... will publishers pay? Will users enjoy without those?

The gp comment is talking about the indirect benefits of having an API, which is a large number of free developers and researchers help you improve your product at a pretty low cost (compared to if you paid them directly

> From a business perspective, you don't want to alienate developers who work to build your ecosystem

I'd love to have free API access to all kinds of companies.

Think how much easier and faster it would be booking your travel if all airlines and hotels let you straight in to the raw pricing and availability data from their reservation systems.

Think how much money you could save if retailers exposed all their product availability and pricing data, live.

They don't, because they all regard this stuff as "commercially sensitive".

Why should Twitter be any different?

Business pricing is very often commercially sensitive as you don't typically want the end consumers to see how much profit is being made from them and how much discount is given to wholesalers.

Twitter data is rarely anything to do with pricing, so that's why it's different.

> Business pricing is very often commercially sensitive [..]

I'm not after internal pricing/B2B pricing/discounting, in my thought experiment I "just" want free API access to information that's already on public websites (eg end-user/retail pricing for rooms/flights/widgets). I'm not aware of any airlines/hotels/retailers that allow even that. In bulk, it's commercially sensitive too.

I've heard that they do a lot of price manipulation with holidays, so I can see why they wouldn't want to provide enough data for customers to track prices over time. Also, you'd be bypassing their advertising.

But yeah, it would be great if we didn't have to use their websites to navigate the data.

> you'd be bypassing their advertising.

Right.

So back to where we started: what's actually wrong with Twitter deciding to insist on eyeballs on twitter.com instead of scripts pointing at their free API?

I wouldn't say that it's necessarily 'wrong', but I do think it's a bad decision. The problem is that they've let people have access and build upon their free API and pulling the rug from under them is going to annoy them and generate bad will towards Twitter.

In some ways it reminds me of the Hasbro/Wizards of the Coast licensing shenanigans where the business is annoying people building an ecosystem around their products which arguably adds a lot of value.

> The problem is that they've let people have access and build upon their free API and pulling the rug from under them is going to annoy them [..]

Yup, although we all know that Twitter just the latest example in a loooong line of companies to have pulled this stunt since, well, what feels like forever.

> and generate bad will towards Twitter

I'm fairly ambivalent about this aspect, and I doubt Musk cares much, it's not as though the haters weren't already hating before this latest incident.

If charging for API usage amounts to developer alienation, then why is Apple the most successful business in the world?

Frankly, I don't think Musk much cares for the developers. From a business perspective, he might be right here - developers aren't the ones watching ads or paying for microtransactions, users are. The developers, at his scale, are a liability. Restricting their capabilities and profiting from their struggle is just part of the modern tech stack. I hate it as much as you do, but calling it "insane" is a hysterical double-standard that crops up just too often on HN.

If the App Store had been free for 15 years and then Apple said "if you don't pay us $100 a year in one week, we'll delete your app" then you can bet that Tim Cook would be seeking new opportunities _within hours_. There's a big difference between "this has always cost something" and "this is going from free to paid with _one week's notice_".

I don't see how third party developers are additive to Twitter in the same way they're additive to Apple at all. If that's true at all, why would Twitter provide services for free? Why would Apple charge at all? (Especially from the beginning).

It's a hard comparison because the value of the Apple ecosystem is so obvious and mutual. Twitter is junk on both sides; unless as a third party dev you just happened to find value in it, in which case you should pay for it.

I understand this is hurting peoples feelings, but it makes complete sense to me from all angles.

> I don't see how third party developers are additive to Twitter in the same way they're additive to Apple at all.

When did you start using Twitter? Twitter as we know it today is largely a product of the community and third party developers. Retweets, hashtags, quote tweets, the use of a blue bird symbol; all came from either the community at large or specific third party devs. The fun bots also drive a lot of twitter engagement; honestly they're the only real thing I miss on Mastodon (while some of the most important ones did come across, a lot didn't). Third party software also makes Twitter a support system, a status reporting system... The list goes on.

However, beyond all that, even if there was a good argument for killing the API free tier (there is not), no-one sensible would do it with _one week's notice_. This will alienate even commercial users who would be willing to pay. If you're using your CRM's twitter integration to track customer complaints, say, unless you're huge the free API is probably sufficient today. With one week's notice, even assuming you hear about it the first day (most will not) good look getting a purchase order for the API within a week if your company is even remotely bureaucratic (most are).

The patron saint of Dunning-Kruger presumably thinks it's a good idea, but can't imagine anyone else does.

Because Twitter makes money from advertising not selling direct to end users.

I used to use Tweetbot which has now stopped working, as a result I don't use Twitter any more, the advertisers have lost one extra person to sell to.

> the advertisers have lost one extra person to sell to

Which is aligned with Musk's strategy thus far, but there's also a good argument to be made that that product should have never been needed if twitter was providing a decent first party client. They should have wanted to provide that to control the advertising. Letting the third parties take that up was a poor strategy -- business-wise that is, I get that people may have liked it and have used it for a long time and are probably sad to lose it.

> I don't see how third party developers are additive to Twitter in the same way they're additive to Apple at all.

They bring the content users want to consume.

> I don't see how third party developers are additive to Twitter

Then you're not in the community in which the third party elements(e.g. tweet aggregation, SSO login, fine grained access control) are vital part of. Which may be substantial, or not, and we as the sheep don't know.

You're right and I'm my sheepish hypothesis here is it's not substantial. I think twitter is largely a toy, if it were to completely vanish tomorrow, very little harm would actually be done to anyone. It's like a PMF question, does twitter have third party developers banging on their door with fists full of cash? No, they have a bunch of freeloaders that have built derivative toys.

Apple can nickel and dime developers because a good app on their ecosystem is profitable.

Nobody is making enough profit via the Twitter API to make it worth staying on. Twitter is a distant #3 social network that itself has never really made money.

> Twitter is a distant #3 social network

This is being too generous. Facebook, Instagram, and TikTok are all far larger than Twitter. Reddit is also just barely larger than Twitter, though at that scale you have to consider who has more bots. Twitter claims slightly more MAUs than Snapchat, but since Snapchat isn't a viral network it probably has no bots to speak of, making it larger than Twitter in terms of real users.

But this comes after pulling the plug on the third-party apps without a word of warning or even acknowledging that they had done it. So those devs turned around and said, well, guess we'll go work on Mastodon apps now. Presumably there was a noticeable drop in engagement as those users cut back their time on twitter. So now twitter is like you can come back if you pay, but we can't tell you what that will cost.

You don't need to pay to use Xcode, or to run apps on your own devices. If you publish to the app store I think there is a $99/year fee or something to have your app signed, plus they take 30% of your revenue.

You don't absolutely need to have your code signed, I can give apps to my friends, they just get a warning about it not being signed and potentially unsafe when it's first installed.

You need to buy a Mac to even develop these apps...

So? You need a computer to run any code at all. I bought a used Mac for very cheap last year, the first Apple product I've owned in ~35 years of working in and around the computer industry. I don't feel ripped off.

Good for you. In other UI frameworks I don't have to buy their specific hardware just to develop for them.

15% unless your app is hugely popular.

I’m sure it didn’t help that Elon pissed off most of Twitter’s advertisers. They jumped shipped as soon as he took over.

> Daily Revenue Has Dropped 40%, 500 Top Advertisers Have Left

https://www.theinformation.com/articles/twitter-manager-dail...

Twitter wouldn't die if they don't pay bills, someone would buy it for huge discount. It seems like it would be the best way forward, depending on who would buy it. A lot of people don't follow news or Elon drama, the brand still has a lot of value. Especially abroad where news about Elon are much fewer.

It'd have to be a huge discount as they'd be buying the debt along with it. Why would someone want to saddle themselves with an unprofitable debt burden?

BofA/whoever lent $12B or whatever to Twitter/Elon. Ultimately they have to settle for lower payments or get Tesla stock or whatever, and the debt is cleared. Or it defaults. Either way, Twitter the service has no reason to shut down, and doesn't carry that debt further. Once the debt cost is sunk, what matters is future marginal profitability.

I agree. The question then is how much the lenders are willing to settle for or whether it'll need to go through bankruptcy first.

Sure. Is that person going to run it for free alone in his basement?

It's not only the code base. It's employees, servers, processes, data...

Even if you get twitter for free, you still need to pay to run it.

The social graph on Twitter is hugely valuable. Even if they went bust and the service would be unavailable for a year, the social graph and brand value are important.

One reason Mastodon and others have trouble is that people don't want to follow random people, they want to follow the same people they did on Twitter. It's their community.

Certain communities did make a successful switch, like Apple and Tech journalist communities, but not a lot of other communities.

> The social graph on Twitter is hugely valuable.

This seems to be taken as given, but how specifically would you expect someone to simply monetize the social graph? Just because data exists doesn't mean that anyone wants to pay for it.

As for the value of the brand itself, we could say the same thing about Yahoo in 2010.

Dear advertisers, so you want to reach _specific_social_demographic? Good news, I can tell you 500k people who have demonstrated 3 points of identification with that demo in the last 14 days and are minimum 5% likely to interact positively with a promoted tweet from a firm in your market sector.

You can really pull a lot of useful data on someone by just looking at their follow choices, self-identification, and reply/retweeting behavior. And that's before you add all the information Twitter maintains on things like profile views or video plays, or the consistency of entity identification in tweet content etc. I can give you very specific kinds of demographics like 'people who enjoy tweets about dogs and football but not cats or politics.'

This market already exists (I get spam offering customer lead lists every couple of days). I suspect most orgs of a reasonable size are already being bombarded similarly.

Augmenting these lists with Twitter's data might improve/expand the lists bur won't create much extra value imo, because the demand side won't change.

There are tools to help you with that. Like https://fedifinder.glitch.me/

Why not both?

>This seems like more management by manchild.

Whaaaaaa but Elon is the god king who will deliver us from evil tech giants!! Are you saying he might be a greater conman then PT Barnum?

I don't think that's too related to cutting down bots. It's about making money from their dataset, which is valid. Why do people think they can hammer an API without paying?

They already have an enterprise tier for people who want to build commercial stuff on top of Twitter, and they already monetize their datasets and have done for years. How do you expect to sign up students/new developers if they have to pay before they figure out how to use the API?

Well they mentioned "basic tier API", so I guess let's wait for the pricing and capabilities, no? My main criticism is they're not being assholes just for charging for an API access to their data.

You can't "wait for pricing and capabilities" as they're cutting access literally one week from now.

No attempt at making this seem like reasonable behavior and communication with developers can succeed here.

And by the way if you expect reasonable cost, you're in for a surprise. Twitter is DESPERATE for cash.

In my case I literally just had a bot running to pull in my own following/followers list so I could diff them and then tweet out the result privately so I'd be aware of any changes to it.

It was useful to check to see if someone else decided to softblock me/deactivated their account (and in at least one case, has helped me to reach out to someone in need). That's not "making money from the dataset", it was a hobby project and I imagine the overwhelming majority of Twitter bots are that. (Most of the ones I used to follow generally just scraped various subreddits to tweet out/source fun art to look at).

I'm not gonna pay money to access my own data, Musk can sink in his own shit for that and I don't blame any of those other hobby bot devs for not wanting to do that either.

_Their_ dataset? The data comes from the users!

No, the data comes from users' interactions with the platform, and they have to run a massive engineering feat to take all of that and run 24/7. It's expensive and so it's reasonable to charge something

> This seems like more management by manchild.

"Manchilds" track record in this Game of Life is by and large a testament to how well he understands its rules. And, maybe, how silly they are.

To be fair, having a lot of money is an excellent way to have more money. Having a lot of money and making some lucky gambles in early technology plays is an even better way to have a lot of money. The fact that every company he owns a significant stake in has a 'manage Elon so that everyone else can get things done' team except for twitter and all of them are not doing insane things, except for twitter, seems to be a pretty strong indication that other enterprises he's involved in are not succeeding because of him, but in spite of him.

It's simple: Use the website. It will never stop working because people use it and you can automate and scrape it just fine. Kind of sucks that you have to go through the effort - but then there is no social media corporation that isn't toxic in some way to it's userbase.

I actually don't mind scraping it that much, and even enjoy the adversarial aspects of writing scrapers. but it makes a lot of high level functionality like filtered streams or historical search much less accessible; I'd probably never have learned a lot of network analysis stuff over the last decade or so if I'd had to to pay to access streaming data first. Also, I think it's going to be harder for academic researchers to get institutional approval to scrape adversarially, so it could put a dent in a lot of social science research by forcing people to chase grants instead of focusing on their code.

It's ironic, for a couple (non-Twitter) projects I wound up scraping because either a) they didn't have an API yet (e.g. early crypto pricing sites) or b) I wasn't confident the API would remain intact over the long term. Kind of depressing.

> I think it's going to be harder for academic researchers to get institutional approval to scrape adversarially

This is a good point about what might happen, but it seems worthwhile to address and fix directly. Personally I don't see why adversarial scraping of a publicly published website should require any more ethical consideration/review than using the suggested API would. Ethical concerns should revolve around humans, not the business desires of non-human entities.

Also quality residential proxies are pricey. You need to rate limit and rotate both IP and puppeteer fingerprint when adversarially scraping.

The website is awful. It's horrific. It was the worst site I visited regularly until I started using Nitter instances. I'd rather not know what is going on than go to it. I wish they'd wind the UI back 10 years, to back when it was pleasant to browse.

The point is that the website is an API, irrelevant if it is a good or bad UI.

Depending on twitter's pricing, this might not be worthwhile.

Scraping web pages is a last resort for when proper APIs are not available.

I look forward to someone teaching Elon Musk what version control is, resulting in him wholesale rolling back Twitter's entire stack to 2008. At last, Twitter is written in Ruby again!

Do you remember, I think it was 2020, when there was a big "hack" and tons of blue check accounts started posting fraud links?

Two important take-aways from that:

1) The attackers just pulled the old "Hey, I'm from IT service desk and need your password."

2) Apparently every engineer had god mode in production

I can't imagine what scary monsters are in that codebase.

I too miss the fail whale.

IIRC the twitter website goes to great lengths to mangle itself to prevent ad blockers presumably.

Of course this used to have the side effect of breaking significant swathes of basic browser UX, especially in areas of accessibility. I assume they’re better now than they once were, but given musk’s historical behavior I assume he won’t consider breaking something like accessibility to be bad or problematic.

> musk’s historical behavior I assume he won’t consider breaking something like accessibility to be bad or problematic.

I don't see why would he spend more to make website less accessible. Not fixing bugs - yes, just about everyone does so. But breaking it intentionally costs money.

Tesla's aren't THE most accessible (by cost) EV out there, but SpaceX def is.

Supposedly the entire accessibility team got laid off [1], so there might not be anyone left to ensure that changes to existing features do not reduce accessibility nor ensure that new features are accessible. Not that I'd expect Musk to care a whole lot about concerns raised in that regard, considering that he supposedly went ahead with Twitter Blue despite significant (and well warranted) concerns from the Trust and Safety team [2]. So Twitter will probably become less and less accessible over time.

[1] https://www.wired.com/story/twitter-layoffs-accessibility/

[2] https://www.businessinsider.com/twitter-sent-musk-risks-paid...

> Tesla's aren't THE most accessible (by cost) EV out there, but SpaceX def is.

SpaceX is... an accessible EV? What?

Most accessible satellite launcher...

Ah yes, because "what's the relative cost and complexity to procure a launch for my satellite?" is famously a real example of an accessibility problem faced by people with disabilities...

I'm pretty sure you know exactly what the poster meant ...

> I'm pretty sure you know exactly what the poster meant ...

No, I don't. It obviously is trying to say something about accessibility and SpaceX, but I don't see how the sense of accessbility being discussed in the thread even applies to SpaceX, much less what claim is being made.

Agree. It's quite the jump going from a user-centric website to a space launch company that is used by corporations and governments. Like, okay, I guess SpaceX is extremely accessible compared to other options in the space industry, but why would that have any bearing on how accessible the Twitter website is for you and me day-to-day?

Starlink has created more information accessibility that twitter will ever will

But that’s not in any way shape or form the “accessibility” being talked about here.

Just because it’s the same word does not mean it’s the same meaning.

I wasn't sure how to put it into words and this was exactly what I had in mind.

Yes, but since we are judging a person for his merit…

Do you understand where you are?

This is a comment thread discussing accessibility at Twitter, in a post about Twitter discontinuing part of its platform. It’s not a place to circlejerk Musk’s achievements, regardless how much you would want that to be so.

Explain, in exact words, what about SpaceX in any way shape or form would indicate that they know how to handle user a11y and how that would transfer over to Twitter— you know, the thing we’re actually discussing here.

Most accessible (by cost) launch provider.

Accessibility is not a by cost thing, and you aren't paying to break accessibility, you're paying to break scraping and maintaining accessibility when doing so costs money. It also requires having engineers working to keep the site accessible, but musk fired them.

Scraping is much easier when sites are more accessible.

Correct, which is why making something accessible and not scrapeable is harder/more expensive than breaking accessibility.

I think it’s a terrible move to kill the stuff like postybirb that a lot of people use to make posts on multiple websites at once. they’ll have to either change their workflow to make a tweet (annoying) or just abandon twitter altogether. just makes the website worse for practically no benefit.

Does not take much effort. Below is an example using curl. For reading Twitter feeds I just get the JSON and read the "full_text" objects. I have simple custom program I wrote that turns JSON of unlimited size into something like line-delimited JSON so I can use sed, grep and awk on it but HN readers probably prefer jq. For checking out t.co URLs I use HTTP/1.1 pipelining.

Usage for reading is something like (but not identical to)

   1.sh screen_name > 1.json
   yy059 < 1.json|grep full_text > 1.txt
   less 1.txt

Usage for checking out URLs is something like (but not identical to)

   unset connection
   export Connection=keep-alive
   yy059 < 1.json|grep full_text \
   |yy030 \
   |grep "https://t.co/.{10}$" \
   |uniq \
   |yy025 \
   |nc -vv h1b 80 \
   |sed -n '/location: /s///p' \
   |ahref > 1.htm
   links -no-connect 1.htm

"ahref" is just a script turns URLs on stdin into simple HTML on stdout

Alternatively if I do not trust the URLs I might use a script called "www" instead of ahref. It takes URLs on stdin and fetches archive.org URLs wrapped in simple HTML to stdout, using the IA's cdx API.

  #!/bin/sh
  SCREEN_NAME=$1
  COUNT=500
  PUBLIC_TOKEN="Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA"
  GT=$(exec curl -A "" -s https://twitter.com/$SCREEN_NAME|sed -n '/gt=/{s/.*gt=//;s/;.*//p;}');
  echo "x-guest-token: $GT" >&2;
  REST_ID=$(exec curl -A "" -H "authorization: $PUBLIC_TOKEN" -H "content-type: application/json" -H "x-guest-token: $GT" -s "https://twitter.com/i/api/graphql/mCbpQvZAw6zu_4PvuAUVVQ/UserByScreenName?variables=%7B%22screen_name%22%3A%22$SCREEN_NAME%22%2C%22withSafetyModeUserFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%7D"|sed 's/\(rest_id\":\"[0-9]*\)\(.*\)/\1/;s/.*\"//'); 
  echo "rest_id: $REST_ID" >&2;
  curl -A "" -H "authorization: $PUBLIC_TOKEN" -H "content-type: application/json" -H "x-guest-token: $GT" -s "https://twitter.com/i/api/graphql/3ywp9kIIW-VQOssauKmLiQ/UserTweets?variables=%7B%22userId%22%3A%22${REST_ID}%22%2C%22count%22%3A$COUNT%2C%22includePromotedContent%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22dont_mention_me_view_api_enabled%22%3Atrue%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_uc_gql_enabled%22%3Afalse%2C%22vibe_tweet_context_enabled%22%3Afalse%2C%22responsive_web_edit_tweet_api_enabled%22%3Afalse%2C%22standardized_nudges_misinfo%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%2C%22include_rts%22%3Atrue%7D"

There's no way I would use the Twitter website as it requires enabling Javascript and not for the user's benefit.

This solution isn't pretty but I can easily keep tabs on Twitter feeds without any need for a Twitter account, a Twitter "API key" or a so-called "modern" browser.

Can u scrape search feeds too? Ie tweets that match a certain string?

You can scrape anything you see in the UI (and sometimes stuff you cannot see). Twitter makes almost no effort to stop people from using their internal APIs, which is why them saying discontinuing the free public API is to stop malicious bots is pretty laughable. Unless they seriously increase their detection abilities for non-approved clients using their internal API, it would take any malicious actor all of a few hours to transition to using the internal API for whatever they want. Honestly, I assumed most bad actors would already be doing this, since things like spamming were already against the ToS of the public API.

What happens if/when they block that Bearer token?

The token has been the same since at least 2020 when Twitter started using GraphQL instead of REST.

Every person visiting twitter.com is using this same token. The token is neither personal nor private.

What would be the point of changing or blocking it.

> Every person visiting twitter.com is using this same token.

Be interesting to see if that stays the same when they're charging for the API but leaving a huge loophole with this token.

Twitter is not alone in using GraphQL this way, having all website visitors use the same token or key. Other websites do it, too, as shown below.

Using GraphQL like this can be an effective dark pattern because to anyone using a "modern" browser that "tech" commpanies control it makes it seem like the text of the website cannot be retrieved without Javascript enabled. That's false, but nonetheless it gets people to enable Javascript because the website explicitly asks them to enable it. Then the website, i.e., "tech" company, can perform telemetry, data collection, surveillance, and other shenanigans.

Sometimes this practice might not be a deliberate dark pattern, it might just be developers who are using Javascript gratuitously. For example, HN search provided by Algolia uses GraphQL. HN puts URLs with pre-selected query terms and a public token ("API key") on the HN website. Everyone that uses those URLs uses the same key.

Unlike Twitter, HN istelf does not ask anyone to enable Javascript. The website works fine without it, including the Algolia search, as shown below.

Usage is

   1.sh query > 1.json

   #!/bin/sh

   curl -A "" -d '{"query":"$@","analyticsTags":["web"],"page":0,"hitsPerPage":30,"minWordSizefor1Typo":4,"minWordSizefor2Typos":8,"advancedSyntax":true,"ignorePlurals":false,"clickAnalytics":true,"minProximity":7,"numericFilters":[],"tagFilters":["story",[]],"typoTolerance":"min","queryType":"prefixNone","restrictSearchableAttributes":["title","comment_text","url","story_text","author"],"getRankingInfo":true}' "https://uj5wyc0l7x-3.algolianet.com/1/indexes/Item_production_sort_date/query?x-algolia-agent=Algolia%20for%20JavaScript%20(4.0.2)%3B%20Browser%20(lite)&x-algolia-api-key=8ece23f8eb07cd25d40262a1764599b1&x-algolia-application-id=UJ5WYC0L7X"

Here is a non-curl version of HN search using custom HTTP generator yy025 and h1b, an alias for localhost address of TLS forward proxy

    #!/bin/sh

    export Connection=close;
    export Content_Type=x-www-form-urlencoded;
    export httpMethod=POST;
    x=$(echo '{"query":"'$@'","analyticsTags":["web"],"page":0,"hitsPerPage":30,"minWordSizefor1Typo":4,"minWordSizefor2Typos":8,"advancedSyntax":true,"ignorePlurals":false,"clickAnalytics":true,"minProximity":7,"numericFilters":[],"tagFilters":["story",[]],"typoTolerance":"min","queryType":"prefixNone","restrictSearchableAttributes":["title","comment_text","url","story_text","author"],"getRankingInfo":true}');
    export Content_Length=${#x};
    echo "https://uj5wyc0l7x-3.algolianet.com/1/indexes/Item_production_sort_date/query?x-algolia-agent=Algolia%20for%20JavaScript%20(4.0.2)%3B%20Browser%20(lite)&x-algolia-api-key=8ece23f8eb07cd25d40262a1764599b1&x-algolia-application-id=UJ5WYC0L7X"|(yy025;echo "$x") \
    |nc -vv h1b 80

Reply on news.ycombinator.com