Twitter now requires an account to view tweets

Which just leads people to scrape.

I don't despise Musk at all. Don't agree with him on everything, but he is a genuine and interesting person.

It's interesting how you assume that most people despise Musk.

That said, there are concerns with data aggregation, as patterns and trends become visible which aren't clear in small-sample or live-stream (that is, available in near-time to its creation) data. And the creators of corpora such as Twitter, Facebook YouTube, TikTok, etc., might well have reason to be concerned.

This isn't idle or uninformed. I've done data analysis in the past on what were for the time considered to be large datasets. I've been analyzing HN front-page activity for the past month or so, which is interesting. I've found it somewhat concerning when looking at individual user data, though, here being the submitter of front-page items. It's possible to look at patterns over time (who does and does not make submissions on specific days of the week?) or across sites (what accounts heavily contribute to specific website submissions?). In the latter case, I'd been told by someone (in the context of discussing my project) of an alt identity they have on HN, and could see that the alternate was also strongly represented among submitters of a specific site.

Yes, the information is public. Yes, anyone with a couple of days to burn downloading the front-page archive could do similar analysis. And yes, there's far more intrusive data analytics being done as we speak at vastly greater scale, precision, and insights. That doesn't make me any more comfortable taking a deep dive into that space.

It's one thing to be in public amongst throngs or a crowd, with incidental encounters leaving little trace. It's another to be followed, tracked, and recorded in minute detail, and more, for that to occur for large populations. Not a hypothetical, mind, but present-day reality.

The fact that incidental conversations and sharings of experiences are now centralised, recorded, analyzed, identified, and shared amongst myriad groups with a wide range of interests is a growing concern. The notion of "publishing" used to involve a very deliberate process of crafting and memoising a message, then distributing it through specific channels. Today, we publish our lives through incidental data smog, utterly without our awareness or involvement for the most part. And often in jurisdictions and societies with few or no protections, or regard for human and civil rights, let alone a strong personal privacy tradition.

As I've said many times in many variants of this discussion, scale matters, and present scale is utterly unprecedented.

Because it was thereby starting to compete with his own for-profit.

He was even asking for a moratorium for 6 months so his company could catch up.

However if you mean implementing some even worse obfuscation (kind of like FB putting parts of words in different divs etc) that is not really compatible with the situation that this needed to be done as more of a temporary emergency measure. And PoW doesn't sound reasonable because it sets mobile devices against the scraper's servers. If all of this was just so easy, scraping would be dead. Good that it isn't.

Also a side note: distributed Web crawler is not unheard of these days, as well as residential IP proxies. Meaning the effectiveness of Proof of Work model maybe also limited.

Correct me if I'm wrong, but surely throttling scrapers (at least ones that are not nefarious in their habits) is a problem that can be mitigated server-side, so I find it somewhat galling that its the excuse.

This works far better when the items requested are small in number but large in volume (that is: a large number of requests against a small set of origin resources). When dealing with widespread and deep scraping, other strategies might be necessary, but these aren't impossible to envision.

Specifically permitted scraping interfaces or APIs for large-volume data access would be another option.

Of course, there's the associated issue that data aggregation itself conveys insights and power, and there might be concerns amongst those who think they're providing incidental and low-volume access to records discovering that there's a wholesale trade occurring in the background (whether that's remunerated or free of charge).

Many people who aren’t that privacy conscious would however object to lots of companies, big and small, sucking their content into their databases for their own uses, then republishing after it’s passed through a few AI models.

Not a very significant distinction: if active users stopped posting, scrappers wouldn't have much of a reason to keep scrapping.

It's shared ownership. You own it, but give Twitter non-exclusive permission to also use it.

This is why news agencies request permission from a twitter account before sharing a picture they took.

I can think of a few possible reasons. They might want more up-to-date info, or they might have no real developers and the scraper was created by a business guru who prompted ChatGPT and didn't understand the code that came out.

Given what else Musk has asserted about Twitter, and how often former or current Twitter devs have contradicted him, it may not even be what Musk said.

> Twitter is not higher quality data than any other web page

Eh, depends how much you can infer from retweet, favourites, etc.

Won't be the only such site, but it's probably better training data than blog posts are these days.

But yeah, I absolutely agree that Twitter doing this caused a lot of damage to any orgs, corporate or government, which wanted to be public, anything from restaurants announcing special offers to governments issuing hurricane warnings. Twitter isn't big enough to assume everyone has an account, like Facebook is.

This is just kicking the can down the road.

We have a major structural problem now. We want data to be free and machine readable, but no startup (and even a giant like Twitter) can afford the server cost to withstand all those machines.

Taking/scrapping/stealing that data out of said platform for the benefit of your over-hyped “disruptive” startup - and implying that others should give you all for free - is the issue.

1. Users or one might say content creators don't own their data. Not just do those platform owners make a lot of money with the content (which they have a license to, as per site ToS) but then you have third parties scraping it now for commercial products. Using the data to train models that are then sold back to some of the same social media users who produced the content for free in the first place wasn't a thing until very recently, it used to be a select few doing machine learning research in the past. The laws are lacking behind the tech development and regular internet users are being exploited because of it.

2. It absolutely is stealing in some cases, and even worse. For example when they scrape it for content which they then use to train their bots to impersonate humans. Or on Twitter, there's a very common type of bot that steals content from young attractive female social media users in China, auto-translated to English, to pose as them. If you're in finance and crypto circles they're swarming with these accounts (guess the scammers know their targets).

3. In general this is only going to get worse from here on. LLM are getting better and better. On sites like Twitter you already have no idea if you're interacting with a human or not. But these "AI" can not actually think for themselves, they can only emulate, they can copy other humans. At least so far. So for the sake of making progress and ensuring we can still have intelligent discussions and find novel ideas online, it's imperative to have a way to keep the machines out. Social media must become sybil resistant or it dies in a vicious circle of self-referencing bots ever parroting the same old talking points, or variations thereof. We urgently need human ID!

Bots aren't people and can't read nor consent. They just consume.

Any page which can be served without first displaying a ToC or other terms which explicitly prohibit access is not protected by a ToC or other license from scraping, as they can be considered a Point of First Contact in each case, as the bot has selected each link from a simple aggregation of all links it encounters (each interaction being "new" in essence).

Now it could be argued that ignoring robots.txt is an explicit contravention of norms and standards which could be viewed as a violation of an implicit licence, but there is no law requiring adherence to robots.txt and thus no mandate that a program even look for it iiuc.

Twitter is maybe at the dog stage. Perhaps it’ll die.

If these data scraping operations are as sophisticated and determined as he claims this measure is insufficient and actually it really hurts Twitter far more than it helps. Case in point: we stopped sharing Twitter links because when you click them in most iOS apps it opens up an unauthenticated web view and presents you with a login screen. So we just collectively decided “ah ok no sharing Twitter” and moved on.

I’m sure there are companies scraping Twitter. I just don’t buy that it’s as big of an issue as he claims it is, and that preventing people from viewing tweets without logging in is a way to mitigate against that (I’d first look at banning problematic IP addresses first, personally).

To me it’s either:

1) a very poor and very temporary mitigation against scraping, that could be bypassed with a bit of effort

2) an experiment in optimising metrics - Musk sees lots of unauthenticated users consuming Twitter, tries to steer them into signing up

3) it’s all just a big mistake

Option #2 makes the most sense to me, but frankly none of them are good

It can add nontrivial load.

You don't need to use a bot to do this, Twitter literally did this to themselves through their own buggy code https://sfba.social/@sysop408/110639474671754723

If Silicon Valley was still being produced, this would make for a great episode.

To be fair, Musk will regularly pay lip-service to the idea of Open communication. I guess that's not literally nothing, but most large site policies have been in the direction of locking down content.

If there ever was a version of Musk that cared about Open access, it's been a while since that version of him saw the light of day. It's very consistent with his overall behavior to believe that he views Twitter content as being primarily his property rather than a community resource, and that he thinks that scrapers/AI companies/researchers are literally stealing from him if they derive any value at all from data that Twitter hosts.

I see where you're coming from, but if Twitter is in a position where it can't roll out those protections right now, given its current head counts, etc... it's not going to be in a position where it can roll out those protections next week. Probably not next month.

So it's less that no one could block companies from scraping Twitter (although anti-scraping mechanisms are probably always going to be a cat-and-mouse game, so I'm not sure that there is ever going to be a perfect easy solution). It's more that if Twitter can't do it right now, nothing is going to magically change any time soon about the situation it has found itself in. And waiting a year (even waiting 6 months) for tools to become available before rolling back this rate limiting would be incredibly self-destructive for Twitter.

The way I see it, they're basically guaranteeing that they will need to roll back these changes before they have a solution to whatever specific problem or irritation Musk is fixated on. They're not going to gain additional engineering capabilities in the next week. And how long does Musk plan to leave rate-limiting in place? A social media site where people can't look at content is just broken.

We are oriented to think that way after ~15 years of the algorithmic engagement-maxi world of Twitter. It always looks like there's a lot happening all the time but look deeper and it's a bunch of people offering their weak takes on hot topics to build their brand.

What was the last thing you remember being must-see sfuff on Twitter?

Is that the best way? No clue. But I'm glad to chip in.

(Note: I'm of the opinion that fediverse-style federation in the context of forums is merely a nice-to-have; the web is already naturally federated, and people should not feel bad if they want to save money/tech complexity/administration complexity by settling for ordinary self-hosted forums.)

Secondly, signing every message wouldn’t be impractical at all, I don’t think. We’ve had the technology to do this for a long time and it’s very simple. What we don’t have is good key management. For average users, this would have to be something provided by their devices (phone or the Secure Enclave in your Mac or whatever) - managing keys and the web-of-trust shamozzle are the main reasons why encrypted email for everyone never took off.

So as long as both your source and destination support account transfers, you can usually switch and even seamlessly bring along most of your followers without them noticing.

No idea about your admin question. All bets are likely off with a bad admin. If you want actual cryptographically guaranteed communication, that doesn't exist in a usable form (except for Secure Scuttlebutt, and that's reeeeally stretching the "usable" part)

My recommendation was much like MX records for email, so you can use a hosted server under your own identity.

This is why I want domains as identities to succeed. I want to own my handle on every platform, but I don’t want to self host.

https://aumetra.xyz/posts/fediverse-nomadic-identities

https://aumetra.xyz/posts/nomad-id-first-steps

Current spec drafts at <https://github.com/WebOfTrust/keri>.

* https://jdebp.uk/FGA/fediverse-third-party-mirrors.html

I have been using Twitter more often lately as I really don't care about what they do to their APIs or whatever, and it's honestly improving lately.

But I guess for publishers it never produced enough revenue to be worth maintaining the integration.

The database has doomed us all! For it is the Seed of all Evil in the hands of the Wicked! ...and arguably the Good, but misguided!

[0] https://github.com/discourse/discourse

https://ziggit.dev/

One is that centralized _business_ has certainly not been the primary organizational mode. You can talk about centralized _government_ (of whatever variety you'd like), but the distinction there is that the centralized government had some sense of itself in an ecosystem - a citizenry, a land, a future, etc. - and businesses do not.

The second is that centralized entities sure wrote a bunch of stuff _down_, but it's hard to say they were the primary organizational mode for any but the last 50-200 years - the reach of serious centralized bureaucracies has only really begun to match their propaganda in the industrial and now computer age. Until extremely recently, the actual effective reach of a centralized bureaucracy was a day's horseback ride - control degrades rapidly as one leaves the core. "Heaven is high and the emperor is far away", as the saying goes.

Edit: With regards to the second note, "Against the Grain" by James C. Scott is a solid read.

If you own a company and want it to last through the ages and you aren't literally the only guy in town, never become number one. Aim high, but don't hit the top.

The same can be said for countries and practically any organization or group. You stay an underdog if you do not want to ever fail.

For most of human history, most businesses were small ma and pa shops operated by a few local people. These days large business chains are the norm. You could say that centralized big business killed the decentralized small ma and pa shops.

I suspect the later. We’ve seen billionaires do it countless times.

[0] https://www.independent.co.uk/life-style/health-and-families...

I used the 'redirect to nitter' Firefox extension and Android app but it got quite unreliable and nobody else that I know uses nitter at all. I think Nitter users would be a tiny, perhaps even immeasurable minority compared to casual readers that now are incentivized to either log in or fuck off (but... FOMO).

Publicly free content from users devolves to garbage content. I think the Chat GPT effect is they're realizing its easier for companies/entities to generate garbage that is at or exceeding the intelligence of comments by actual people (a low bar). Sure there are pockets of usefulness but this is tiny amongst the firehose of garbage.

If all that is publicly available on social platforms is just garbage nonsense, people will just stop going if any barrier is thrown in front of them. The internet as a technology stack is fine. This is how social media dies (hopefully).

A potential saving grace: I bet within a year or so it will be easy to self-host LLMs that are easy to fine tune and run. Then there will be a few open source tools that you can use yourself, privately, to capture your level of interest while reading, and periodically make a reader/summarize/filter agent.

This is not scary if people can fairly easily run it all themselves, keeping their data private. It would help wade through crap, and there is some irony in using LLMs to have personal filtering and summarization.

Compared to future versions of ChatGPT, Bard, etc. the models that individuals can self host will be much weaker, but I think that they will be strong enough for personal agents and eventually be cheap to run, and affordable to fine tune.

The clock is ticking until you’ll need to decide which internet package you prefer, based on content providers and limited by region.

At least Yahoo will probably come free with the base package.

Those sites never fit my definition of open anyway (free, permissively licensed technology and content). The ones that do are smaller, aren't a monoculture and seem to be pretty untroubled so far. No one wants to scrape the little Mastodon or Lemmy instances or other small community sites I pay attention to.

Big deal if something is a threat to the social media business. The social media business is a cancer which should be destroyed anyway. Go outside and engage in real socializing instead of the depression-spawning, teen-girl-murdering version peddled by Zuckerberg and Musk. It's much better and once you change your habits you'll never look back. Maybe it has something to do with all the vitamin D you get from being outside.

The only time I run into this Fuck You pattern is Instagram refusing to let you replay a video.

https://github.com/LemmyNet/lemmy/issues/622

I would not bed with these fellows myself.

Between account walls and search’s indexing problem, it’s become very hard to find small to mid sized active communities on your own. In fact this problem seems to be something people are trying to solve in Reddit communities via related subreddits on the sidebar.

So having gone from using search engine’s to crawl for relevant content that was out there, people are now creating content specifically to end up in Google’s search results - destroying the value search once had. Indicated by what people have done on Reddit, and these discussions about finding alternatives, it seems we are well on our way back to webrings. I welcome this.

This is not true. Communities can be browsed without login on their home instances.

Every user hostile move they've made basically halves my usage.

I used to spend way too much time on Twitter, now it's 10 minutes a week and I don't feel like I miss much.

This is actually the most mood-lightening comment I've seen on this matter. A return to the internet of 15 years ago doesn't sound so bad to me. (Of course, the mood darkens again when I remember that it won't really be that, because, e.g., that internet of 15 years ago couldn't handle the bot-spam of today.)

That's actually a nice way of putting it! Better than the "fix" version, because this is clear on the consequences.

I feel like it might be applied to everything from OS UI design (Windows 11), web platform redesigns (Google's icons) to whatever is going on with social media (the silly enshittification term describes this) and many other things.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

(I've a slightly more updated set locally, can share those if requested.)

The one thing I see as a future issue is that people are starting to post comments that clearly look like they were manufactured by ChatGPT and friends. Or that could just be the way some people talk and I've spent too long with ChatGPT now and start to smell it everywhere.

https://news.ycombinator.com/item?id=22930391

[dang]: "Push notifications seem to jack up the nervous system in a way that's good for engagement but not necessarily for users..."

I believe this eternal loop of change is a trap that impatient people force themselves in. Instead of accustoming and learning the current state, they rush into another one with a change that has unclear implications. As a result, they never get where they are and lose any track of where they were or where they’re heading at. Their only comfort can be found in a constant change.

These few sites are like home to me. One of them I visit with years-long pauses and every time I return it’s the same user experience. That’s invaluable.

I've been sort of live-blogging the experience on the Fediverse: <https://toot.cat/@dredmorbius/tagged/HackerNewsAnalytics>, as well as in some of my HN posts.

My current tack involves looking at sites (as reported in parentheses at the end of each HN front-page post title) and classifying those. With slightly more than 30% of sites categorised, I can classify about 65% of all HN posts.

For the full dataset (17 years), that's roughly:

     1  63913  35.73%  UNCLASSIFIED
     2  22589  12.63%  blog
     3  15112   8.45%  general news
     4  13823   7.73%  tech news
     5  12851   7.18%  programming
     6   8622   4.82%  corporate comm.
     7   8459   4.73%  academic / science
     8   7294   4.08%  n/a
     9   5324   2.98%  business news
    10   3803   2.13%  general interest
    11   2151   1.20%  social media
    12   2074   1.16%  software
    13   1613   0.90%  technology
    14   1463   0.82%  video
    15   1144   0.64%  general info (wiki)
    16   1009   0.56%  government
    17    724   0.40%  misc documents
    18    720   0.40%  law
    19    702   0.39%  tech discussion
    20    620   0.35%  science news

The breakdown-by-year analysis is in development, but if anything programming-specific content as increased in prevalence. Political discussion seems not to have (though it rose significantly ~2014). Cryptocurrency and blockchain-specific sites also peaked about that time (I suspect much of that discussion is now mainstream). General news has always been a huge portion of HN discussion, as have individual (and corporate) blogs.

Note again that this isn't about discussion and comments, or even the titles or article contents (I'm thinking of looking at those, it's ... a challenge for me).

But across nearly 200,000 front-page stories, on which nearly half of all HN discussion occurs (based on another API-based study looking at comprehensive posts), the overall trending seems at first blush to be pretty consistent and if anything improving over time.

(As with all preliminary results, I'm hoping I won't have to eat my words here. Though I'm reasonably confident in most of this.)

From the classifications above, the places you might find some that "suffering" would be in general news, genral interest, and social media categories. All but the first of those are single-digit percentages, and a lot of that general-news content is about technology, business, finance, and science, all of which would crowd out the sort of social and political issues which seem to generate strong feelings.

The "UNCLASSIFIED" sites are a wide mix, though most are probably a mix of blogs, corporate / organisational communications, and the like. The mean posts per site is 1.739951, so gains from additional site-categorisation are pretty slim. I have captured a lot of obvious patterns via regexes and string matches, so academic/science and major (or even minor) blogging and social media sites aren't a large fraction.

More recent discussion here: <https://news.ycombinator.com/item?id=36524001>

I've written about this a fair bit over the years if anyone wants more: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

* ("good" here meaning "as good as possible under the circumstances")

posts not being able to submit images is a huge part of what makes HN valuable (to me).

once HN needs to handle images, HN falls apart

Like SSH keys, where you manage your own identity and then share a public key to each instance that identifies you to that instance.

Like an identity client you could self manage if you wanted to. Make it optional, portable, and transferable. So you can choose to let a server host manage your identity, or migrate to a self managed identity.

If I had more time on my hands….

Something like this?

https://mander.xyz/communities

> An instance dedicated to nature and science. > The main focus of this instance is the natural sciences, and the scope encompasses all of the STEM fields.

Signing up for a Twitter account is free and can be done by literally anyone in the organization.

It is very clear that it is due to them not wanting to comply with bill C-18.

Reported on by every major news outlet that I’ve seen

> The move comes in reaction to the federal government's Online News Act, Bill C-18, which would require the tech giant to pay Canadian media companies for linking to or otherwise repurposing their content online

https://www.cbc.ca/player/play/2221275715931

You wouldn't put quotes around that word for cell-phone companies using licensed bandwidth, or airlines using public airspace, would you?

Coming from the UK I don’t want competition in basic utilities like there is there. I don’t want to have to shop around for the best energy price every year, or to be at the mercy of the free market on pricing for the most basic of necessities like power and water.

It's the hysterical marching cry of young naive people railing against the ever-increasingly vague boogeyman of neoliberalism but the actual evidence doesn't back it up.

You talk about ideology over logic while demonstrating that exact same thing.

> The AER report contains no consistent correlation between higher bills and privatisation.

> The ABS index of electricity prices across Australia, showing movement of electricity prices over time, also doesn't demonstrate a link between privatisation and price rises.

> Whether comparing electricity bills, prices or the relative price index of electricity in each state, there is no consistent link between privatisation and what consumers pay for their electricity.

> Experts say the biggest influences on what people pay for electricity are costs of transmission and distribution. They say these costs have risen in recent years irrespective of whether the owners of the transmission and distribution networks are privatised.

https://www.abc.net.au/news/2015-03-25/fact-check-does-priva...

I don't actually think the Australian grid is a good example of privatisation failure. There's little proof that we have meaningfully lost anything aside from the marching cry of the left.

Google is more like a malaria carrying species of mosquito. Little bites that you barely notice, but in aggregate are actually much more dangerous.

I know the metaphor is a bit stretched but even in the late 90’d it was clear what Microsoft did. Google has gone from “don’t be evil” to “this is the definition of open source” to …

These same government spies run an international network of torture centers.

In the end it’s still private companies (with the exception of public broadcasters or websites).

I was going to host the screenshot on imgur, but I'm not sure if we can trust them anymore...

As far as snark, I see several other examples of that—also intended for Musk—in this thread. They don't strike me as either offensive or particularly constructive, so it's not clear to me why my comment was called out here. (Especially considering that there are a few other comments that definitely go beyond the acceptable levels of user-to-user snark as I understand them.)

I can avoid sarcastic comments about billionaires in the future, if that's a problem. If the issue was snark directed at another user, that wasn't my intention.

I'll also say that the "snark" rule you cited, while well-intentioned, seems very broad and selectively applied here.

My second reaction was “I wonder how hard it would be to take a list of twitter handles and scrape their feeds into an S3 bucket that’s fronted with a CDN.”

I can't tell if it's different people talking now or if people really are that fickle. Probably a bit of both.