Hacker News Reader

Comment by hathawsh | Hacker News Reader

From your link (thanks!):

> 9. Making up words or misspelling hurts

> Words that are identified as “unknown language” are given 0.01, which is a huge penalty.

Does that mean if I tweet about coding and use identifiers like "setUserName", which is not an English word, the tweet gets a huge penalty? If so, that's disappointing.

View on HN

That jumped out at me as a possible misreading of the code. Is it detecting the language of the whole tweet, or just a word as the author claims?

Demoting a tweet that's entirely unidentifiable as any human language seems fair enough.

Man if someone asked me to build a system to merely identify whether a unicode string is human language or not I would flatly refuse. There are thousands of spoken languages, many of them with no standard written form, some that are transcribed into multiple different writing systems, some with no writing tradition at all and with only ad-hoc transliteration unique to each user and use.

Even being 90% confident would be a massive undertaking, and "speakers of this language may/may not use the internet" feels like high stakes for getting it wrong.

It seems a little niche but I'm sure a few times a year some far out town gets connected and suddenly there are speakers of a previously unknown-to-the-internet language newly online.

Note that the metric here is "is the tweet in one of the languages spoken by the user". This hypothetically allows more nuanced implementations than you contemplate.

For example, they could have a language "unrecognized" and assume everyone speaks it.

I broadly find this useful: I see tweets in other languages when they're retweeted by people I follow, and about half the time I machine translate them. But I don't want my whole feed to be that.

Well if someone asked me to do that, I would suggest that it’d be based off their recent tweet history and not just one tweet. And I would make my case in the meeting.

Second, it’s already been done so my next suggestion would be to look what at all the computational linguistic majors have been up to.

I think it would be pretty easy with the language models we have available these days.

And there’s always the options of having unsupported languages or inferring it from user settings or user location.

From a product perspective you will need this feature though if you want worldwide coverage, because very few people are polyglots and most people don’t speak English as a first language.

The actual code comment doesn’t mention “words” but rather if the “tweet language” isn’t in one of the user’s “understandable” languages. As such, I assume your example is perfectly fine (would be extremely surprising if it wasn’t).

Whether the user implies the reader or author, I don’t know, I assume the reader as that would make most sense.

https://twitter.com/aakashg0/status/1641976943141699584?s=61...

Yeah that struck me too. I can see the reasons why you'd want it but the collateral damage on that must be huge.

For example do they check for the common but nonstandard transliteration systems arabic speakers use? There have to be similar systems in other languages that don't use the roman or cyrillic alphabets too right?

Or for that matter what about languages twitter simply isn't aware of? There are thousands with native speakers after all, does this make it basically impossible for them to organically use twitter together?

Also RIP the Conlang community on Twitter...

I speak a language that is highly colloquialized, and in its casual written format includes a rather inscrutable system of abbreviation (one of the features of this system is to basically omit nearly all vowels). I had always figured that this language would be impossible for machines to translate, but I just tested it and both google translate and ChatGPT can accurately identify the language and translate the slang into English (Google didn’t pick up some of the subtleties between similar dialects, but still provided a correct translation). So I’m somewhat optimistic that they could be potentially managing these problems quite well.

What's the language?

Indonesian. ChatGPT could quite reliably tell the difference between Indonesian and Malaysian. Google translate seemed to have a bias towards thinking it was Malaysian. But if I tried Indonesian mixed with Javanese slang (which is a common way of talking), they would both just say it was Indonesian. I only tested a few phrases though, so maybe it breaks down at some point.

Why do we assume that Twitter, a global communications company which has had offices in Dubai, might not consider the nonstandard transliteration requirements of Arabic, the 4th largest language in the world, which would mean that they would now be only showing content that is explicitly non-Arabic to Arabic speakers?

We aren’t giving them enough credit here IMO!

I think the correct reading of the code is that if English is your only language, non English tweets would be weighted 0.01x on your timeline.

Reply on news.ycombinator.com