Hacker News Reader

Comment by elil17 | Hacker News Reader

It literally blocks "Asian restaurants near me." The fact that someone at Apple made a simple word based filter and didn't think of the possibility that words have different meanings in different contexts is absolutely astounding to me.

View on HN

You’re assuming a lot about the implementation. In fact I strongly doubt that somebody at Apple literally put “asian” as a blanket filter. It’s more likely an opaque machine learning feature gone awry.

You can get out of anything if you put machine learning in between the developers and customers

And what's extra depressing is that they're usually not lying. For nontrivial AI, the developers can neither sufficiently answer why it does x or why it doesn't do y.

They're working on it: https://venturebeat.com/2020/01/30/google-researchers-releas...

It's worth pointing out that, post-firing, the present tense doesn't really apply to Timnit Gebru...

Machine learning is almost magical in that way: complicated, contextual evaluations of things can be delegated to a black box, and since there's no longer a human people can get angry at for a contested decision, emotions aren't as high and the company can just shrug its shoulders at that silly little computer. It's a cheap way to abdicate moral and social responsibility.

It's one reason that AI ethics is a broader field than people worrying about the existential risk of AGI breakout.

Then it's on Apple for not validating and doing checks on the output of their ML models. Apple isn't some scrappy 5 person startup.

How would you comprehensively test for stuff like this?

A company doing a quarter of a trillion in revenue might be able to figure it out.

Take list of top 1000 search terms and run by human reviewers. Take list of top 1000 search terms that are flagged and run by human reviewers. Take list of 1000 sensitive terms (pay a few interns to do come up with them) and run by a human reviewer. Then review top false positives and random false positives.

Apple probably spent more money on the UI components for this feature than all of that would have cost.

Aren’t we supposed to be an industry that prides ourselves on tackling hard problems? We can’t build high tech pattern recognition systems and then just throw our hands up when there’s a false positive.

Does "asian" sound like an obscure term that needs "comprehensive" testing to discover?

Plus, how about partial rollout?

Well, having a list of a few million valid & invalid pages as a test case would probably help.

If the "machine learning feature" is acting as if it's a simple word-based filter, it's a simple word-based filter.

I wouldn't be surprised if they trained a model on text scraped from somewhere like pornhub and this was the result. It's only surprising they didn't accidentally catch more ethnic terms.

Does it also block "teen" and "lesbian"?

It does in fact block "teen", but not "lesbian".

I wonder why?

Because it's 2021. In 1990 or even 2000 it would have also blocked lesbian.

I'm implying that Apple likely looked for that term specifically and unblocked it, while they didn't care to check for the term Asian. But I wonder why?

I implied the same, and also kind of gave away the reason: because it's trendy for companies to pretend to care for LGBTQ now, and censoring it in 2021 would be a faux pas (Yes, Cook is himself gay, but that's not it, because he never was one for causes. Despite being ultra-powerful and rich, he only "went for the cause" for himself and Apple after it became Disney-safe to do so and every company did it - unlike the gay pioneers of the 60s to 90s who did it under really hostile environments).

It seems to block "ebony" as well, which is often used to refer to black people in adult content.

I did my own test to see and I could find one search term including the word Asian which was allowed (the search was "The History of Asian Nations," which is a book title). I think you're right, it must be more complex than a blocklist.

I tried over a hundred other terms (such as "Asian cuisine" and "college with best Asian American studies program") and none of them were allowed.

OTOH, if you sesrch app store for “sax”, it will show you results for “sex”, so I’m sure that balances it out.

It's possible the wordfilter is provided by some outside service and it wasn't inspected thoroughly.

OT but I have never heard of anyone saying Asian restaurant.

It's literally a category in Doordash.

If you're telling your friend about your dinner last night, you probably say "Chinese restaurant" but if you're trying to discover new restaurants through search you probably use "Asian restaurant".

Ever heard 'asian fusion restaurant'?

It's more than Chinese or Japanese cuisines. People might often want something asian to eat, but not decided on the particular yet...

Heck, my delivery app has "Asian" as a food category for searching...

Reply on news.ycombinator.com