Hacker News Reader

Hacker News

Comment by syphilis2 | Hacker News Reader

Full thread

syphilis2

"The goal of the project is to make text hard to read for computers yet fairly easy to read for humans"

At first look it doesn't pass the search engine test. https://duckduckgo.com/?q=I%27ve+bien+aloune+whyth+jou+eensi...

View on HN

Obscuring "I've" sneaks Lionel past the Duck.

https://duckduckgo.com/?q=eyeve+bien+aloune+whyth+jou+eensid...

Check the 3rd result, it's the lyrics.

Sorry this might be OT, but I found it interesting that the 3rd result for you is the lyrics and for me it is something completely different (Lionel is just the tenth result). I thought Duckduckgo returns the same results for everyone and that the results are not personalized. Does someone know what kind of personalization does Duckduckgo use? In settings, region and language can be selected, but I have it set to 'All Region' and 'Browser preferred' which is en-US. Does Duckduckgo after all track me and this way customizes the results? Or am I missing something here?

It's probably just the region/language settings.

Set region to Austria, and Ritchie is result nr 1. set Region to "All Regions" and he's result nr 10.

That seems reassuring. I think I have panicked too soon.

Interesting. Yes, number 10 for me (All regions; Browser preferred lang.) The Instant Answer (IA) is lyrics to "Inside Out" by Eve 6.

I do get a result for lyrics at number 10. I meant to imply Lionel was wearing a hat and dark sunglasses to sneak by the Duck with only one entry of the first 40 results. Not so bad compared to the original query! If only he had removed the fake mustache...

I suspect there is some part of their algorithm using levenshtein distance behind the scenes (or something similar). Spellfucker would not help with this.

actually it does deal with this particular metric somewhat in a bunch of cases.

e.g. replacing one letter with three that sound similar but are different adds 4 to the distance - which is quite a significant distance already.

So perhaps an algorithm that combines soundex with Levenshtein distance would handle this?

That's actually very impressive!

Actually, it fails on Google. DuckDuckGo just apparently has an excellent spelling correction system.

i think it uses bing's spell correction.

Good test! No this is one way: you know the obfuscated.

What if you want to use it in the opposite direction? Let's call it the CIA/NSA direction, where you know what unobfuscated phrase you want to find, but not how someone may have obfuscated it. This is arguably much harder, especially if you do not store some sort of representation of the content "as sounds/phonemes".

The point is that if you are the CIA, searching through a bunch of obfuscated text, just like you are already (hopefully) already compensating for basic spelling errors, it apparently would not be very difficult to expand the scope of that same mechanism (in the same way that Duck Duck Go has) to support searching through and understanding this obfuscated text.

Absolutely.

Something I've had in the back of my mind for a while is the idea of swapping out a regular Lucene tokenisation & analysis for one that treats phonemes as tokens instead of (stemmed/analysed) words being the tokens... with a similar arrangement on the query side. I think it'd have interesting capabilities... this being one.

"The goal of the project is to make text hard to read for computers yet fairly easy to read for humans"

What's the advantage to this?

Maybe provide a challenge for NLP researchers?

Spam comes to mind.

I don't think anybody would have the patience for a spam message that had gone through this.

Defeating censorship, frustrating accounts to analyze or index communications, etc.

Spamming people and making it impossible for blind people to use the internet?

Reply on news.ycombinator.com