437 karma · joined October 10, 2006
The word vectors have been aligned in multiple languages. Using an approximate nearest neighbor search we are able to find the nearest vector to the input in multiple languages very quickly.
To keep the example simple, we did not try to filter the data through hand-built language dictionaries. In fact, we simply drop words in other languages that also appear in the English .vec file. Words like "ciao" appear frequently enough in otherwise English sentences that the example code drops it from Italian, and so is not shown in the results:
% curl -s "https://dl.fbaipublicfiles.com/fasttext/vectors-aligned/wiki..." | grep -n ciao 50393:ciao 0.0120 ...
One improvement would be to filter out any words that do not appear in a hand-curated dictionary instead of filtering out words that already appear in English. We decided not to show how to do this because we'd already introduced a few concepts, like aligned word vectors, approximate nearest neighbour searches, and wanted to keep the example as simple as possible.
We show results quickly because we check if your search is in the zone file. We keep a copy of the zone file in memory, and check searches against that. Some names are "unconfigured" for a variety of reasons, and do not appear in the zone file. So this means we might show a name as available, but when you go to register it, it's gone. This sometimes leads people to believe that we've registered it for ourselves. If you check the WHOIS record, you'll usually find that it's been registered for years before you searched for it.
The only major registrar that I know of that did this was Network Solutions. They did it for a few days in 2008 before getting trashed in the press:
https://arstechnica.com/uncategorized/2008/01/network-soluti...
More recently, when we show a name as available, we'll check again with VeriSign (who runs the .com registry) to make sure. And if it's not available, we'll switch the color from green to red -- a good example is eager.com. It's not in the zone, but not available to register.
It uses the streaming API, and will work well for you.