Hacker News Reader

Comment by libria | Hacker News Reader

Wow, does that mean we finally get full text search (i.e., LIKE '%%')? Bonus points for OR, JOIN's and multiple inequality filters.

Since they forbid these operations on the datastore, I'm guessing this will either not scale as well or be significantly more expensive. Since the sign up sheet implies you can combine technologies, I suppose you can use a "hybrid" approach and keep denormalized data in the datastore.

View on HN

The Google App Engine team has been working on a full-text-search service for GAE projects. They gave a demo at Google I/O back in May. Here's a video of their presentation:

http://www.youtube.com/watch?v=7B7FyU9wW8Y

http://www.google.com/events/io/2011/sessions/full-text-sear...

http://static.googleusercontent.com/external_content/untrust...

I believe this is available in both the Python and Java dev environments.

Additionally, http://code.google.com/appengine/docs/python/prospectivesear... might be of interest.

Yep, it currently limits you to a 10G database size. Makes sense, though - SQL techniques like LIKE filters or joins really do require all of your data to be in memory with a very high-bandwidth, low latency interconnect - ie, it all has to be in memory on a single box.

This is the reason the datastore forbids these operations - because they're extremely difficult to efficiently implement and still scale indefinitely (without making other, potentially very large sacrifices).

What? A join requires all of your data to be in memory? I've sure as hell seen databases that executed queries with joins and the data was not entirely in memory...

Well, sure, it doesn't _require_ it. But the alternative is higher query latency. You have to collect a filtered set of the join column from table A, then ship them over to the servers responsible for table B. If your interconnect has high latency or low bandwidth, this can be painful - particularly if the intermediate set of keys is very large.

Hence why the datastore simply disallows this - yes, it's _possible_ to make joins work on larger datasets. But it's not possible to make _arbitrary_ joins work well on larger datasets.

Joins on indexed columns don't require anything apart from the index to be in memory in order to do the filtering.

Or preferably, Sphinx support. From the announcement I can't see that you can't do it, but not that you can.

InnoDB and MyISAM are not good at full text search even on local disk, no reason to make that worse.

Somewhat kinda germane post from Percona: http://www.mysqlperformanceblog.com/2009/09/10/what-to-do-wi...

Reply on news.ycombinator.com