Hacker News Reader

Comment by stavros | Hacker News Reader

Does this relate to http://elasticlunr.com/? mdBook uses the latter, and I was wondering if there's a similar library for Python with index compatibility so I can provide my own search from them.

View on HN

If you want to build search in Python you can get a very long way using the full text search engine built into SQLite, which is available in the Python standard library.

I have some tools for using that here: https://sqlite-utils.datasette.io/en/stable/python-api.html#...

Thanks, but I didn't explain it well: I don't want to generally build search, I have a static site using mdBook and I want to use the mdBook indexes from Python to generate search results for that site.

Yes, it is a port of that library with some improvements.

You might want to consider mapping an index to an ets table-based data structure instead of an immutable object managed by a GenServer, it will give you a way to share it between processes without having to awkwardly copy a potentially huge data structure all over the place.

I do have thoughts about performance too but I was following the "get it working then make improvements" route :). Thank you for the suggestions.

This makes sense, and I think you've taken the correct route. I look forward to trying this in one of my projects and comparing to my current postgres-only backed search strategy. For my use case losing the index between restarts isn't a deal breaker, so hopefully I'll have some useful feedback.

That's great. I will be looking forward to this.

Love it. You're doing exactly the right thing.

I don't understand how this works. Is data read from ETS somehow shared more efficiently than data shared via a regular message? (which iirc is always copied)

It's still copied but if you are using an ets table you're likely only copying a small subset of the data per query instead of schlepping the whole index every time.

It’s still copied, but a process can quickly become a bottleneck in parallel code (every request to a process is sequential).

An ETS table can be concurrently read (and tweaked even further for that use case if desired).

Like eproxus mentioned, it's still been shared through normal process messaging but improvements will be made regarding this.

I'd say even using mnesia as an option for disc copies.

mnesia had very difficult to debug consistency issues that can crop up. Have these been fixed?

Reply on news.ycombinator.com