Building a full-text search engine in 150 lines of Python code (2021)
bart.degoe.de
bart.degoe.de
https://gist.github.com/benob/69d48421f88f5dcc2b26a204d3251d...
Most of the difficulty in search is dealing with the sheer volume of data. The algorithms themselves are pretty trivial for the most part.
I love that kind of bullshit poetry.
pure (n, guard (factor /= n) $> factor)
Which I think is more or less the same as this python line. return [factor for factor in factors if factor and not factor == n]
The article does fancy stuff with memory caches which I believe is easy to do in python but I need to understand the Haskell code better.[0] Haskell: A Great Procedural Language https://entropicthoughts.com/haskell-procedural-programming#... https://news.ycombinator.com/item?id=42754098
Returns a tuple: Left side is n, right side is (Just factor) if factor is not n, or Nothing if it is.
A more direct translation:
# py
[token for token in tokens if token]
-- hs
filter (not . null) tokens
The full functions: # py
def analyze(text):
tokens = tokenize(text)
tokens = lowercase_filter(tokens)
tokens = stopword_filter(tokens)
tokens = stem_filter(tokens)
return [token for token in tokens if token]
-- hs
analyze :: String -> [String]
analyze = filter (not . null) . stem_filter . stopword_filter . lowercase_filter . tokenizeIf you'll humor me, I'd love to take another shot, but from the more interesting direction!
Adding a type might clear stuff up a bit:
-- hs
pure (n, guard (factor /= n) $> factor) :: IO (Int, Maybe Int)
# py
print((n, if factor == n then factor else None))
So without the for loop, there's no list to comprehend. Adding the list back gives something more equivalent to a standard for loop in Python: -- hs
for numbers $ \n ->
pure (n, guard (factor /= n) $> factor) :: IO [(Int, Maybe Int)]
# py
for n in numbers:
print((n, if factor == n then factor else None))
Which could also be written as a list comprehension in either language: -- hs
[pure (n, guard (factor /= n) $> factor) | n <- numbers] :: IO [(Int, Maybe Int)]
# py
[print((n, if factor == n then factor else None)) for n in numbers]
Note that they behave a bit differently though, and it's the reason I haven't included `return` in the python line (what does `print` return?). Python will loop through the list and actually run the `print` function on each element, while Haskell will loop through and collect all the `IO` into one function to run later. Although it's starting to get pretty un-Pythonic, you can hack the behavior into Python with something like: # py
return lambda: [f for f in (print((n, if factor == n then factor else None)) for n in numbers)]
Which would need to be run using something unholy like: # py
factorize(numbers)()Or you can do,
return filter(None,tokens)
Not obvious, but giving None is like giving "lambda x : x" to filter().https://docs.python.org/3/library/functions.html#filter
> If function is None, the identity function is assumed, that is, all elements of iterable that are false are removed.
Eg, arr.filter(Boolean)
If not, I'm now curious why the `None` special case was added to `filter`..
While this doesn't explain why filter accepts None as the lambda, I think the fact that there are no builtin identity function kind of explains why. It's easier to write None rather than put an identity lambda. I reckon it also makes it easier for a compiler to optimize this specific case.