return [token for token in tokens if token]
What the what? return [token for token in tokens if token]
What the what?It is iterating over a list (tokens) and creating a temporary variable (token). It tests the truthyness of it (if token), which means None and '' will return False and thus be excluded, and then returns it (the first token).
list(filter(None, tokens))
Or list(filter(lambda x: bool(x), tokens))