word_bag: dict[str, int] = dict() # Multiset
for line in raw_corpus:
words = line.split()
for word in words:
if word in word_bag:
word_bag[word] += 1
else:
word_bag[word] = 1
Try: word_bag = collections.Counter()
for line in raw_corpus:
word_bag.update(line.split())
then, to filter: word_bag = {word: count for (word, count) in word_bag.items()
if count >= min_frequency}
or, if you can avoid printing the filtered word bag: return set(word for (word, count) in word_bag.items()
if count >= min_frequency)
Also, that: raw_corpus : list[list[str]],
should be list[str]. It's your corpus which is a list[list[str]]. (And you might want to use "line_words" instead of "line"Could you explain why lines are processed independently?
for i, line in enumerate(corpus):
for j, target_word in enumerate(line):
Does it make a difference that some lines have fewer words than others?I would think that
for positive_words in sliding_window(flatten(corpus), window_size):
using the sliding_window and flatting recipes at https://docs.python.org/3/library/itertools.html#itertools-r... would be more appropriate. Though using it directly would not handle the leading/trailing partial windows.I liked seeing the 'η' - I haven't yet used non-ASCII in my code!