Bounter: A Python counter that uses limited memory regardless of data size
github.com
github.com
>>> from bounter import bounter
>>> bounts = bounter(size_mb=1)
>>> bounts.update(str(i) for i in xrange(1000000))
>>> bounts['100']
0L
> Bounter implements approximative algorithms using optimized low-level C structures, to avoid the overhead of Python objects.
"approximative algorithms" isn't a commonly used phrase (at least according to Google) and this sentence doesn't say anything about what the actual trade-off is (loss of accuracy in the counts). A possible alternative would be writing to hard disk so what trade-off is made isn't obvious.
I also don't know what the precision and recall percentages in the example table mean.
further down, "Such memory vs. accuracy tradeoffs are sometimes desirable in NLP, where being able to handle very large collections is more important than whether an event occurs exactly 55,482x or 55,519x."
if you are on github, perhaps raise an issue to add an example similar to yours as well
I don't know about you, but a boolean flag completely changing the behavior of a collection definitely violates least surprise to me. They should be different classes with descriptive names, IMO.
They actually are completely different classes internally. The ‘bounter’ function is just a convenience wrapper (factory). For power users, the internal classes offer more control and parameters.
If you have any suggestions / concerns, please raise an issue on github. It’s a new library, we’re looking for feedback!
You can't really expect a data structure using a finite amount of memory to contain an infinite amount of data. Otherwise it wouldn't be advertised as a "counter", but as a magic bag of holding.
Chunking / iterative processing like this is an approach people commonly use with pandas which otherwise isn't great for data sets too big for RAM.
http://pandas-docs.github.io/pandas-docs-travis/io.html#iter...
Space Saving: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.94....
Frequent: https://stackoverflow.com/questions/3260653/algorithm-to-fin...
And there is a variant with a Sketch function, known as "Filtered Space Saving". I cannot find a working link to the paper, but here is a Golang implementation of it: