Do hash tables work in constant time?
daniel-lemire.com
daniel-lemire.com
He asserts hashtables are not amortized O(1). Ok, interesting, maybe something with caching? latency of disk access in huge data stores? No. He concludes hashtables are not O(1), because integer multiplication is not O(1). No mention of putting an upper bound on the hash key size, like the size addressable memory, or electrons in the universe, just that it's not O(1)
Lucky for us, he's backed it up with a clever and insightful lack of experiments.
edit spelling and grammar.
"Am I being pedantic? Does the time required to multiply integers on modern machine depend on the size of the integers? It certainly does if you are using vectorization. And vectorization is used in commercial databases!"
Sorry, I did not run the experiments this time around.
I'm a head in the clouds kind of guy, i'd have been fine if you'd said, here's all this extra stuff you might be overlooking. But when you frame the discussion with concrete measurements... I go looking for concrete measurements.
That said, I broke the rule on not saying something in writing, that i wouldn't say face to face. (well, i would, but only after i knew you better)
I apologize for being snarky. That was uncalled for.
foo.c:6: note: not vectorized: unsupported use in stmt. foo.c:11: note: LOOP VECTORIZED. foo.c:2: note: vectorized 1 loops in function. 0 real 0m0.018s user 0m0.017s sys 0m0.001s
int main () { int a[256], b[256], c[256]; int count = 100000; int i; for(i=0; i < 256; i++) { a[i] = i; b[i] = 255 - i; } do{ for(i=0; i < 256; i++) { c[i] = a[i] * b[i]; } } while(count--); printf("%i", c[0]);// avoid c being optimized away. }
so, i'd assert that vectorized multiplication is a speed win rather than a loss. upping the count a few orders of magnitude indicates vectorized multiplication grows slower than regular multiplication.
So, I claim the vectorization of multiplication is probably not detrimental to the Big O of a hashtable.
ymmv.
From the other direction, if a hashtable implementation uses arbitrarily sized numbers to compute hashes, it probably has bad design.
There is always a maximum number of native values that can be operated on at one time by the processor. The maximum cycle costs are also known. Between the two of those, we have a maximum time spent on the computation, and it is a constant.
Considering the size of numbers during algorithm analysis matters when the numbers are huge - that is, when the numbers are larger than the maximum values that can be represented natively by the processor. When that happens, "numbers" become a data structure unto themselves and must be analyzed as such.
Hashtables derive their efficiency from the number of keys that you can encode for a given size hash. If the collision chance is low the number of 'chained' items in each hash bucket approaches one.
To take the length of the key and the number of bits in it as a factor in the complexity of the hash table shifts the problem to a sub-problem that has nothing to do with the original one. As far as the hash table algorithm is concerned that is a different algorithm, whose fixed cost per item is taken into account.
Sure, it is slower to compute a longer hash. But that is not where the majority of the savings are. The majority of the savings are in not having to traverse the N items in the keyspace to find a given key.
Constant factors - and this is one of them - are simply left out because in the greater scheme of things they are but a very small item. Any constant factor, be it 5, 10 or 50 as viewed from the far side of the code calling the hash store is simply that, a constant.
You might be able to optimize on it by using a clever algorithm, you might be able to work it down all the way to '1'.
But that will not change the complexity at all.
This is the same reason that "radix sort" (http://en.wikipedia.org/wiki/Radix_sort) isn't a linear operation even though it doesn't seem to have a log(N) dependence on the input set. The number of radix buckets scales as log(N).
That said, of course, this doesn't mean much in practice, where primitive operations on keys live in L1 cache and the pointer indirections require DRAM cycles.
This is true, and it is applicable to other concepts besides hash tables. Can you sort n numbers in O(n log n)? Well a comparison is not really constant-time, in the wall-clock sense of time. If the n numbers are distinct, then each is going to need at least O(log n) bits. So a comparison is O(log n) by the clock, and our sort becomes O(n log n log n).
Part of the solution is to be more precise. For example, when we say that Merge Sort is O(n log n), we usually do not mean that it takes that long to run; we mean that it can require that many comparisons. Counting primitive operations can be useful, even if those operations are not constant-time from a wall-clock point of view. And as long as we are clear on what the primitive operations are, everything is fine.
By the way, there is another problem with the statement that hash-table insertion is amortized constant-time: what if all the keys end up being in the same bucket? The true statement is that HT insertion is amortized constant-time for average data. (So it's constant-time on average on average. :-)
It may take longer to multiply 128 bit data structures than 64 bit data structures, but it is constant with respect to the size of the hash table.
That's all somewhat theoretical sure, but the point is to challenge your assumptions. (People do use vectorization, right now.)
You're assuming that people are not aware that multiplication is not always a constant-time operation. That assumption generally doesn't hold among people with any sort of academic CS background. Multiplication algorithms - and the fact that they were typically O(N) in number of bits - were covered in my intro machine architectures course.
In practice, there are many, many other things that can negatively impact hash table performance in a much bigger way. Like cache misses - one cache miss costs far more than an integer multiplication. Or interpreter overhead from using, say, Ruby instead of C (bad example, since Ruby's hashtables are implemented in C, but the general point holds). Or the difference between -O0 and -O3. Or how early versions of Java Hashtables would often devolve into linked lists because the hash function only looked at the first 8 characters of a string.
"You're assuming that people are not aware that multiplication is not always a constant-time operation."
I made no such assumption. But I suggest people consider the case of vectorization where, all of a sudden, it may matter quite a bit.
Exactly. That is the whole key to complexity analysis.
Otherwise you end up drowning in details without getting more meaningful answers.
You're analyzing the hash table algorithm, not the hashing algorithm used to produce the keys.
I don't see that I ever did use it for a number of bits. I said, to have n numbers with distinct values, you need O(log n) bits in each number. n is not a number of bits.
In any event, he was correct when he said: "The problems that result from picking the wrong model of computation are well known and addressed by most textbooks. I have not discovered anything new." There are problems that you can "solve in polynomial time" by encoding the problem as some HUGE number (where the number of bits is exponential in the size of the problem) and doing arithmetic. This is where you really see the breakdown of the constant time arithmetic model.
Yes, this guy is confused about Big-O notation — nothing to see here, please move along :)
m- the size of the hash table m- the size of each element in the hash table
The 2nd m should be a different variable, and it is constant with respect to the first m, which is the one we care about.