Hacker News Reader

Comment by mgkimsal | Hacker News Reader

"... throw serious doubts on the popular myth that count() is slow in PostgreSQL."

Whew! Glad that was only a myth! I'll remind myself of that whenever my code takes > 10 seconds to return the count() from a table. "Just a myth - this isn't really happening". I'll try repeating that to myself while waiting for the results - probably should only take 10-15 repeats of the phrase before the row returns, right?

Whoah - just did a check on a real table with a whopping 2 million records - postgresql 9 just blazed through that in < 7 seconds to give me

select count(*) from student; count --------- 2032609 (1 row)

w00t! 7 seconds! I'm not even sure how that myth got started, let alone why people still believe that hogwash.

View on HN

http://wiki.postgresql.org/wiki/Slow_Counting

Are there DBMSs where this doesn't happen?

I just did a SELECT COUNT(*) on a table here in our QA environment; 20.5 seconds to count 42385875 records from one table and 68.8 seconds to count 191906711 records from another table (Oracle 10g 64-bit).

"The fact that multiple transactions can see different states of the data means that there can be no straightforward way for "COUNT(*)" to summarize data across the whole table; PostgreSQL must walk through all rows, in some sense."

When operating on a table, my understanding is that pg will select a version and operate on that version. If other versions are being worked on in transactions - that's a different story. Why can't metadata about the number of rows be assigned with the table version data, so after every operation, you'd know what the number of rows was at at that moment in time?

PostgreSQL does not track any versions at the table level, instead it tracks the versions for every row. This means two queries can modify different parts of the table concurrently without any lock contention.[1]

In PostgreSQL every row has two numbers. The transaction ID it was insert in and the transaction ID it was deleted in. An update is an insert plus a delete.[2] When running a select in PostgreSQL you just traverse the table and for each row check these two numbers to know if you are allowed to see the row.

The details above are PostgreSQL specific but most other databases have the same problem with there being no way to know the exact count without actually counting the rows.

Footnotes:

1. There is contention currently in PostgreSQL when writing the database journal (used for crash recovery and replication).

2. There are some optimization which are done here. For example HOT to avoid index updates.

I was unclear the part that is a myth is that they are slow in _PostgreSQL_. COUNT( ) is actually slow in general on databases with high concurrency requirements. Because you have to sacrifice something to make COUNT( ) fast.

If you had read his benchmark you would have seen COUNT( ) was slow in both databases (three times as slow in MySQL).

You can do a fast, conditionless count that ignores current transaction state and therefore risks being slightly out of date, but usually good enough for most use cases:

    select n_live_tup
    from pg_stat_user_tables
    where relname = 'mytable'

Way more people should be using this.

Reply on news.ycombinator.com