In Postgres, because it uses MVCC, every count requires verifying the table row in the data segment. This means a sequential scan over the contents of the table to verify the rows exist and are visible to the current session.

Few operations have this particular limitation, and it's been a sticking point in the Postgres world for years. Unfortunately there's no easy way around it.