Most unix filesystems use inodes. inodes have the same format whether they represent a file or directory. So directory have a (usually unused) size member. NTFS doesn't use inodes, but the records in the MFT work the same way.
When adding or removing a file from a directory, you have to update the modification time of the folder, so you have to rewrite the entire inode anyway. Updating the size/file count at the same time would be free.
In my opinion the likely reason why file count isn't tracked is a lot more pedestrian: It wasn't tracked initially and we can never add it to existing file systems because the metadata would get out of sync if the FS was mounted on a kernel with no count support.
I base this assumption on the fact that many modern file systems do indeed keep track of the count.