At least historically, the definition of "byte" has been "the smallest addressable unit" which on most of today's machines is 8 bits, but not all (I believe the SHARC floating point DSPs are still 32-bit bytes, and there are probably more). Using bits for things makes sense from this perspective as the number of bytes would depend on which processor it is connected to.

I know that these days people consider that a byte is 8 bits by definition, but I think that probably isn't a great linguistic move (we already have a word for that - octet)