Not all packets are created equal. ATM (Asynchronous Transfer Mode) cells have a fixed 48-byte payload, but IP (Internet Protocol) packets have longer headers and variable-length payloads up to about 1500 bytes. ATM is designed for low latency audio transfer, whereas IP incentivizes using bigger packets to reduce overhead at the expense of more latency.
Historically, sure. But at a realistic modern internet speed of, say, 15 MB/s, 1500 bytes takes 0.1ms to transfer. Even at 1.5 MB/s it takes 1ms. Typical VoIP call latency apparently ranges from 20ms to 200ms, so packet size is not a major contributor.