VoIP is definitely not the issue here. Codecs can be, and are, fast. I have no idea what cellular providers are doing to mangle the voice path so bad, but it certainly isn’t inherent to VoIP.
I have no idea what they are doing but it sucks. I have in many occasions found that making a call using WhatsApp works much better - both in quality and latency - than making a phone call. Phone lines are so atrociously bad, they've put me off from doing a voice call unless it's absolutely necessary.
I thought the delay would be because VoIP uses packet-based communications rather than a direct connection.
Most landline networks are also digital now for exchange to exchange communications, so even those will be packet based
Not all packets are created equal. ATM (Asynchronous Transfer Mode) cells have a fixed 48-byte payload, but IP (Internet Protocol) packets have longer headers and variable-length payloads up to about 1500 bytes. ATM is designed for low latency audio transfer, whereas IP incentivizes using bigger packets to reduce overhead at the expense of more latency.
Historically, sure. But at a realistic modern internet speed of, say, 15 MB/s, 1500 bytes takes 0.1ms to transfer. Even at 1.5 MB/s it takes 1ms. Typical VoIP call latency apparently ranges from 20ms to 200ms, so packet size is not a major contributor.