I think in principle, an application didn't need to know the exact format of an IP address, even if connecting directly to an IP. A simple idea that could have made application code much more IP-agnostic would have been for SOCK_ADDR_IN to take the IP in string format, not as a four-byte value. That way, lots of application code would not need to even be recompiled to move from a 4 byte IPv4 address to a 16 byte IPv6 address, whereas today they not only need to be recompiled, they need to be changed at the source level to use a new type that allows for both.
Of course, code that operates on packets, in the TCP/IP stack of the OS would have still needed to be rewritten. But that is far less code than "every application that opens a socket".
Of course, this only applies to code that uses IPs only to open connections. There's lots of application code that does more things with IPs, such as parsing, displaying, validating etc. All of this code would still need to be rewritten to accept IPv6 addresses (and its much more complex string representations), that part is inevitable.