Part of the problem is that IPsec deals in authenticating peers and authorizing them to the IP addresses they sport, but IP address numbering is fluid and difficult to tie to authentication or authorization databases. All too often sites permit large prefixes to all clients with certificates from some CA.
Another part of the problem is that IPsec protects IP, not TCP and friends. This means that part of a TCP connection's packet flows may be protected by SAs with one peer, and later by SAs with... a different peer -- this may sound strange, but given the issues w/ authorizing IP addresses (see above) this is very much possible.
The right answer is to tie the upper level protocols (e.g., TCP) to IPsec policy automatically so that for the entire lifetime of a connection the local and peer IDs are "latched". This has been specified [0] (disclaimer: I'm the author), but not implemented. API-wise this would manifest as socket options you could set and/or get to specify or learn local credentials, peer name, quality of protection, etc.