There are plenty of difficulties hiding in the OSI stack. MTU is a fun one.
Even if you map Internet protocols to the OSI model -- which is imprecise at best -- TCP represents the transport layer, and IP the network layer. They're not a single component.