Ports operate a different layer in the OSI model - the transport layer, not the network layer (which IP operates on). Ports are TCP and UDP concepts (e.g., a different application can bind to TCP port 443 than UDP port 443, as they're essentially different ports). Some protocols don't have ports at all, e.g., ICMP, which instead has a "type".