The normal way to "work directly at IP level" (or even below) is to use a RAW socket. The kernel will deliver all traffic matching the IP (or other things - raw sockets can essentially own the whole NIC) to your socket, and from there it's all up to you. The kernel will only look at the IP headers, the rest of the packet (TCP/UDP headers specifying ports, for example, if any) will be delivered to the raw socket.
Not sure of the implementation details, but in principle the same logic about specificity of socket binding could be done by the kernel when deciding which socket to actually send the data to. Any other logic (CRC validation, packet ordering, confirming packets have been received, MSS etc.) are all disabled, you just get the raw packet as it was received on the wire, based on the IP headers (an the same happens when you send a packet on a raw socket).