I'd say you should be counting the lines of code needed to implement TCP/IP – as well as DHCP, ARP, Ethernet, maybe DNS, and anything else I've forgotten that your server needs to speak. Consider lwIP, the most common implementation of those protocols on microcontrollers: it's meant to be minimalistic, but it's still some ~35,000 SLoC, not including tests. For comparison, a minimalistic implementation of TLS is Amazon's s2n, which is ~12,000 SLoC. Adding s2n to lwIP would only represent an increase in code size of about one third.
Of course, a more typical HTTP/HTTPS server will be using the Linux kernel for networking (with some userland bits for DHCP and DNS), and OpenSSL for TLS – overall representing multiple orders of magnitude more code. But I suspect the proportion between TLS and non-TLS would be somewhere in the same ballpark, depending on how you count.
The problem is that on Linux, the TCP/IP stack is in-kernel, universal for everything on the system, and overall unobtrusive – it mostly 'just works' – while OpenSSL is in userland, has an unwieldy API and unstable ABI, can be linked in many different ways, requires a lot of configuration, and is overall a pain in the neck. So as a developer, the complexity of TLS weighs heavily on you, while the complexity of TCP/IP is something you can largely forget about.
Personally, I think TLS adoption would have been quicker if it had been implemented in the kernel and it just became a switch you could flip in the socket API; something like:
socket(AF_INET, SOCK_STREAM, PF_TLS)
But for better or worse, that's not how things panned out.