The connection overhead is removed but that is only one of the various overheads that exist.
edit: I understand your question a bit differently after reflecting on it more. I think you may be saying that 1 connection never ends in long-polling. This is not the case and is only this way if keep-alive is used.
Each server->client data exchange happens by "completing" the request and the client is configured to immediately make a new request. This means every message (or groups of messages depending on your tolerances) is a completed HTTP request.
What you're thinking of, an HTTP response that the server keeps appending to and the client is expected to read and process incrementally, is also a real thing though. This is typically referred to as HTTP streaming. Examples of this would be SSE (text events) or MP3 streaming (audio).
(Not to be confused with HTTP Live Streaming (HLS) which is typically not based on any long lived requests at all!)
The client can either terminate the connection and make a new one, or make a second parallel request for sending and leave the first waiting for a new server response. I'm actually not sure which is more common (despite implementing apps that can do long polling, I've never paid attention at that level), nor am I entirely sure how this works with keep-alive connections (eg, can you stop waiting and issue a new http request without closing the TCP session?).
In all cases, each request from the client involves sending http headers, and if the client is a web browser, there's going to always be a few hundred bytes at least (cookie, referer, accept, user-agent, etc).
Here more info: https://stackoverflow.com/questions/12555043/my-understandin...