Some notes:
- Socket.IO supports several different transports, including AJAX. To have persistent connections you need to make sure that you are using WebSockets (native or Flash) as your transport. Are you certain that you are actually using WebSockets? Socket.IO attempts to automatically pick the best transport and unless you verify, the only difference you will see is lower performance.
- You need to compare Socket.IO using AJAX as a transport, vs raw AJAX so that you can control for how much overhead the Socket.IO library (client and server) is adding to the test. Actually, comparing all the Socket.IO transport options would be useful. Doing a Socket.IO disconnect is not a good way to trigger a non-persistent transport in Socket.IO; you are likely causing both client and server to do a bunch of extra cleanup (Socket.IO has some advanced features that require tracking connections/channels/etc).
- If Socket.IO is using WebSockets as your transport, then your "Socket.IO non-persistent" test is still testing WebSockets, not AJAX or one of the other real non-persistent connections. I would not be surprised that WebSocket connection setup and teardown might be slightly slower than AJAX connection setup. HTTP connection setup has been around since the beginning and had a lot of time to be optimized. WebSockets are relatively new and connection setup time is not something that people will probably ever care about (WebSockets are not designed to be used this way).
- You should really use something other that Date() for your timers. The accuracy of Date is close to the latency of WebSockets themselves and so I strongly suspect that if you are doing this to localhost, what you are actually measuring in "Socket.IO persistent" vs "AJAX" is the Date rather than the connection latency Note that the latency per request in either case stays around 3ms and the Date object returns a value in milliseconds. The "Socket.IO non-persistent" is probably just a measure of Date with the overhead of Socket.IO cleanup. I suggest using high performance timers: http://updates.html5rocks.com/...
- You need to do your test between two different computers preferably with at least a switch in between. There are a lot of network stack shortcuts that happen on modern computers when you communicate over the localhost/loopback network.
- You are waiting 500ms between requests. I would suggest making that a variable. Interesting things will start to happen as you lower the interval. Also, as part of that, I think you probably want to put your start counter into the message data and then extract it on the other end to do the calculation. This is important because as you lower the amount of time you may start getting overlapping sends and receives (which is where things start getting interesting).
- You should really generate your random string outside the send block so that you aren't adding latency and CPU usage to the performance loop. Also, it would be useful to be able to tune the size of the data being sent to both larger and smaller sizes than 4K.
Finally, you mention in your lead off that one of the problems with WebSockets is that they are persistent so you may reach your limit of concurrent connections compared to AJAX. But in actual practice, with well designed and configured servers, you will reach server limits with AJAX requests FAR sooner than with WebSockets.
For example, consider an application that occasionally sends updates to a few clients and has 10,000 clients "connected".
AJAX is the worst case because every client has to establish a new connection to poll the server for data so to get sub-second latency for updates, the clients would need to make a new connection every 500ms or so. This means the server is having to setup 20,000 new HTTP connections every second.
Using long-poll (server hangs onto the client connection if there is no data) the client will probably have to re-establish a connection every 30 seconds (depending on server/firewall config/etc) to poll the server. This is much better, but that is still over 300 connections being established every second to the server (and only if they are evenly spread out). Also it's worth noting that with a long-poll scenario, you have most of the clients with established connections most of the time. So you'll run into connection limits just as quickly (actually sooner for various reasons) than with persistent connections.
With WebSocket connections, there is the initial connection setup to connect the clients, but once the clients are connected, if the server has no data to send, there is essentially NO traffic between client and server.
Looking forward to seeing your code and configuration. How about posting to github so that others can try and replicate your results and suggest improvements?