Another reason for using OpenSSL was to eventually get the server component running on unix-based OS. I should have explained that in post because it does seem odd using OpenSSL on Windows when I could just as easily use Crypto API.
"I still don't understand the authentication model here"
Sorry if I misunderstand but If by that you mean the way in which packets are transported between 2 systems;Length first, then data. I just wanted a really lightweight method of transmitting data so that it would be easier to implement in assembly rather than using TLS, SSH or some other well established protocol that would require lots of code. It's possible to use some really lightweight implementation of TLS but I assume the code would grow significantly.
There is support for SSL + TLS through SChannel and I probably will write something with this in the future just to see how it works out.
If you mean I could simply embed whatever keys are required inside the payload and use those for encryption? I thought with some form of key exchange, it would make the traffic more difficult to analyze.
If session keys were inside payload then it's just a simple matter of extracting keys from this and decrypting traffic.
I looked at poly1305 for authentication briefly but must test it out, it does look more compact than using AES + SHA3.
Until you mentioned here, I wasn't aware truncated SHA2 hashes were immune to length extension attack but even without the HMAC code, SHA2 still generates more code than SHA3.