Concerns have to be decoupled as mush as possible. It is even more likely that the user will use some serialization, and there is no point at all to implement serialization inside the library! Because then, also it is likely that the user want to make some CRCs to check message integrity, or cypher de message so lets embed such funcionality in the library too :)
I have developed and used code many times using a serial port (robotics, industry), and in many occasions such extra async framework or the streams are not used. I know they are quite convenient and useful for many many cases, but it is still a separate concern than implementing the low level "hardware" comunication, and thus deserve decoupling it its own component, exactly the same as serialization. In that way, when using Async for network sockets, the SerialPortHW concept wouldnt be involved at all, it would be in a different concept not included in the current project.