IMO, the first layer could be a proper abstraction of the basic OS serial port functionality, without any notion of sync/async issues. Well, it will typically be sync. Keep it simple, it could be maybe just 2-6 files, depending on the approach to tackle different OSs. A common interface for open, config (baud, and the like), send, receive bytes, close. Lets call it SerialPortHW
Then, sync and async streams can be defined, or at least their interfaces and common functionality, which would be independent on any notion of physical transportation as SerialPortHW. Les call it AsyncStreams. In no way SerialPortHW has to depend on AsyncStreams (SerialPortHW->AsyncStreams), and the opposite is the same. They do not read logical, the concept "SerialPortHW" does not have implicit a requirement to know anything about async issues. If a function is slow to send bytes over the wire, it is slow, so what. It is fully functional, well defined, stand-alone and independent functionality that deserves its own entity.
Then, if the async serial port funcionality is desired, another component, lets call it AsyncSerialPort can be built composing/deriving AsyncStreams and SerialPortHW.
Sorry for the long response, I love arguing about SW design :)