Then I end up with a driver that conforms to emulated hardware and not it’s real counterpart
That's how unit tests works.
It seems like that's a concern with any testing strategy that mocks out some part of the system. Obviously there's no getting around actually testing against the hardware, but it seems like it could still be useful for the same reason tests with mock implementations are useful generally.