Basically the drive is saying "yup, it's all on NAND - not in some internal buffer. You can power off or whatever you want, nothing will be lost".
Some drives are doing work in response to that FLUSH but still lose data on power loss.
That's a very important distinction. You can't assume just because a write completed before the flush that it's actually durable. Only if it completed before you sent the flush.
I'm not very confident that software is actually getting this right all that often, although it probably is in this fsync test.
Or what if the device issues a pci write to the completion entry that passes a flush command being submitted on the wire?
I think the only interpretation that makes sense is from the perspective of a single software thread. If that particular thread has seen the completion via any mechanism and then that thread issues the flush, then you know the write is durable. Other than that, the device makes no promises.