When you asked about the filesystem, I assumed you were asking about which filesystem feature was being used, since hyperspace itself is not provided by the filesystem.
Someone else mentioned[0] fclones, which can do this task of finding and replacing duplicates with reflinks on more than just macOS, if you were looking for a userspace tool.
You only get CoW on APFS if you copy a file with certain APIs or tools.
If you have a program that does it manually, you copied a duplicate to somewhere on your desk from some other source, or your files already existed on the file system when you converted to APFS because you’ve been carrying them for a long time then you’d have duplicates.
APFS doesn’t look for duplicates at any point. It just keeps track of those that it knows are duplicates because of copy operations.