Storing in EXIF seemed also the go-to solution for me.
Until I ran into deduplication and backup management problems due to the changed files.
I've solved that by materializing every photo to a deterministic folder path. So it's not possible to have the same exact photo in my library.