Saw this project the other day when looking through go resources, sounds like it's at least part-way to what you describe: https://github.com/photoprism/photoprism