Ask HN: Options for distributed peer-to-peer image datasets?
Requirements:
#1 I need others to contribute personal photos and videos.
#2 Since the number and file sizes of relevant photos and videos is likely to be quite high, I think an architecture where photos/videos and their annotations are all stored in their own local machines is better than expecting people to upload GBs of data to some central location.
#3 I'd like contributors to always retain access control of their photos and videos - they can revoke subsets of their files from the dataset at any point.
#4 It'll probably also require creating an annotation solution that can distribute annotation tasks to volunteers, but the photos and videos still remain on local machines and only temporary copies with limited access may get uploaded centrally until annotated and then get deleted.
Questions:
Does some software to do all this - or some of it - already exist? If not, is something like IPFS a good enough storage solution for these requirements? Any other suggestions?
I'm not concerned about issues with distributed learning/compute for now. If the dataset is good enough, eventually some solution will probably emerge.