Pretty awesome concept! Could each of the objects hold there annotations locally, and let other objects query them? Or would there have to be be a central server?
The idea was to have a central database, otherwise it would be impossible to do any kind of computation/search on it. App.net might be a candidate for something like this.