The physical object as I understand it provides a source of random data, with the property that it is fast to do a single lookup (shine an arbitrary light pattern on it), but slow to copy the whole data.
This means that an attacker who steals the object, but doesn't know which lookups will be done, won't be able to copy all the data.
Now my objection is that the only lookups that the person who steals the object will not be able to do, is the ones based on random patterns that have not been published, i.e. the ones that are generated next time Alice and Bob meet.
But from that point of view, it would be no less safe if Alice and Bob simply generated a one time pad and stored it when they met.