You are correct. Humans will be the weakest link in this security link, specially at the origin. Bitcoin etc solve this by using PoW. But here there can be issue if there is a malicious first entry

That said, a case might be made for a traceable and immutable book which shows the movement of each diamond. But that might run into privacy issues. This can be solved by using a private blockchain but then again it goes back to trusting someone who has the blockchain with them.