Thing that burned me was forgetting about page cache and THP, two runs of the same binary ten minutes apart differed by 15% until I dropped caches and set khugepaged off. Also worth reporting variance across reboots, not just across runs.
Having built something similar with CLIP on an M1, frame sampling rate is the whole ballgame. One frame a second on 12k videos is days, keyframes only got me to an overnight run.
Agreeing on the model gap, the bit left out is that even human-to-human it breaks. Same drug, same dose, wildly different response depending on genotype and gut flora.