The biggest risk of vision LLMs for OCR is that they might accidentally follow instructions is the text that they are meant to be processing.

(I asked Mistral if their OCR system was vulnerable to this and they said "should be robust, but curious to see if you find any fun examples" - https://twitter.com/simonw/status/1897713755741368434 and https://twitter.com/sophiamyang/status/1897719199595720722 )