Neural nets are known to struggle with tabular data. Have you tried fine tuning or attaching a decoder somewhere that you train on your task? Zero-shot inference might be asking for too much.
Not disagreeing with you, and I'm not a specialist, but it's funny that lot of papers seem to claim exactly the opposite.