Then just tell him exactly what you've been feeling. Describe things that he has said & how they made you feel.
There is a chance that he has absolutely no idea the effect he is having on you.
As I mentioned, I didn't eat at all yesterday and I asked the boss if I could leave earlier to eat. He said I couldn't and that "our schedule is our schedule" so I had to be there until 6pm.
This is abuse, clear and simple.
They guy sounds fairly insensitive which sucks to deal with, but be careful not to twist everything he does just to fit that narrative.
However, I still think that you need to communicate to him that the way he is behaving is deeply unprofessional as a colleague and completely unethical as another human being.
The 6pm issue is a kinda moot & one could ask all kinds of questions such as 'why didn't you eat earlier?' but I honestly think it's irrelevant.