As I mentioned, I didn't eat at all yesterday and I asked the boss if I could leave earlier to eat. He said I couldn't and that "our schedule is our schedule" so I had to be there until 6pm.
This is abuse, clear and simple.
However, I still think that you need to communicate to him that the way he is behaving is deeply unprofessional as a colleague and completely unethical as another human being.
The 6pm issue is a kinda moot & one could ask all kinds of questions such as 'why didn't you eat earlier?' but I honestly think it's irrelevant.
They guy sounds fairly insensitive which sucks to deal with, but be careful not to twist everything he does just to fit that narrative.