Conferences are all about coffee breaks. The coffee breaks are the conference.
Sitting around passively listening to lecturers is not a conference. It is a lecture, or as we call it these days, "live-action Vimeo".
Serving food and drinks to everyone at the same time is a way of keeping everyone together in the same space, where they can bump into each other, look over each other's shoulders, and convene spontaneous hallway meetings. That is what conferences are for. Otherwise we could just use Reddit.
In the end, you need catering because people at your conference want the catering. I spent hundreds of dollars per hour for the chance to stand with my colleagues in a space that's especially designed for meetings, but now I need to leave that space for several precious prime-time hours every day just to find food and eat it? Please, just sell me catering already!