1) I typically go to lunch with my friends/coworkers and talk about projects we're working on together, and usually we come up with some deeper insight than we have together in meetings. I'd miss this.
2) My lunch time is my personal time during the day where I unwind from the morning's activities and take a break. If my brain has to "be on" for a deep conversation, I wouldn't get much of a break and might a) get tired, b) get sloppy, c) lose motivation later in the day.
3) Lunch is what, $20 tops? Closer to $10? My time is worth more than $10-20 for an hour's worth of conversation about someone's project. Say I have some interesting idea/insight/solution and we talk about it, and that idea/insight/solution ends up being really important to them. I ended up getting free lunch, they could end up making a bunch of money. The scales are not even.
I'm fully aware that other people (maybe most people?) would be excited about getting free lunch, so maybe I'm an outlier, but I thought I'd drop my two cents in.