To each their own but if I was interviewing someone who was already talking about how they really want to go work somewhere else - I'd say exactly what he said. Go apply there.
Bringing someone into a sustainable business is an investment. They aren't going to show up and be a rock star on day 1, it doesn't work that way. They are going to show up and have to learn how things work at that company and figure out how to fit in and make a difference in that context. That takes time. Learn the code, learn the process, learn the people, learn the customers. Who does that in a day? Nobody. A new employee is maybe useful in a year or so in the world I know. As a hiring manager I know I'm going to use up a year or so of salary before I get any return on my investment. I don't expect people to stay around forever but I want them to stay around long enough that it works out for everyone. This guy didn't sound like a good investment.
Just my opinion but I'm running an 18 year old company and my people have been here for at least 8 years, that may have skewed my view of things.