1. Design / architecture / product skills. Here I mean 'design' in the software sense: can you design a good API, can you structure a large project well? Do you have good enough product intuition to work well with a product manager and a designer?
2. Past experience. What have they shipped in the past? How did they deal with difficulties like conflict between teams, deadline misses, uncooperative team members, etc.