tl;dr: either go with someone you know and launch something together. if strangers: brainstorm, hack something together in 48 hours and thoroughly go through their background. if they've done a startup before, ask them to walk you through PG's 18 mistakes and identify which they've made (and their plan for doing things differently).
it's actually a two-sided problem. how do technical and non-technical co-founders who are potentially strangers and often have wildly different skill sets evaluate each other?
i asked a few hacker-founder friends for their take. one (well-known) serial CTO said: "I don’t have negotiation skills and I’m afraid of talking to business people. They play this "wordsmithing" game and I can’t keep up. Before I know it, they talk me into accepting less than I deserve and I get screwed.”
turns out, being taken advantage of was a common theme among the technical founders i talked to. most have been burned. in some cases it turned them off forming partnerships altogether.
what about the other side? i was a non-technical founder when i launched my first startup, which failed partly due my inability to judge a good from a bad developer. i eventually realized that being non-technical put me in a foolishly helpless position so i did two things: 1. i taught myself to code (i started with Pygames, moved to Project Euler and wound up working on small Python projects and learning ObjectiveC); 2. i got lucky when i met a developer at a hackathon last year. we became friends and i asked for his help interviewing technical co-founders. he wound up being interested in the problem i was trying to solve and contributed to the project. when i wound down that startup and told him about my next one, he wanted to get involved.
but not before suggesting i quickly hack together an MVP, get some initial user feedback and even land a few potential customers first. :> i'm so glad i did and he was smart to ask for this. he now understands how i approach problems, saw my ability to hack something quick and dirty, and witnessed some serious hustle as i did all of that in 48 hours. the point is we have some shared history now and the net result is trust. we also understand each other's strengths and weaknesses and talk about them openly.
(note: i also did a post-mortem on my previous startup and identified which of PG's 18 mistakes i'd made. it was a great exercise and i recommend it to anyone who has had to wind down a startup before launching their next. do it for yourself but share it with prospective co-founders and team mates.)