(coming from the standpoint of a historically non-technical individual contributor who is leaning towards technical now, and asking because I want to learn):

But isn't that exactly why you as an interviewer would pose a problem, set up the context that is potentially similar to the work your project would require, and see how the interviewee navigates that?