Show HN: We made a hiring challenge because Claude can 1-shot our interviews | Hacker News Reader

Show HN: We made a hiring challenge because Claude can 1-shot our interviews

atomsnotelectrons.com

4 pointsjgru4 comments

We're Tutor Intelligence, a robotics company building generally capable robot workers for American industry. We've been thinking about what technical evaluation should look like in a world where AI agents can 1-shot our hardest hour-long coding interviews, and this is one of our first experiments.

The challenge: command 5 robots in a 60x40 warehouse to fulfill 1000 orders. Your score is the number of timesteps to complete everything. Robots can move, pick items from pallets, dock to pallets (so they move together), and fulfill orders at the edge of the warehouse. Simple rules, but the optimization problem has surprising depth (it's NP-hard in about 10 different ways) with lots of room for strategy and creativity.

This is actually a simplified version of a real problem we work on. Warehouse coordination is one of those domains where the gap between a naive solution and a good one is enormous, and there are many valid approaches.

We built a web visualizer so you can see your solution play out, and a leaderboard if you want to submit. AI agent use is encouraged (probably necessary). So far only my cofounder and I have submitted, so we genuinely have no idea how good solutions can get.

Sharing this early because we'd love feedback on the problem design. And yes, we're hiring (that's why we made it): 70 people, Series A, based in Boston, founded out of MIT. But mostly just curious if others find this problem as interesting as we do.

Open articleView on HN

When I read the title I thought you made a challenge Claude couldn't solve but that's not what you are doing. You are taking a pragmatic approach to the world we live in. I like it.

- It would be good to put what you are planning to learn from this interview process. - Looks like submission is only a text file. Why not ask for chat transcript? - Also, would be useful to let people know what happens after submission/selection.

Submission actually takes you to a full featured visualizer! So you can go back and forth on it and see your robots run. You can then choose to submit to the leaderboard if you'd like (otherwise nothing goes to our servers) which will collect your email. No plans to do anything with those yet besides serve the leaderboard.

Hoping to get a better understanding of whether success on this correlates with things that we think are important for AI-enabled software engineering success. I think this is largely a question of the problem depth, and how much does a solution still need to be driven by that person's creativity, vs the model suggesting the next obvious idea.

Sounds like a take-home challenge, but with a twist that you actually expect people to use AI, so that AI is not "cheating"?

Are you worried about the usual take-home challenge problem of user getting outside help? Either friends solving the problem for them, or paid help doing the same?

Our hope is that beyond being a take-home challenge people can get competitive and it can turn into more of a passion project. I think the problem has sufficient depth that this could be the case! Hence the leaderboard and visualization tooling.

Around cheating, we've never been too worried about this. The real cost is wasted time later down the pipeline, but we can always tell then whether the person lines up with the work in the takehome.

Reply on news.ycombinator.com