> We have decided to organize these like departments similar to the way you might hire out humans. I'm not 100% sure if that's the best approach, but I will say it's been easier for people to understand because they're more mentally easily able to traverse the bot org chart if it somewhat reflects a traditional business org chart

I think this way of thinking is going to be very important to drive adoption of AI systems because the human analogies benefit from the pre-existing domain knowledge and expectations of people.

I like using the exam analogy for evals as a qualifier for work for your "AI hires" so you can trust them to work on a specific domain.

I'd be quite curious to see what your approach to evals/testing/tracing and agent/system mutation is.