Hi @RoiTabach, OP here
Yep. We can integrate with every solution that supports OpenTelemetry :) so it's pretty native, just use the integration skill
npx skills add kelet-ai/skills
18 karma · joined December 20, 2013
Yep. We can integrate with every solution that supports OpenTelemetry :) so it's pretty native, just use the integration skill
npx skills add kelet-ai/skills
langsmith/langfuse/braintrust collect traces, and then YOU need to look at them and analyze them(error analysis/RCA).
Kelet do that for you :)
Does that make any sense? If not, please tell me, I'm still trying to figure out how to explain that, lol.
YEP, Good catch! Kelet as input/prompt in Hebrew :)
From what we discovered analyzing ~33K+ sessions, most of the time when the agent selects the wrong tool, it's because the tool's description (i.e., prompt) was not good enough, or there's a missing nuance here.
That goes exactly under Kelet's scope :)
Great points, but I think there's a domain confusion here . You're describing infra/code RCA. Kelet does an AI agent Quality RCA — the agent returns a 200 OK, but gives the wrong answer.
The signal space is different. We're working with structured LLM traces + explicit quality signals (thumbs down, edits, eval scores), not distributed system logs. Much more tractable.
Your Bayesian point actually resonates — separating good from bad sessions and looking for structural differences is close to what we do. But the hypotheses aren't "100 LLM guesses + k-means." Each one is grounded in actual session data: what the user asked, what the agent did, what came back, and what the signal was.
Curious about the dimensional analysis point — are you thinking about reducing the feature space before hypothesis generation?
Yes. I definitely assisted LLM in writing it. Yeah - I should have stripped it better.
Yet it's f*ing painful to do error analysis and go through thousands of traces. Hope you can live with my human mistakes
Anyway, in practice, I saw it go from 5-6 print-debug cycles down to 1-2 targeted stops.
I built a small CLI that wraps the Debug Adapter Protocol and exposes basic debugger actions so agents can call them directly. Each command blocks until execution pauses, then returns the full context.
The CLI is just a thin layer. The useful part is a debugging “skill” on top that teaches the agent how to debug methodically with it, define clear hypotheses, validate/invalidate them, etc.
Originally I had a (very large and unfriendly) conftest file, but it was quite challenging to collaborate with other team members and was quite repetitive. So I wrapped it as a plugin, added some more functionalities and thats it.
This plugin wraps some boilerplate code in a way that is easy to use specially for the eval use-case. It’s minimalistic by design. Nothing big or fancy
Then it hit me - evaluations are just (kind of) tests, so why not write them as such using pytest?
That's why I created pytest-evals - a lightweight pytest plugin for building evaluations. It's intentionally not a sophisticated system with dashboards (and not suitable as a "robust" solution). It's minimalistic, focused, and definitely not trying to be a startup
# Predict the LLM performance for each case
@pytest.mark.eval(name="my_classifier")
@pytest.mark.parametrize("case", TEST_DATA)
def test_classifier(case: dict, eval_bag, classifier):
# Run predictions and store results
eval_bag.prediction = classifier(case["Input Text"])
eval_bag.expected = case["Expected Classification"]
eval_bag.accuracy = eval_bag.prediction == eval_bag.expected
# Now let's see how our app performing across all cases...
@pytest.mark.eval_analysis(name="my_classifier")
def test_analysis(eval_results):
accuracy = sum([result.accuracy for result in eval_results]) / len(eval_results)
print(f"Accuracy: {accuracy:.2%}")
assert accuracy >= 0.7 # Ensure our performance is not degrading