Hacker News Reader

Comment by theturtletalks | Hacker News Reader

Why is Pi not in the benchmarks? Deepseek beats Strands and its built on Pi so that’s all I needed to know.

Base Pi doesn’t seem like an apt comparison here. Strands comes with MCP servers, subagents, and web fetch tools built in. To get those on Pi, you have to add addons, and then you get into the space of “which addons should be added to make it an apples-to-apples comparison? Which MCP addon do I use, the fastest one, or the most popular one?”. My guess is they chose OMP because it’s the defacto standard for answering those “which addons should be considered out of the box” questions for Pi.

I figured it was lack of MCP support. I would just include a note why it was excluded. Have you checked out FX? It seems to be like Pi but will more things out of the box like MCP.

If "pi install npm:pi-mcp-adapter" is preventing them from including pi in benchmarks, then they might not be competent to run trustworthy benchmarks.

There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark? And who should be the tie breaker when someone thinks they should use a different plugin? Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.

It’s like you didn’t even read my comment.

It's a fair point, but for any well-established capability, the plugin with the most monthly downloads (overwhelmingly) is generally going to be "the one". For MCP, the top plugin [1] has 1M/mo, and the next one down has less than 2k/mo.

I do agree that in the case of benchmarking, it's a bit silly to put the onus on the benchmark runner to try and pick the right plugin for each capability that won't make someone point to /that/ as the reason Pi didn't do as well as expected.

[1] https://pi.dev/packages/pi-mcp-adapter?name=pi-mcp-adapter

> There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark?

The one that everyone else uses, the most popular. Exactly the one I listed.

I wrote "pi install npm:pi-mcp-adapter". It would install the mcp adapter.

> Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.

That's the whole point of Pi, it doesn't have to reach checklist level parity with Strand. Just benchmark it with its default config. Add mcp adapter if your benchmark really requires it, but that's it.

> It’s like you didn’t even read my comment.

I did, but it seems you didn't read my comment, or you didn't understand that the command I gave you is an actual recipe, and not a pseudo instruction.

Deepseek harness is not actually built on Pi harness. It's an independent project.

This is true, but the team said on Twitter that they were heavily inspired by Pi and this whole team used it extensively:

https://x.com/tianyi/status/2088306143772946499

Deepseek was cheaper, but also less accurate. "Beats" isn't a fair assessment.

That’s fair, but Strands is advertising their harness needing way less tokens which does relate to cost.

In that same vein, Pi is less bloated then Deepseek and Oh My Pi, which are built on top of Pi. Isn’t it dubious to leave it out?

OhMyPi is essentially Claude like bloated and Pi is so barebones that many will just get frustrated trying to add bricks after bricks to make it usable for complex workflows - so I don't really see any point of it being in such benchmarks. Pi should have added option to easily add "packs" (and then also tweak tehm) sort of things. When I tried it I actually thought that's what OMP will be, but it was not. It was a different harness altogether in a way and not a nice way.

I wish we could use just the frontend of omp harness-agnostically, like the provider switching, stats dashboard, subscription pooling etc. are excellent features, but it doesn't seem inconceivable that we can have all that with a swappable "actual" harness. I've tried to make omp more pi-like by lazy loading most of the tools rather than dumping them into system prompt. haven't benchmarked this yet.

Do you have any write up on that - how you went about it? Also, in that case why not just make Pi slightly OMP like? Really curious.

Came here to say this. They have oh-my-pi in the benchmark but not pi, but those are very different animals. pi is lightweight out of the box so has very little start-time overhead. (And will not spin up agents like crazy.) pi might do worse if those things are actually important for solving the problem, but it certainly has a shot at being most efficient.

hi Albert from the Strands team. We can look into a Pi run. Our researcher is working on a deep dive for our benchmarks, so we got resources to test against other harnesses

Reply on news.ycombinator.com