AI Engineer Notebooks – free, framework-free RAG/agents/evals on Colab | Hacker News Reader

evals on Colab

github.com

112 pointscalmrocks15 comments

Man, I could have used something like this last year.

Upon review, it appears that fumbling my way through the dark with Claude and Google led to the same place, in nearly all cases.

However, this is all written by Claude — it has too many em-dashes to not be, does it not? So, maybe that's why we ended up in the same places.

Does anyone know of any other resources in this vein?

If I get told about a swappable seam one more time I am going to lose it.

Ha, fair. Just grepped it — "swappable seam" shows up 11 times across the repo, which is about 10 more than it should. Trimming it down. Thanks for reading closely enough to notice.

But were they load-bearing?

If you switch from claude to codex(or really any other frontier lab) after a spending a little bit of time with how claude writes, you will immediately smell claude's writing from a mile away.

Ha, guilty — I use Claude a ton for code, never thought much about prose style. Also not a native speaker, so half this stuff just reads normal to me lol. Fair catch, trimming it down.

This is pretty basic stuff, just build a bunch of tools with any openAI compatible api and you'll get good. I think stuff like this is kinda overcomplicating simple problems. "Framework free", bruh its just jsonRPC for the most part.

This was probably made to show off for an interview or something, hence why they're throwing throwing around the term "FDE" so much. Its meant to impress a very specific type of person (gullible managers).

Actually yeah, that's kind of the original intent — not trying to go deep on any topic, just trying to get someone unfamiliar to a starting point fast. Once you've got that base, you're free to go deep on your own.

This comment from the author shows that they did not even read what was written, here, before they published it. https://news.ycombinator.com/item?id=49473449

What did it say? Deleted now

Glad to hear they are prioritizing evaluation right from the start. Usually people just throw together a rag pipeline on the knee and then judge the metrics by eye, skimming three responses in the terminal

One thing that evals are super important from the get go are where the harness+model inference is part of the product, e.g. if you are doing voice ai, building out a test harness to test the system is a non trivial first step.

Do you have any idea or resources as to where to look for a guide on one? Or what pitfalls to look out for?

I think for voice-ai every one builds their own harness, and thats why there isn't a common one out there. Also if you are a serious company that has a llm in your production loop, you would have to build an in house thing because its so critical to your product. Its like performance-engineering, most performance engineer work is done in house and it varies wildly. Another reason is that because voice quality is a vibe measure(intonition, pitch, human variation), its impossible to make the whole thing deterministic.

I use pi and built a harness for just an llm that calls a bunch of tools. That got me 50% of the way and it would be fast. Then build it for STT and TTS, this will be slower but it will get you far. There are a bunch of tools out there for building basic harnesses.

Why does applied AI intentionally exclude a framework/harness around AI? The job is to harness the power of AI, and a harness is a critical part of that.

Reply on news.ycombinator.com