yelmahallawy

Gotchu. Yeah that's pretty quick, awesome thanks!

Makes sense, simpler=better. Thanks!

The guy who reviews all of this, is his role in the company fully dedicated to reviewing these eval pipelines?

Yeah, it feels like an unsolved problem still. I've also seen many teams spend hours on human review in eval pipelines (and this accumulates with each new model that gets released).

Have some good takeaways / feedback on this? First time I hear about Braintrust (the eval platform) so I'll look into it but I'm curious on your experience with it so far.

And I think this is a common problem actually — figuring out what to measure and how to measure it – it's not black and white. What I do is have a few dimensions to measure it against (this may or may not fit your use case): relevance, instruction following, clarity, hallucination rate, etc. but even then, it becomes hard to measure things like 'clarity'.

I'd love to hear more about what you're working on (if you're open to sharing!).

I like to play with knowledge base powered chatbots but what's most useful to me (and probably my primary use case) is coding agents since I use CC every day. Recently I just heard about Minimax m2.5 which apparently is a pretty good coding agent (they say it's comparable to opus 4.6) but I haven't tried it yet — plus it'd take a lot of time to figure out whether it's better or not.

Any takeaways on Promptfoo?

Any takeaways? Has it been helpful? OpenAI just acquired them so it's probably useful but I was curious to hear more from people who've actually used it.

Yeah it's a super tedious process and I was hoping that _maybe_ there is a tool out there that can help with this.

Ah good read, thanks for sharing!

Yeah that's essentially what I'm looking for. Since now that AI has become such a core part of most businesses, it's pretty critical to use the _best_ models + prompts for whatever your use case is.

Ah, interesting – yeah only swapping out the model isn't super insightful since models perform differently given different prompts. I'm going to look into GEPA, thanks!

This is interesting approach, thanks for the insight! If I may ask, _approximately_ how long does it take to test a newly-released model with the current strategy?

Do you play with the temperature/top k parameters at all?

This makes sense. I am particularly interested in your invoice processing app example because the accuracy of those outputs can be quantitatively measured from 0%-100% accuracy.

I'm curious as to what is _good enough_ and how many iterations it takes to get there. Is 100% the only acceptable threshold? If so, how many iterations does that take? What does that process look like? Okay let's say 100% accuracy is too difficult to reach, then how do you choose your minimum acceptable threshold (is 95% accuracy good enough? is 90%?). Do you have a dedicated set of outputs and documents used for evals? I'd love to hear more about this example (if you worked directly on the evals for this app).

What I've noticed is that it's hard to measure outputs that aren't binary right or wrong, and that's where most human intervention is needed. The biggest examples of this are chatbots and coding agents – basically any output where you can say "hmm well that's a good response, but there is a better response" and that's what still _feels_ like an unsolved problem, benchmarking those kinds of responses.

On top of that, there are combinations of models+prompts that give different results. For example a prompt could yield a great response from Claude, but the same prompt could yield a mediocre response from Gemini. Not just that but different models have different capabilities (example of this is that composite function calling doesn't work the same way for all models).

I'm asking because I'm generally curious on how teams are solving this today – and it _seems_ like there is no gold standard for evals yet although it's gaining interest.

How I do evals today is by testing an output across different dimensions (and it can vary based on use-case): relevance, instruction following, clarity, hallucination rate, etc. which sucks a lot of time (and can never be fully accurate because how do you fully measure something like "clarity"?), and I feel like there's a better way out there.

Location: Toronto/San Francisco Bay area Remote: Yes Willing to relocate: Yes Technologies: React, Next.js, Node.js, Typescript, PostgreSQL, GraphQL, Resume: https://drive.google.com/file/d/1BTNnLB11txX5_Idboe0QduYuWUr... Email: youssef.elmahallawy01@gmail.com

I'm an entrepreneurial and product-focused engineer looking for full-stack engineer positions. I have built tons of projects that have scaled (biggest one has 200k users) and monetized them.

By having users pay for credits. There's 3 different types of credits, prompt credits, chat credits, and image generation credits. Prompt credits are used when generate prompts, chat credits are used when testing ChatGPT on your improved prompts, and image generations credits are used when you generate an image based on your improved prompts prompts.

I just built a website that uses GPT-4 to create prompts for ChatGPT, DALLE•2, and Midjourney where you can test out the prompts directly on the website. I'm looking for some feedback on the website + some initial users to validate the idea.

Thank you! I will definitely test and iterate.

How do I position myself to get luckier?

How does one get their projects from 0 to 1 (or even get beta users) if all of their socials have <10 karma. All of my previous HN threads were basically me trying to shill my projects, but no one viewed it or commented (along with my posts on twitter, reddit, and discord). How can someone like myself properly market on social media without paying to market?

They also sometimes say opposite things, so in case you were going to say "just listen to them all", it's a little bit harder than that.

Or you can just use https://www.withserve.com/ to run CRUD on your spreadsheets. No database needed.