Hacker News Reader

Comment by jamesponddotco | Hacker News Reader

I'm impressed with the speed, really impressed, but not so much with the quality of the responses. This is a prompt I usually try with new LLMs:

> Acting as an expert Go developer, write a RoundTripper that retries failed HTTP requests, both GET and POST ones.

GPT-4 takes a few tries but usually takes the POST part into account, saving the body for new retries and whatnot. Phind in the other hand, in the two or three times I tried, ignores the POST part and focus on GET only.

Maybe that problem is just too hard for LLMs? Or the prompt sucks? I'll see how it handle other things since I still have a few tries left.

View on HN

I'm a human and I don't have the slightest idea what you're asking for.

Do you use Go? It makes sense to me

The RoundTripper throws me off if anything. RetryRequest, RetryOnFailure, anything could be more descriptive.

It's an interface in the http package: https://pkg.go.dev/net/http#RoundTripper

Til. Thanks, I hate it.

Does anyone outside the Go community call it a "RoundTripper"? I know what a retry is (and things like exponential backoff) and what GET and POST are, but not that, but I also hate Go, so...

EDIT: ah, followup replies elucidated me, it's just a goofy name for a Go-only thing

Thanks, can you send the cached link please? I'd also suggest trying Chat mode for questions like this, where there are unlikely to benefit from an internet search.

Just tried your query now and it seemed to work well -- what are your thoughts?

https://www.phind.com/search?cache=tvyrul1spovzcpwtd8phgegj

Here you go:

https://www.phind.com/search?cache=k56i132ekpg43zdc7j5z1h1x

I'll give chat mode a try. Didn't see that it existed until now.

EDIT

Chat mode didn't do much better:

https://www.phind.com/agent?cache=clsxpl4t80002l008v3vjqw5j

For the record, this is the interface I asked it to implement:

https://pkg.go.dev/net/http#RoundTripper

Thanks for the links. It seems like it switched to Phind-34B, which is worse.

Phind-70B seems to be able to get the right interface every time. Please make sure that it says Phind-70B at the top of the page while it's generating.

In the link it says "Phind-70B", how do we know if it switched to 34B?

The first link definitely says Phind-34B on my browser.

The second one was definitely saying phind 70b on me. Now it is all messed up though.

“RoadTripper”? Or “RoundTripper”?

Ops, haha. Interesting that GPT-4 still got it right though.

Phind still forgot about POST, but at least now it got the interface right.

https://www.phind.com/search?cache=ipu8z1tb3bnn7nfgfibcix38

I'm not sure what you mean that it "forgot" about POST? Even as an experienced Go developer, I looked at the code and thought it would probably work for both GET and POST. I couldn't easily see a problem, yet I had not forgotten about POST being part of the request. It's just not an obvious problem. This is absolutely what I would classify as a "brain teaser". It's a type of problem that makes an interviewer feel clever, but it's not great for actually evaluating candidates.

Only on running the code did I realize that it wasn't doing anything to handle the problem of the request body, where it works on the first attempt, but the ReadCloser is empty on subsequent attempts. It looks like Phind-70B corrected this issue once it was pointed out.

I've seen GPT-4 make plenty of small mistakes when generating code, so being iterative seems normal, even if GPT-4 might have this one specific brain teaser completely memorized.

I am not at the point where I expect any LLM to blindly generate perfect code every time, but if it can usually correct issues with feedback from an error message, then that's still quite good.

This isn't a brain teaser at all. It's a direct test of domain knowledge/experience.

There are countless well-documented RoundTripper implementations that handle this case correctly.

This is the sort of thing you whip up in three minutes and move along. To me it seems like a perfect test of LLMs. I don't need an injection of something that's worse than stackoverflow polluting the code I work on.

That's because it's better at classifying than at generating.

Eg. Tree of thoughts, ...

Reply on news.ycombinator.com