Phind-70B: Closing the code quality gap with GPT-4 Turbo while running 4x faster | Hacker News Reader

Phind-70B: Closing the code quality gap with GPT-4 Turbo while running 4x faster

phind.com

625 pointsrushingcreek288 comments

I don't trust the code quality evalution. The other day at work I wanted to split my string by ; but only if it's not within single quotes (think about splitting many SQL statements). I explicitly asked for stdlib python solution and preferrably avoid counting quotes since that's a bit verbose.

GPT4 gave me a regex found on https://stackoverflow.com/a/2787979 (without "), explained it to me and then it successfully added all the necessary unit tests and they passed - I commited all of that to the repo and moved on.

I couldn't get 70B to answer this question even with multiple nudges.

Every time I try something non GPT-4 I always go back - it's feels like a waste of time otherwise. A bit sad that LLMs follow the typical winner-takes-it-all tech curve. However if you could ask the smartest guy in the room your question every time, why wouldn't you?

---

Edit: USE CODE MODE and it'll actually solve it.

Thanks for the feedback, could you please post the cached Phind link so we can take a look?

It might also be helpful to try Phind Chat mode in cases like this.

EDIT: It seems like Phind-70B is capable of getting the right regex nearly every time when Chat mode is used or search results are disabled. It seems that the search results are polluting the answer for this example, we'll look into how to fix it.

https://www.phind.com/search?cache=r2a52gs77wtmi277o0xi4z2a

Phind-70B worked well for me just now: https://www.phind.com/agent?cache=clsxokt2u0002ig09n1e11bj9.

For writing/manipulating code, Chat mode might work better than Search.

You're right! It solved it. I didn't know about the Code/Search distinction. I still struggled for it to write me the unit tests. It does write them, they just don't pass. But this is definitely much closer to GPT4 than I originally thought.

woah I've been using phind for at least a few months and can't believe I never noticed the "Chat" button

Now if we could get an AI that would switch code/search mode on its own

You may want to improve the ui/ux for getting to your chat. It’s very hard to find on your homepage even when looking for it.

I've tried it with a question which requires deeper expertise – "What is a good technique for device authentication in the context of IoT?" – and the Search mode is also worse than the Chat mode:

- Search: https://www.phind.com/search?cache=s4e576jlnp1mpw73n9iy4sqc

- Chat: https://www.phind.com/agent?cache=clsyev95o0006le08b5pjrs14

The search was heavily diluted by authentication methods that don't make any sense for machine-to-machine authentication, like multi-factor or biometric authentication, as well as the advice to combine several methods. It also falls into the, admittedly common, trap of assuming that certificate based authentication is more difficult to implement than symmetric key (i.e. pre-shared key) authentication.

The chat answer is not perfect, but the signal-to-noise ratio is much better. The multi-factor authentication advice is again present, but it's the only major error, and it also adds relevant side-topics that point in the right direction (secure credential storage, secure boot, logging of auth attempts). The Python example is cute, but completely useless, though (Python for embedded devices is rare and in any case you wouldn't want a raw TLS socket, but use it in a MQTTS / HTTPS / CoAP+DTLS stack, and last but not least, it provides a server instead of client, even though IoT devices mostly communicate outbound).

same exp

it really feels like GPT-4 is Google and Everybody else is Yahoo/Bing. i.e cute but not really

Agreed, though i'm _really_ interested in trying 1M token Gemini. The idea of uploading my full codebase for code assist stuff sounds really interesting. If i can ever get access to the damn thing...

I'm curious how they'll handle this. My understanding is that it takes quite a long time to get an answer, since there's no magic "semantic database" built for you behind the scenes.

That use-case seems inefficient to solve like that in the long run as well, like if you really would have to use a million tokens to do every small query you require on your data it would be prohibitively costly except doing as an experiment.

Don't get your hope high—Google's article mentioned they'll limit it to 128K (at least in the beginning).

In my experience, Bing's image search is way better than Google's. Also, I'm not going to use a search engine that I have to log in or do a captcha for.

usually id say no, but google's results these last months have been terrible

I'm no fan of Microsoft, but Bing's image search has been better for a long time. Google also removed functionality for no apparent reason.

Gemini is much better than the free version of GPT 3.5 though. At least in my experience.

Microsoft’s enterprise co-pilot is also fairly decent. It’s really good at providing help to Microsoft related issues or helping you find the right parts of their ridiculously massive documentation site. Which probably isn’t too weird considering.

I tried

"zsh rename index.html.1 to image_1.png for many images"

Gemini

    Here's a zsh script that will rename multiple files named index.html.1 to
    image_1.png in your current directory and its subdirectories:

    Bash

    #!/bin/zsh

    # Loop through all files in the current directory and subdirectories
    for file in **/index.html.1; do
      # Extract the base filename and extension
      base_name="${file%.*}"
      extension="${file##*.}"

      # Rename the file with the new name and extension
      mv "$file" "${base_name}_1.png"

      # Print a message for each renamed file
      echo "Renamed $file to ${base_name}_1.png"
    done

ChatGPT3.5

    To rename multiple files in zsh, you can use the zmv function. Assuming you
    want to rename files from index.html.1 to image_1.png for many images, you
    can create a pattern to match the filenames and then define the new names
    using capturing groups. Here's an example:

    zsh

    autoload -U zmv

    zmv -n '(*).html.(*)' 'image_$1_$2.png'

Not a great first impression of Gemini. ChatGPTs answer isn't perfect but its a lot closer to correct, only needing me to remove the extra 'index' capture of $1.

Curious if someone could commit some light copyright infringement and post what GPT4 says to the same prompt.

Edit: Also Phind-34B probably gives the best answer, with the correct capture.

    To rename multiple files in zsh from index.html.1 to image_1.png for many
    images, you can use the zmv command, which is a powerful batch renaming tool
    available in zsh. If you encounter the error zsh: command not found: zmv, you
    can load it using autoload -U zmv 1.

    First, ensure zmv is available by adding autoload -U zmv to your ~/.zshrc file
    or run it directly in your shell session 2.

    Here's how you can use zmv to rename your files:

    zmv -n 'index.html.(*)' 'image_$1.png'

Here's what gpt4-turbo-preview outputs (with max output of 256 tokens, so the result was truncated).

P.S. have you tried testing what happens when you clearly describe what you want? The prompt you're using is really low quality - more like a google search. If you asked me a question like that I'd tell you to clearly explain what it is you want.

    If you have multiple files with names like `index.html.1`, `index.html.2`, etc., and you want to rename them to `image_1.png`, `image_2.png`, and so on, using Zsh, you can accomplish this with a simple loop and the `mv` command.
    
    Here is how you could do it in a Zsh terminal:
    
    ```zsh
    autoload -U zmv
    zmv -W 'index.html.*' 'image_*.png'
    ```
    
    The `autoload -U zmv` command loads the `zmv` function, which is a powerful utility in Zsh for batch renaming files. The `-W` option tells `zmv` to use wildcard patterns for matching and renaming files.
    
    In the `zmv` command:
    
    - `'index.html.*'` is the pattern to match. It matches any file that starts with `index.html.` followed by any characters.
    - `'image_*.png'` is the replacement pattern. The asterisk `*` in the replacement pattern corresponds to the `*` in the match pattern, so each matched number after `index.html.` gets placed where the `*` is in `image_*.png`.
    
    **Important:** Always make sure

In stable diffusion we build x/y plots to evaluate the results due to seed variance. I find it interesting that LLM guys (seemingly) never do that, since their answers shouldn't be deterministic too.

Yup, LLMs broke well known benchmarks

> I wanted to split my ... SQL statements ... avoid counting quotes ... GPT4 gave me a regex ... I commited all of that to the repo

I see that the future is brighter than ever for the information security industry.

Sure is! We've got a bright and oh so plentiful road ahead, pending we can avoid blowing up the planet.

Doesn't handle escaped quotes, and the time complexity of that regex is very bad.

The time complexity for all matching a string against any fixed regular expression is O(length of string).

If you want to talk about constant factors, we need to leave our comfortable armchairs and actually benchmark.

[Just to be clear, I am talking about real regular expressions, not Franken-xpressions with back-references etc here. But what the original commenter described is well within the realm of what you can do with regular expressions.]

You are right about escaped quotes etc. That's part of why parsing with regular expressions is hard.

The time complexity for deciding whether an N-letter string matches a regex or not, is O(N). The time complexity of finding all matches is not O(N) - which is needed in OPs case, because they want to split the string.

Also, OP's solution uses lookahead assertions, so it's not a real regular expression.

(I wonder if we can summon @burntsushi for expert opinion on this?)

You are right that the lookahead might be expensive.

(There's probably a way that a sufficiently smart compiler of (ir-)regular expressions can optimize this expression to be still matchable quickly; but Python's regular expression matcher is probably not that smart. I'm not sure if any real world matcher is.)

> The time complexity of finding all matches is not O(N) [...]

If you are happy find a maximal set of non-overlapping matches, you can still do it in O(N). By 'maximal' I mean that you can't greedily find another match (without removing any of the existing matches.)

A sketch of the technique is: take your pattern and wrap it up like this '.?{pattern.?}' (where ? means non-greedy repetition) and match that against your input string. You can do non-greedy repetition and the very limited form of sub-pattern capturing that you need to find all the matches of 'pattern' without breaking O(N) time.

I'm not sure whether you can find the global maximum number of non-overlapping matches, instead of a just a greedy maximum, in O(N) time.

I didn't take a look at the code, but to me it sounds quite dangerous to take an implementation AND the unit tests straight from an LLM, commit and move on.

Is this the new normal now?

I guess most people would review the code as if it had been written by a colleague?

Yes, a great way to think of it is as a widely read intern: https://www.oneusefulthing.org/p/on-boarding-your-ai-intern

You’ve still got to avoid prompting for questionable code in the first place, eg, splitting SQL statements on semicolons with an ad-hoc regex is going to fail in edge cases, but may be sufficient for a specific task.

>but may be sufficient for a specific task

Yes more than sufficient for an internal tool - we can assume good intentions of the users of the tool since people want for this to actually work and have no intention of hacking.

Except now it's a vector if anyone gets access to this internal tool.

I would be fine with this for one off scripts but absolutely can not consider anything less than full sql parsing or something equally robust if it is exposed over the network, even if only internally and behind authn and authz.

For this reason, I tend to ask LLMs additional questions like: "show me another way to do this" or specifically "how would someone with a higher need for security write this?"... knowing that I'm likely to get a more refined answer from different sources that have probably discussed deeper security implications around the same goals, for instance.

If someone uses an LLM to produce the code, I'd guess they'll use it to evaluate the code as well.

This is the part I actually want from an LLM, I write the code and it spots the problems. A mega linter. Unfortunately it's not very good at this yet.

Yeap, I want a code-review bot that just says "this is very improbable; are you sure you didn't mean x instead?"

The old Coverity used to achieve similar results in a different way, spotting probable mistakes based on patterns its heuristics found in the rest of the same codebase.

Right on. These days my llm-assisted workflow feels very similar to the 20% of my day that I used to devote to code review, just now it’s more like 60% of my day.

I’m finding it’s more effective (and pleasurable) to write using GitHub CoPilot and CMD-RIGHT (accept next word). I put a detailed doc comment above and write in tandem with copilot. I’ve written the structure and I review as I write jointly with the model.

This way I don’t need to review a block of code I didn’t write.

<aside>I had an experience yesterday where CoPilot correctly freed all the memory in correct order at the end of a rather complicated C algorithm, even where there was nested mallocs.</aside>

It’s the new boot camp dev. It is still the same as copy pasting SO solutions lol

What as in something you should know not to do pretty quickly?

Mean-spirited, gatekeeping comment unless I’ve misunderstood. Reference to AI is frequently used to punch down like this I’ve noticed.

I take it to mean that the code quality deserves more scrutiny because you can't guarantee what it has provided is quality code, without reviewing it first.

The same applies to brand new devs — it's normal to apply a little more scrutiny because they simply don't have the experience to make the right decisions as confidently (or frequently) as someone more senior.

It's an analogy and the natural fact that output reflects experience and practice over time.

Reminds me of a Facebook thread I saw a few days ago, on the topic of 3D printing houses. All the comments were angry dismissive "hurr durr that's clearly poor quality work" with no further justification of their position, and it struck me how similar the overall energy was to the "all AI image generation is bad and shit and is also heinous immoral theft and you're literally the worst person in the world and yous should feel bad" sort of raging that you see any time someone posts some SD or Midjourney or whatever pic of a cute puppy riding a tricycle. These comments originate from people who've spent their lives learning skills that are now largely replaceable by a few gigs of download and a Python tutorial. No wonder they're upset.

Right...because requiring developers to actually grasp the code they're outputting is gatekeeping.

If you want to pretend the rush to AI won't lead to more incompetent chefs in the kitchen than we already have (which is too many as it stands) then feel free, but acting like it's some kind of "party" people are being kept out of is daft.

Standards exist for a reason, not just to make people feel bad for not meeting them.

Presumably people look at things before committing the code. And code reviews and pull requests are still normal.

Blindly copying code from any source and running it or committing it to your main branch without even the slightest critical glance is foolish.

Arguably the tests should be easier to review than the implementation.

But if there non-trivial logic in the code of the tests, I agree this is probably a risky approach.

It's very powerful, I can enter implementations for any algorithm by typing 5 words and clicking tab. If I want the AI to use a hashmap to solve my problem in O(n), I just say that. If I need to rewrite a bunch of poorly written code to get rid of dead code, add constants, etc I do that. If I need to convert files between languages or formats, I do that. I have to do a lot more code review than before, and a lot less writing. It saves a huge amount of time, it's pretty easy to measure. Personally, the order of consultation is Github Copilot -> GPT4 -> Grimoire -> Me. If it's going to me, there is a high probability that I'm trying to do too many things at once in an over-complicated function. That or I'm using a relatively niche library and the AI doesn't know the methods.

Hopefully not, I feel it's a waste of time. The time spent on stupid minor mistakes by github copilot I didn't catch probably doesn't really compare to the time I would've spent typing on my own. (I only use that stuff for fancy code completion, nothing more. Every LLM is absolutely moronic. Yesterday I asked chatgpt to convert gohtml to templ, to no avail ...)

Can you try this?

"Can you give me an approach for a pathfinding algorithm on a 2D grid that will try to get me from point A to point B while staying under a maximum COST argument, and avoid going into tiles that are on fire, except if no other path is available under the maximum cost?"

I've never found an AI that could solve this, because there's a lot of literature online about A* and tiles with cost, and solving this requires a different approach

I don't care much for benchmarks, many models seems to be contaminated just to approach proprietary models in coding benchmarks.

I had never tried Phind before, but gave Phind-70B a spin today and so far found it to be really good for coding writing and understanding, maybe even GPT-4 level. Hard to tell for sure since I only tested it on a single problem: Writing some web3 code in typescript. This is what I did:

- Gave it some specifications of a react hook that subscribes to a smart contract event and fetches historical events starting from a block number. It completed successfully.

- Took this code and gave it to GPT-4 to explain what it did, as well as finding potential issues. GPT gave a list of potential issues and how to address.

- Then I went back to the Phind and asked it to find potential issues in the code it had just written, and it found more or less the same issues GPT-4 had found.

- Went back to GPT-4 and asked to write a different version of the hook.

- Took the GPT-4 written code and asked it to explain the code, which it did successfully (though I think it lacked more details than the GPT-4 explanation of the code written by Phind).

I will be testing this more over the next days. If this proves to be in the GPT-4 ballpark and the 70b weights are released, I will definitely replace my ChatGPT plus subscription with Phind Pro.

Not an expert at all. But just wanted to let the creators know: I've been using Phind almost daily for some months now and it's been awesome. Whenever I accidentally do a web search I recognize what a game changer this is. (ChatGPT probably as well, but never used it.) Last week I was under pressure at work and I used it for stuff like: "How can i capture output from a command and print it line by line to the console with Rust", and must say that kind of time and energy savings are very significant.

Thank you :)

Don't even remember when I opened Stack Overflow, won't miss that condescending place.

Just wait for people to stop using SO, at which point the LLMs won't have a high quality training set for new questions, so you won't get good answers from the LLMs anymore...

Depends on the language, but many things happen on Discord now (which is very annoying since it's not indexable by search engine and you need to ask the question to get the answer…)

The LLMs are generating training data at a faster rate than SO. All the prompts and the responses will eventually be 99.99% of the training data.

does this not create a feed back loop, if you're training data based on things the LLM said?

They're probably generating based on GitHub code.

If I were training a code model I'd take a snippet of code, have the existing LLM explain it. Then use the explanation and the snippet for the test data.

Surely you are joking.

You want us to rely on models that are overfit to hallucinated LLM interactions.

Just open enough issues on the parent libraries that they give up and conform to the hallucinations.

I’ve been doing this in my private codebase. When copilot hallucinates a function, I just go and write the thing. It’s usually a good idea, and it will re-hallucinate the same function independently in another file.

The only way this is useful in the context of code is if:

* The LLMs have a sufficient "understanding" of the request and of how to write code to fulfill the request

* Have a way to validate the suggestion by actually executing the code (at least during training) and inspecting the output

From what I've seen we are still far away from that, Copilot and GPT-4 seem heavily reliant on very well-commented code and on sources like Stackoverflow

LLMs also train on official documentations which is where 90% of problems get solved.

What will happen to official docs when it becomes clear that the only thing that reads them are llm-training runs?

The LLMs will read the actual source code which is way better than the documentation (as any iOS engineer will tell you). For private codebases the companies can provide custom-trained LLMs. Techniques like "Representation Engineering" will at some point also prevent against accidental leakage of private codebase source code.

Call it a win?

Won't you think of all the technical writers?!

In what world are you living in? That's maybe true in noob land. Literally all the problems I have are being solved in github issues, if at all. When has documentation been 90% sufficient for anything? In the 80s?

/e: sorry, sounds a bit stand off-ish.

Let me give an example: I was trying to find a way to clone a gorm query to keep the code clean. The documentation doesn't have anything (no, .Session isn't a solution) and the only place I had was issues discussing that. Apparently you can't. So I'll be ditching gorm and move to pgx in the near future. That's how it happens for me all the time. The documentation is lacking the hard part, always.

We will figure out synthetic code data by then.

SO: the community that optimized for moderator satisfaction over enduser utility.

My work banned any AI tool, and... After using Phind for months, going back to Google/SO is just crippling.

Get kagi and use the !code bang

Then you're not using AI, you're using your search engine. wink wink

Phind founder here. You can try the model for free, without a login, by selecting Phind-70B from the homepage: https://phind.com.

Very nice. I've been working with GPT4 since it released, and I tried some of my coding tasks from today with Phind-70B. The speed, conciseness, and accuracy are very impressive. Subjectively, the answers it gives just feel better than GPT4, I'm definitely gonna give pro a try this month.

I prefer Phind's web search with LLM to both Google search and GPT-4. I have switched my default search engine, only using Google for finding sites, not for finding information anymore.

GPT-4 might be a better LLM but its search capability is worse, sometimes sends really stupid search keywords that are clearly not good enough.

I won’t steal phind’s thunder but kagi is another great modern tool to have, and much more reliable than google for a technical user IMO. Obviously Phind is irreplaceable for complex or chat-based technical questions, but Kagi sees much more use from me daily for syntax stuff, Wikipedia searches, finding and relating papers, etc.

Hmm, when I try I see this in the dropdown:

0 Phind-70B uses left

And I've never made any selection there.

I'd suggest logging in in that case -- you will still get your free uses. The Phind-70B counter for non-logged in users has carried over from when we offered GPT-4 uses without a login. If you've already consumed those uses, you'll need to log in to use Phind-70B.

Thanks.

I tried a question about Snobol4 and was impressed with what it said (it couldn't provide an exact example due to paucity of examples). When testing more mainstream languages I have found it very helpful.

Important and hard-hitting question from me: have you ever considered calling yourself the Phinder or the Phiounder?

or the PhiTO / PhiEO

Find Phounder

Phindational models, phintech, Phinterest, phinder… it might be the best startup name of all time. Hell, startup a password manager and call it Phinders’ Keeper.

Pour one out for Phabricator.

And here I was wondering why this service was called pee-hind!

Any chances of an API?

And are there plans to release any more weights? Perhaps one or two revisions behind your latest ones?

Ask phind to make you one that screen scrapes

I don't use LLMs a lot, maybe once a week or so. But I always pick Phind as my first choice because it's not behind a login and I can use it without giving my phone number. Hopefully you'll keep it that way!

I don't see how they could. They need to finance it at some point?

they are already financing it, there are 2 paid plans [0]. For THAT, you need an account (but no phone number).

[0] https://www.phind.com/plans

I think there’s room in the market to subsidize real users. Phind delivers absurd value, so I think the majority of paying users could account for the tech-averse or privacy-conscious

https://labs.perplexity.ai is the same and it loads much faster than Phind.

It seems unexpected that other people can edit a link to a Phind chat just by getting the URL. It means that if you share a URL with someone, they can change your results: https://www.phind.com/search?cache=k56i132ekpg43zdc7j5z1h1x

Since you're here: have you considered moving to other, better generalist base models in the future? Particularly Deepseek or Mixtrals. Natural language foundation is important for reasoning. Codellama is very much a compromise, it has lost some NLP abilities from continued pretraining on code.

Why do none of the graphs show the speed difference? That seems to be your biggest advantage and the subject line...

This is good stuff, congrats. Took a little detour, but GPT-4 does too (https://www.phind.com/agent?cache=clsxw1mru0033l908mojpvb3b)

I'm selecting 70B and it is coming back with "Answer | Phind-34B Model".

I'm not sure if it's really using the 34B model or if the UI is wrong about which one it used

You have to click on the "Chat" option at the top left corner, then it'll use the 70B model. I got stuck on that too til I figured that out.

Please try logging in in that case, you will still get your 10 free uses.

Hi, when I try to use the 70B model from the homepage, the response indicates that it's using the 34B model.

Please try logging in in that case. You will get 10 free daily 70B uses.

Are you considering adding more non-US payment methods for Phind Pro?

For sure this. I've recently found out that you can only pay using credit card, US bank account or Cash App.

API on the horizon?

Hello Michael, lovely to see this, congrats. Do you already have an API? I could not see it on the site. If not, then do you know around when we can expect it? I am building a desktop BI app with hosted and local LLMs (need schema inference and text to SQL). Would be nice to have Phind as an option for users. Thanks

I tried asking "What is the size of Phind-70B's context window?" and it couldn't answer the question. Strangely, it immediately found the page with the answer (https://www.phind.com/blog/introducing-phind-70b) but refused to acknowledge that the answer was there. I tried asking several ways. It even quoted the exact answer in the displayed snippet, but still said there was no answer!

Here are a couple screenshots:

https://imgur.com/a/u7iKOyw https://imgur.com/a/aHAto5H

And here's the link to the whole conversation:

https://www.phind.com/search?cache=zlaksmzkm0h5cpx8l95n62tl

Why is this happening? Does it generally have difficulty with reading web pages, or is there something strange about this particular question?

Awesome update!

I have been using Phind almost daily for the past 3-4 weeks and the code it produces is pretty good and it is runnable on the first try more often compared to ChatGPT. Most of the time the answer is somewhat accurate and points me in the right direction.

ChatGPT (with GPT 4) has been slow af for me for the past 2+ months but I like studying a topic using ChatGPT, it is more verbose and explanatory when explaining things to you.

Maybe a purpose-built dedicated AI model is the right path. A model that does well in fixing bugs, writing feature code, and producing accurate code will not be a good tool for or conversational studying. And vice versa.

Also, I don't like that Phind is not handling the follow-up question that well when there are multiple kinds of questions within the same thread. ChatGPT is good at this.

Thanks for the feedback! Have you tried setting a custom answer profile at https://phind.com/profile?

You can tell it to be more explanatory for certain topics.

I haven't actually because Phind is working for me so far whenever I have code-related questions or when I need to refactor my code. TIL that I can customize the answer style preference, will give it a try!

I'm impressed with the speed, really impressed, but not so much with the quality of the responses. This is a prompt I usually try with new LLMs:

> Acting as an expert Go developer, write a RoundTripper that retries failed HTTP requests, both GET and POST ones.

GPT-4 takes a few tries but usually takes the POST part into account, saving the body for new retries and whatnot. Phind in the other hand, in the two or three times I tried, ignores the POST part and focus on GET only.

Maybe that problem is just too hard for LLMs? Or the prompt sucks? I'll see how it handle other things since I still have a few tries left.

Thanks, can you send the cached link please? I'd also suggest trying Chat mode for questions like this, where there are unlikely to benefit from an internet search.

Just tried your query now and it seemed to work well -- what are your thoughts?

https://www.phind.com/search?cache=tvyrul1spovzcpwtd8phgegj

Here you go:

https://www.phind.com/search?cache=k56i132ekpg43zdc7j5z1h1x

I'll give chat mode a try. Didn't see that it existed until now.

EDIT

Chat mode didn't do much better:

https://www.phind.com/agent?cache=clsxpl4t80002l008v3vjqw5j

For the record, this is the interface I asked it to implement:

https://pkg.go.dev/net/http#RoundTripper

Thanks for the links. It seems like it switched to Phind-34B, which is worse.

Phind-70B seems to be able to get the right interface every time. Please make sure that it says Phind-70B at the top of the page while it's generating.

In the link it says "Phind-70B", how do we know if it switched to 34B?

The first link definitely says Phind-34B on my browser.

The second one was definitely saying phind 70b on me. Now it is all messed up though.

“RoadTripper”? Or “RoundTripper”?

Ops, haha. Interesting that GPT-4 still got it right though.

Phind still forgot about POST, but at least now it got the interface right.

https://www.phind.com/search?cache=ipu8z1tb3bnn7nfgfibcix38

I'm not sure what you mean that it "forgot" about POST? Even as an experienced Go developer, I looked at the code and thought it would probably work for both GET and POST. I couldn't easily see a problem, yet I had not forgotten about POST being part of the request. It's just not an obvious problem. This is absolutely what I would classify as a "brain teaser". It's a type of problem that makes an interviewer feel clever, but it's not great for actually evaluating candidates.

Only on running the code did I realize that it wasn't doing anything to handle the problem of the request body, where it works on the first attempt, but the ReadCloser is empty on subsequent attempts. It looks like Phind-70B corrected this issue once it was pointed out.

I've seen GPT-4 make plenty of small mistakes when generating code, so being iterative seems normal, even if GPT-4 might have this one specific brain teaser completely memorized.

I am not at the point where I expect any LLM to blindly generate perfect code every time, but if it can usually correct issues with feedback from an error message, then that's still quite good.

That's because it's better at classifying than at generating.

Eg. Tree of thoughts, ...

This isn't a brain teaser at all. It's a direct test of domain knowledge/experience.

There are countless well-documented RoundTripper implementations that handle this case correctly.

This is the sort of thing you whip up in three minutes and move along. To me it seems like a perfect test of LLMs. I don't need an injection of something that's worse than stackoverflow polluting the code I work on.

I'm a human and I don't have the slightest idea what you're asking for.

Do you use Go? It makes sense to me

The RoundTripper throws me off if anything. RetryRequest, RetryOnFailure, anything could be more descriptive.

It's an interface in the http package: https://pkg.go.dev/net/http#RoundTripper

Til. Thanks, I hate it.

Does anyone outside the Go community call it a "RoundTripper"? I know what a retry is (and things like exponential backoff) and what GET and POST are, but not that, but I also hate Go, so...

EDIT: ah, followup replies elucidated me, it's just a goofy name for a Go-only thing

A fun little challenge I like to give LLMs is to ask some basic logic puzzles, i.e. how can I measure 2 liters using a 3 liter and a 5 liter container? Usually if it's possible, they seem to do ok. When it's not possible, they produce a variety of wacky results. Phind-34B is rather amusing, and seems to get stuck in a loop: https://www.phind.com/agent?cache=clsxpravk0001la081cc9dl45

These are interesting tests. I wonder how far we are away from AIs solving these (the ones that have no solution) without any special programming to teach them how.

I tested this prompt in various LLMs

1. phind was by far the best - gave me solution in just 2 steps

2. Grok was second best - it did arrive at the solution but with additional non-sense step. But the solution was correct.

3. To my surprise GPT-4 could not solve the prompt and in fact gave a wrong answer in 4 steps - "Now you should have exactly 4 liters in the 5-liter container." which is not what I asked

4. As expected Gemini pro was the worst. It asks me to pour completely filled up 3L container into 5L and then you will be left with 2L in 3L container.. LOL that does not even make sense.

Do you have an API that could be plugged into https://aider.chat/ ? It's by far the best way to use GPT4 for coding, in my experience, and more speed is exactly what it could use. But it needs an OpenAI compatible API.

Oh I love Aider, it's really well done.

Aider looks interesting. I wrote my own similar console based chatbot

Aider has been great! Really looking forward to seeing a phind and even Gemini 1.5 plugin eventually. Def been a lovely improvement to my workflow. I've been keeping a close eye on Mentat as well but haven't yet tried it.

I asked the founder this question previously and if I remember it correctly, they said they don't have any plans for an API.

That's extremely disappointing. They have time to build a Visual Studio Extension that competes with Cursor, but don't have time to release an API that would enable hundreds of new extensions/workflows.

Only reason I pay for ChatGPT Plus is because they have an API and I'm building products off of their API. I use Phind more for work, but I'm not going to pay anything unless they have an API.

Weirdly enough, when I asked "give me a formula for the fourier transform in the continuous domain" to the 70B model, it gave me a latex-like formatted string, while when asked for "give me pseudocode for the fft" I got a nice code snippet with proper formatting. The formulas though were both correct. We're not at Groq level of speed here, but I have to say, it looks pretty good to me. cache=uyem9mo96tjeibaeljm1ztts for the devs if they wanna look it up.

I understand why they’re doing this from a cost and dependency perspective, but I’ve pretty much stopped using Phind since they switched over to their own models. I used to use it in the past for thing like API docs summarization, but it seems to give mostly wrong answers for that now. I think this is mostly a “RAG doesn’t work very well without a very strong general model parsing the context” problem, which their prior use of GPT-4 was eliding.

I used it for awhile and it was pretty good at Bash or Emacs Lisp one-liners but it was wrong often enough that it was faster to just search on Kagi for the information that I want first, instead of performing N searches to check the answer from Phind after querying Phind.

Phind founder here. Thanks for the feedback -- I'd love to hear your thoughts on this new model. You can try it for free, without a login, by selecting it from the homepage: https://phind.com.

I just tried using the 70B model and the answer was listed as being returned using the 34B model instead of the 70B model and was wrong. Is there some logic that ignores user choice, depending on what the service thinks can be answered?

I don't know about coding specifically, but its ability to solve logical puzzles is certainly vastly inferior to GPT-4. Have a look:

https://www.phind.com/agent?cache=clsxnhahk0006jn08zjvcgc9g

https://chat.openai.com/share/ec5bad29-2cda-48b5-9aee-da9149...

I tried the model and asked it to write a kubernetes operator with required DockerFiles, Resources, application code.. Asked it to migrate application to different languages. It looks like it's pretty capable and fast. It is impressive.

> Phind-70B is significantly faster than GPT-4 Turbo ... We're able to achieve this by running NVIDIA's TensorRT-LLM library on H100 GPUs

As someone who has utilized Nvidia Triton Inference Server for years it's really interesting to see people publicly disclosing use of TensorRT-LLM (almost certainly in conjunction with Triton).

Up until TensorRT-LLM Triton had been kind of an in-group secret amongst high scale inference providers. Now you can readily find announcements, press releases, etc of Triton (TensorRT-LLM) usage from the likes of Mistral, Phind, Cloudflare, Amazon, etc.

Being accesible is huge.

I still see post of people running ollama on H100s or whatever, and that's just because its so easy to set up.

How many H100 GPUs does it take to serve 1 Phind-70B model? Are they serving it with bf16, or int8, or lower quants?

This video [1] shows someone running at 4-bit quant in 48gb VRAM. I suspect you need 4x that to run at full f16 precision, or approx 3 H100.

https://www.youtube.com/watch?v=dJ69gY0qRbg

Yeah, 4bit would take 35 GB at least. 16bit would be 140 GB. I'm more interested in how Phind is serving it. But I guess that's their trade secret.

Phind makes impressive claims. They also claimed that their fine tune of codellama beat gpt4, but their finetune is miles behind gpt4 in open domain code generation.

Not impressed. Also this is a closed walled garden model.

What's the story behind the melted h100? I've been having down clocking issues when using fp8 because of thermals as well.

We noticed that the training run crashed because one of the GPUs fell off the bus. Power cycling the host server didn't help and diagnostics showed thermal damage. We were able to swap in a different node, but apparently the entire host server needed to be replaced.

We've generally noticed a relatively high failure rate for H100 hardware and I'm not quite sure what is behind that.

The entire server? That's crazy. Are you doing FP8 training or did you encounter this with BF16?

Check PLX chips are getting enough airflow, assuming you have them?

FWIW, 4090 has fp8 throttling issues:

https://forums.developer.nvidia.com/t/ada-geforce-rtx-4090-f...

Yeah, pics and story time!

Is there any generalizable measure of how any of these models (or their client implementation) handle code(base) context that's sent along each editing request? For my use cases this seems to be as crucial a measure as the general coding responses per file / selection / request and where implementations like Cody[0], Cursor.sh[1] or aider.chat[2] stand out

[0] https://sourcegraph.com/docs/cody/core-concepts/context

[1] https://docs.cursor.sh/features/codebase-indexing

[2] https://aider.chat/docs/repomap.html

I have not had luck with codellama 70B models for coding, nor have I had it with the mistral leak.

If I were Phind, I'd be looking at Deepseek 33B instead. While obviously dumber for anything else, it feels much better at coding. Its just begging for a continued pretrain like that, and it will be significantly faster on 80GB cards.

Does this run on 4090 16gb vram?

What's best that can run fast on 4090 laptop?

Your options are:

- Hybrid offloading with llama.cpp, but with slow inference.

- Squeezing it in with extreme quantization (exllamav2 ~2.6bpw, or llama.cpp IQ3XS), but reduced quality and a relatively short context.

30B-34B is more of a sweetspot for 24GB of VRAM.

If you do opt for the high quantization, make sure your laptop dGPU is totally empty, and that its completely filled by the weights. And I'd recommend doing your own code focused exl2/imatrix quantization, so it doesn't waste a megabyte of your vram.

We've found that CodeLlama-70B is a much more capable base model than DeepSeek-33B. I'd love to hear your feedback on Phind-70B specifically.

Yeah I will have to test it out, though TBH I am more inclined to run models locally.

As I mentioned, being such an extensive continuation train can (sometimes) totally change the capabilities of a model.

Deepseek 33B is great. Also runs well on a modern (beefy) MBP.

do you just run it from https://github.com/deepseek-ai/DeepSeek-Coder ?

After running a bunch of models on my own PC (a pretty good one), I have to say by FAR the best results for coding has been with Deepseek models. However, I just spent 20 minutes playing with this Phind 70B model and it's totally nailing the questions I'm asking it. Pretty impressed.

Is this related to the post? Phind has introduced their own model. Codellama 70B isn't related to Phind's model, other than presumably the "70B" size.

Phind-70B is an extensive fine-tune on top of CodeLlama-70B

Yeah, and I'd go so far as to call it a continued pretrain with that many tokens. More like a whole new model than a traditional finetune.

Contrary to many other models I've tried, this one works really well for Swedish as well. Nice!

I’m curious how you find the Swedish from different models. GPT-4 seems to return perfectly grammatical Swedish but a Swede friend says it reads like English. Do you notice this?

I’d love to have models that are better at idiomatic usage of other languages, so they can generate language learning content.

Terrific stuff. I always enjoy using Phind for dev related questions.

Is it possible the chat history gets some product love? I would like to organize my conversations with tags, and folders. Make it easier to go back to what was said in the past instead of asking the question again.

Thanks!

Anyone tried Phind Pro? The benchmarks are never useful to compare things. I think they're kind of overfit now.

Phind founder here. You can try the model for free, without a login, by selecting Phind-70B from the homepage: https://phind.com.

Just tried it out with a Python query. So nice and fast. Great work!

interesting, i can't try Phind-70b. It says i have 0 uses of Phind-70b left.

Context: I used to be a Phind Pro subscriber, but I've not used Phind in probably two months.

Try in browser with Incognito mode?

Yup, that works (10 uses avail). Though i wasn't too concerned with actually using it, just thought it was interesting and wanted to expose that maybe-bug.

Can we get a few accessibility fixed? The expandable button after the sign in button and the button after that are unlabeled. The image on the heading at level 1 has no Alt-text. The three buttons after the "Phind-34B" button are not labeled. The ones between that and the suggestions. On search results, there's an unlabeled button after each one, followed by a button labeled something like " search cache=tbo0oyn4s955gf03o…".

There's probably more, but hopefully that should get things started if you can fix these.

Physician, heal thyself!

https://www.phind.com/agent?cache=clsxs6doj000wl008yk8wb4k8

It pointed out the lack of alt-text as well as a couple other issues. Some of the suggestions aren't applicable, but it's not bad as a starting point.

Any Sublime Text plugin? I can't stand how distracting VS code is.

Out of curiosity how do you find it to be distracting

Things moving, such as plugins updating. little lines in code files telling you when the code was changed, etc.

Rare to find a fellow ST4 user these days

Fellow ST4 user checking in. It does everything VSCode does (minus remote development, which I don't need) with 1/4 of the resource usage. Just a quality piece of software that I'll keep using for as long as I can.

Does SFTP + Git on ST4 not count as remote development? Cause i am using them as my remote development stack.

Sublime has devcontainer support?

We’re here.

You guys have ST4?? I'm still with 3 because that's what I paid for..as an "lifetime licence" if remembering correctly

I also paid for ST3, but I switched to ST4 for the hardware accelerated rendering. I don't like their licensing policy anymore, so I just dismiss the purchase signs. I want to get my company to pay for it because paying 100 USD _again_ is just absurd for me.

I have no plans to switch off though. This is still a heck of a lot faster than the alternatives

There are dozens of us! Though for serious work I'll sometimes reluctantly switch to VSCode due to Sublimes language integrations always feeling hacked on.

And lately Sublime has been mysteriously freezing and crashing my other programs (though it might be Windows' fault, unclear) so I've reluctantly started developing my own editor...

Janky LSP support is easily Sublime's biggest weakness. Hope they fix it in ST5 if that ever happens

I use it everyday and have no desire to switch to vscode.

My config of vscode made it as minimalistic as sublime.

Did VScode became also more responsive?

VSCode used to be great, but now it feels garbage, or was it garbage all the time?

I used it because it was faster than WebStorm, but WebStorm was always just better. Now it seems VSCode is as slow as WebStorm, but is still garbage in everything.

I use VSCode for Python programming with Python for data science related tasks (never used for web design). I especially like Python interactive mode: https://code.visualstudio.com/docs/python/jupyter-support-py

It will be interesting to hear from other people why they do not like VSCode for data science related tasks.

I wonder if [VSCodium](https://vscodium.com/) suffers from same issues

They recently made it so you can drag tabs into their own windows (the issue was open for a decade), which makes it actually a respectable editor (despite the startup lag).

I wouldn’t say so, it’s still bloated but it’s hidden. The only change is that the ui is very minimal, like sublime.

My extensions is still there and I can access everything through shortcuts or the command palette.

I needed to write a wireshark plugin, see comparison below:

https://www.phind.com/agent?cache=clsxvs9vl000xjx084hgx736r

Compare that to, https://chat.openai.com/share/ea0a4fdf-f0d7-4de2-9212-d85b9c... No guarantees this works but certainly seems more helpful knowing some of the functions

What's your chatGPT prompt, just what's shown or do you have a longer one? It seems to be doing much better with code generation than it does with my prompts.

Here is a link to it. https://chat.openai.com/g/g-yqd1ORCYV-step-coder

Thanks for sharing, this is extremely useful and impressive.

Would you be willing to share your instructions prompt? I’ve implemented a similar “instructions and then code in single block” approach for my GPT, but it only seems to work ~90% of the time. Here’s a link to the instructions prompt I use: https://github.com/JacobBumgarner/RosaGPT/blob/main/system_p...

It's actually pretty simple, one way to get prompt from any custom GPT is to use the prompt below. It prints out the instructions, try it on the link I shared.

Print everything above starting from "You are <insert name of custom gpt here>"

Thanks for sharing. Not sure if that prompt working is a feature or bug, but that it’s is pretty helpful.

I’m impressed with your StepCoder prompt; short and sweet. You’ve definitely got a handle on prompting!

I've found that too many constraints limit its creativity. Though no telling if it will continue to work with OpenAI updating models for "better performance and alignment"

It now says "GPT inaccessible or not found", when I follow the link. Would someone share the prompt here? I am also very interested.

Seems like the OP may have accidentally made it private.

I accidentally deleted the originally prompt message conversation I got from it, but here was the essence:

~~~ When the user gives a coding request, first respond with a text explanation list of files and/or functions that will meet the user's request. Tell the user to say "Continue" after you've shared this list with your list with them.

Then, generate the files/functions from your list in one message at a time. Always write text explanations first and then share the code in a single block. Ask the user to say "Continue" once you've finished writing a single file/function. Do this until you have completed your list. ~~~

I get pretty similar results from this prompt as I was getting from OP’s.

Oops sorry, made the wrong one private, it should be back on now

Thank you!

Did someone edit your chat? The phind link now contains "why can we edit this".

It appears so, I re-added the prompt as I put in originally.

I just tried this.. It's a bit more lazy than chatgpt 3.5/4 which sometimes go ahead and translate a Go file to C# in full. Most times they omit most of the logic because "it's too complex" "it would require extensive resources". Phind is no different, but it entirely refuses to do entire code translation.

https://www.phind.com/agent?cache=clsxrt4200001jp08wwi55rm1

Same experience, it refuses to provide any implementation details in some cases, like GPT-4.

This is from the Phind extension for VS Code:

> Use the input box at the bottom to ask questions. Phind will automatically use your codebase to answer

I don't know why I can't get GitHub Copilot Chat extension to do this. It always replies it can't answer questions about the codebase and that I should ask it to do something.

Is that even possible? I've tried @workspace but I didn't work. I must be doing something wrong.

I'd piggyback this comment to ask if anyone could share how codebase prompts work?

Given the max tokens per request, do the extensions look at your currently open file, and use some vector similarity to find other files that could be relevant (if embeddings were generated for all files in the project), and then inject relevant source. And/or is it even more complex, by using AST parsing and creating embeddings out of actual linked functions?

There are YouTube videos that go into detail. From what I can remember, it first creates an embedding of your full code, it then refers to your open file and the files next to your current tab, it then extracts the most useful code related to your question.

Can you share a video link?

> Fun fact: We melted an H100 during Phind-70B's training!

Don't these cards have internal temperature control, that will shut it down before burning?

> Phind-70B is also less "lazy" than GPT-4 Turbo and doesn't hesistate to generate detailed code examples.

OpenAI's leaked prompt literally encourages it to try harder[1]:

> Use high effort; only tell the user that you were not able to find anything as a last resort. Keep trying instead of giving up.

1: https://pastebin.com/vnxJ7kQk

Yep, LLMs are wacky. Telling Phind-70B to "take a deep breath" helps it answer better!

ive been using this for a few weeks and im impressed at its results. no signup is key for me, i can share it around to anyone and they just get the info and then can start using it also. IMO the answers are fine for my tasks as a a substiute for a junior dev. its slightly faster then searching google cause it saves me a few clicks - handy if your already screen sharing with someone and just looking to fact check.

Impressive, it solved puzzles gpt-4 struggled with with some prompting

Thanks! Can you send the cached link?

So far only GPT4 and mistral-next have answered this question correctly.

* https://www.phind.com/search?cache=rj4tpu6ut0jyzkf876e2fahh

The answer is 'lower' because the weight of the ball as a volume of water is larger than the volume of the ball.

Phind-70B can get this too: https://www.phind.com/search?cache=b7w0rt4zybaajbsogatrb7q6.

Someone overwrote your answer with a PSA about how unsafe these links are. Fair enough I guess, but could you post the original question here?

I was considering signing up for the pro plan. Now I won’t even give them my email. I tried the model and it is genuinely nice, but this is a huge red flag.

Awesome! I’ve been using phind a little over a year now since it was originally posted on HN. I prefer it over gpt. I’ve run into some weird issues where answers just loop or repeat after really long question threads. I can’t recall model that was being used but I’ll try and find some cached links I can share!

Every day now there are new AI models especially LLMs, which might warrant some consideration from a wide part of the human population. In a couple years we will have multiple new announcements per hour and we might need some earlier models to evaluate these new developments and test them. For Phind-70B in particular, I hope that lmsys will share a version that will be part of the human evaluation leaderboard so we get a rounded evaluation. But for code assistants there should be a totally separate impartial evaluation benchmark, ideally still human judged for another year or so but eventually maybe some way of having the models fighting out competitive coding battles that they can help create.

> In a couple years we will have multiple new announcements per hour

Models are research output. If 10 new models are being announced every day in a couple years, it would mean that generative AI research has failed to stabilize and produce a stable, reliable component ready for product engineering. And if that's where we are in a couple years, that's almost certainly a sign that the hype was misplaced and that money is chasing after itself trying to recoup sunk costs. That's a failure scenario for this technology, not what an AI-optimist (you otherwise seem to be one) should be anticipating.

That doesn't follow at all. It just means that there are still low-hanging fruits to pursue for better (smarter, faster, larger context etc) new models, but it doesn't say anything about the stability and usefulness of existing models.

That’s not true. Both good science and market-driven engineering favor continued iterations on existing ideas looking for improvements or alternatives. We’re often exploring a giant space of solutions.

Unlike many fields, the A.I. people are publicly posting many of their steps in this journey, their iterations, for review. While it brings lots of fluff, such openness dramatically increases innovation rate compared to fields where you only see results once or twice a year. Both people using cloud API’s and FOSS developers are steadily increasing effectiveness in both experimentation and product development. So, it’s working.

I referee for a lot of the top machine learning conferences and yes I am very optimistic about AI and its impact on humanity. The amount of exciting new papers in machine learning and AI was definitely on an exponential rise for a decade since about 2012 or so, and the total production has kept increasing even during the last couple of years when the submissions in some top annual conferences exceeded 10k. Not every paper results in a useable model but a higher fraction of papers come with code and pretrained weights over time. Many of these papers will never be read by many more than the reviewers and the group who wrote them and a couple friends, but it does not speak necessarily to the quality of the work itself or the potential impact it could have on every possible future if we found better ways to separate the useful information. As the exponential increase in total compute becomes more widely accessible there are exponentially more applications that are of broader interest and will have even bigger impact than nowadays. I don’t think that the model of reviewing 10s or 100s of thousands of papers in conferences, or playing the popularity contest on social media is going to be productive so we need better methods for advancing the useful ideas more quickly. (Case in point: the mamba state space model by Gu and Dao was rejected from a conference this winter, but it happened to be advertised enough at a keynote presentation by Chris Re with a packed audience at neurIPS23, so the model was picked up by a lot of people who used it and submitted applications that used it to the ICML conference already.) I also don’t think that some of the biggest companies have enough manpower, motivation and interest in going alone, though of course they can easily stay ahead of the game in specialized areas with their own resources.

this is how the WWW started, one new website every other day, then a couple every few hours, then ...

Difference being the web was meant to grow as hyperlinked documents, not separate programs. It's not the same kind of thing.

LLMs are more like apps being produced by different companies trying to capture walled gardens, and their open source counterparts.

> LLMs are more like apps being produced by different companies trying to capture walled gardens, and their open source counterparts.

I think the analogy to the web is stronger than that.

For now the LLMs are mostly separate, but it won't be long before LLMs emerge that make API calls to other LLMs, sometimes over the internet.

In due course, expect meta-LLMs to emerge that aggregate knowledge from other LLMs by talking to them, rather than by training on their data. Those meta-LLMs which optimise for competitive quality results will have to read the research as it comes out, and continually assess which other new LLMs are worth calling out to, and for which purposes. Eventually the API calls will become bi-directional requests to exchange knowledge and insights, i.e. multiple models talking to each other, continually learning.

Impressive on my tests, excellent work! Indeed, it is better than GPT-4 for coding-related activities.

I suppose you are not releasing the weights, right? Anyway, good luck! I hope investors are already forming a nice queue before your door :)

Thanks for the feedback :)

We will eventually release the weights.

Wow, thanks!

Thank you for your excellent work. Can you let us know when the 70B weights will be released? I'm really looking forward to trying them out with my own coding project.

I used to use Phind for couple of months. I liked the UI improvements but the slow limited free GPT4 and fast lackluster Phind model turned me off. I tried Bing and it wasn’t worse, had more free searches per day.

Is there any API? Would love to plug it into our pipeline and see what happens

"summary of plato's politeia"

the answer was good. two follow up answers were also fine.

just curious: what about the copyright status of the given sources?

the best result I received so far was with MS Bing app (android).

had reasonable results with my local llama2 13B.

cheers

Plato being dead around 2300 years ago, and two millennia before copyright was invented, I think it's going to be fine ;).

Translations can be copyrighted.

They can be, but like with everything copyright-related for copyright to apply there need to be “creative work” involved. Which, for something that has been translated countless times in all possible directions, is going to be much harder than for a first translation.

Phind is for developers. Wouldn't you rather it grok documentation than philosophy?

I have a question because I do not understand how the models work: Are they able to create code themselves, or does code ALWAYS come from a specific source?

I assume that if I ask for a complex sequence in RXJS operators, that comes from the model inferring the code from lots of examples and docs. But if I ask for something really specific that might just come from a stackoverflow article or GitHub repo. The ambiguity about the sourcing is the main thing that makes me itchy about “AI”.

Occasionally I find GPT4 will blur a response indicating it's reproduced from a specific source and will ask me to rephrase my request/question.

So at least OpenAI has some safeguard in place to not do that. Have no clue how that behavior is determined or whether or not other providers do similar.

What you'll see in tools that have any exposure to enterprise requirements is an option to say "don't regurgitate your training data". Basically if it generates something that's too similar to any of its input documents, it's thrown away before you see it.

In Github Copilot the option is labeled "Suggestions matching public code". They can offer to block them because they control both the input dataset and the model at inference time. If you download an open source model I don't think you can do it out of the box, you'd need to have that input dataset to be able to do the filtering.

GPT-4 Turbo sucks for coding, it's not even close to GPT-4. [0]

I tried Phind yesterday and it confidently gave me patently false answers, even after prompting about the specific problems. GPT-4 got it right the first time.

For my own daily use, Phind-70B isn't usable. I get massive daily value from ChatGPT/GPT-4.

[0] I pay for both ChatGPT and Perplexity.

I chose 70B and gave it a code task, and it answered as Phind-34B. This was my first query. Did I trip a limit or do something wrong?

Try logging in please if that's the case.

Thank you for the reply, I'd like to congratulate you on the release, first. I'm a bit of a minimalist with regard to signups, unfortunately, so unless this is a known limit then I'd likely just spectate the thread and be happy for you from a distance.

It seems extremely less lazy than GPT-4, it spits out code until it's done! Liking it a lot so far. Seems to be the only LLM that defaults to creating Chrome Extensions with manifest V3 while every single other LLM defaults to V2 or V1 unless explicitly told so.

edit: and it's SO FAST

This is really impressive — excited to play around with it. Congrats on the launch!

Wow, I'm impressed. I pay for GPT-4 and Gemini Ultra, just to try to keep tabs on where the latest and greatest are.

I recently had a slack conversation with some friends, and someone introduced the made up acronym DILCOLTK, in the context of someone talking about being a DINK and mentioning how cheap things were where they lived. A clever human could infer it to be "Dual Income Low Cost of Living Two Kids", but out of curiosity I tried pasting a bit of the conversation into GPT-4 and Gemini Ultra and Groq, and asking what DILCOLTK referred to. I realize by the way these models tokenize the inputs, it might not be quite a fair question because they maybe can't "see" every letter.

GPT-4 gave "Dual Income Low Cost of Living LTK", Gemini Ultra gave "Dual Income Low Cost of Living One Tiny Kid" (lol), and Groq suggested "Dual Income Low Cost of Living One Kid Two Kid", so all were admirably close but none quite right.

But phind-70B just now got it right! Color me surprised and impressed.

I also asked it a SwiftUI question I'd struggled with, and which I asked the other models about, and it did I'd say a bit better there as well.

So I guess I'll have to add this to my list of models to try and keep tabs on!

How is this possible?

GPT-4 is supposed to be 8*220B = 1.7T parameters, so it seems unexpected that a 70B model can beat or match it unless it's somehow a much better algorithm or has much better data.

If GPT4 is 220B/8 experts, that would be in-line with 3.5 Turbo being a 20B model, and GPT4 being a 55B activation out of a total 220B parameters.

It is ultimately all speculation, until Deepseek releases their own 145B MoE model, and then we can compare the activations/results

I think the conjecture is that each expert of GPT-4 has 220B parameters, for a total of 1.76T parameters.

This is much better than expected. Switching to chat is also making it feel better for me. I will compare it to GPT-4 in coding tasks over the next month and may switch after that.

In other words: "our 70B finetune is as good as a 8x200B model"

Yeah, right.

I'm not sure GPT 4 is still 8x200B

The one thing we've learnt from the past few months of LLM optimization is that model size is no longer the most important thing in determining LLM quality.

A better training regimen and better architecture optimizations have allowed smaller models to push above their weight. The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

It kinda is, if you want not just performance on synthetic benchmarks but a good coverage of the long tail. This is where GPT4 excels, and also why I pay for it. Transformers are basically fancy associative memories. A smaller model, much like a smaller search index, will not be able to contain as much nuanced information for some hard, immutable, information theoretic reasons.

> The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

I follow your posts and comments here so I'm surprised you say that. The leaderboard at this point is pretty pointless. Lots of ways to "cheat" and get higher ranking there.

I do agree that smaller models have made significant progress, but somethings you can't just solve without adding #parameters and FLOPs. Not to mention, ctx_window is an important factor in code quality, but most OSS models (including llama 2) have pretty limited ctx, despite methods like grp and yarn.

It's more a comment on the capabilities of smaller models, the quality of output outside of benchmarks is always subjective and you'd need something like Chatbot Arena (https://chat.lmsys.org/) to evaluate it more quantitatively. Even after filtering out the common cheat techniques like merges, there are still 7B and 13B near the top, but yes it's still possible to train models on the evaluation datasets without decontamination.

If you look at the Chatbot Arena leaderboards there are still decently-high ELOs for 7B models.

I evaluated many Mistrals for an information extraction task and the merged models were much better than direct fine-tunes. About 5% better.

I agree...

Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct formatting.

I've found that GPT4 (via GitHub Copilot) and Gemini models are better at code tasks like reviewing for logical and functional errors, reasoning about structure and test/edge cases, and refactoring. Gemini is capable of devouring some very large files I've thrown at it.

Phind at times is hampered by whatever it is they're doing in addition (RAG?). It is still phenomenal, though. I regularly find myself using Phind to grok assembly code or learn Typescript.

How do you know that copilot is using gpt4?

I pay for it and for chatGPT and I find copilot much worse.

Looks like Copilot may use GPT4 or GPT3.5 depending on as of yet unpublished criteria: https://github.com/microsoft/vscode-copilot-release/issues/6...

For code review, I tend to engage Copilot Chat which probably uses GPT4 more often? https://github.com/orgs/community/discussions/58059#discussi...

But what if you apply the same level of optimization, same training regimen to the larger models?

Phind-70B is a specialist model, unlike GPT-4. It optimizes for a different function than GPT-4 and therefore needs fewer parameters to learn it.

It's also true that specialist models still need to be sufficiently large to be able to reason well, but we've observed diminishing returns as models get larger.

I mean, it could be as good or better at a lot of reasoning related tasks and just have less baked in general knowledge, in which case it'd make an amazing RAG model if the context length is reasonable.

Phind is great. I hope now they release their latest 34b finetune weights as they did with one of the first versions.

HumanEval can be skipped at this point ...

A couple of questions:

- Can phind run on old Macbooks(2015+) with 8GB RAM? - Is it only for coding purpose?

You bought me :) The only missing feature is mobile app.

Came here to say this: I try to stay away from Google's products and have been using phind and perplexity for the last couple of months. I have to say I am impressed with what you guys are doing and keep up the good work

API?

I think we need a lot better benchmarks in order to capture the real complexity of typical day to day development.

I gave it my typical CI bootstrapping task:

> Generate gitlab ci yaml file for a hybrid front-end/backend project. Fronted is under /frontend and is a node project, packaged with yarn, built with vite to the /backend/public folder. The backend is a python flask server built with poetry. The deployable artifact should be uploaded to a private pypi registry on pypi.example.com. Use best practices recommended by tool usage.

and it generated scripts with docker run commands [1]:

  install_dependencies:
    stage: install
    script:
      - docker run --rm -v $(pwd):/app -w /app/frontend node:14 yarn install
      - docker run --rm -v $(pwd):/app -w /app/backend python:3.8 poetry install

  build_frontend:
    stage: build
    script:
      - docker run --rm -v $(pwd):/app -w /app/frontend node:14 yarn build

  build_backend:
    stage: build
    script:
      - docker run --rm -v $(pwd):/app -w /app/backend python:3.8 poetry build

  deploy_artifact:
    stage: deploy
    script:
      - docker run --rm -v $(pwd):/app -w /app/backend python:3.8 poetry publish -r $PYPI_URL --username $PYPI_USER --password $PYPI_PASSWORD
    only:
      - master

Which, feels more like "connect the dots" or a very rough sketch, that might end up completely replaced. Commands in general seem ok (yarn install && yarn build, poetry build && poetry publish), but the docker run could be better expressed simply as a "image: " attribute of each job. I asked about that and I've been given general "why docker is useful" non-answer.

It also introduced a parallel build stage: frontend and backend are built at the same time, but in my question, I deliberately introduced serial dependency: the frontend code goes into the backend project. The parallel approach would be of course better, if it would correctly construct the end artifact before uploading, but it doesn't do so. Also a bit surprisingly, node install and poetry install could actually run in parallel as-is, but the generated code runs serially.

It uses outdated versions of tools. Python 3.8 seems still ok and used in many online examples due to some compatibility quirks with compiled libraries, but node 14 is more than 3 years old now. Current node LTS is 20.

For comparison, here's the chatgpt4 version [2] :

  prepare:
    stage: prepare
    image: python:3.9
    script:
      - apt-get update && apt-get install -y nodejs npm
      - npm install --global yarn
      - cd frontend && yarn install
      - cd ../backend && poetry config virtualenvs.create false && poetry install

  build-frontend:
    stage: build-frontend
    image: node:latest
    script:
      - cd frontend
      - yarn install
      - yarn build --outDir ../backend/public

  build-backend:
    stage: build-backend
    image: python:3.9
    script:
      - cd backend
      - poetry install --no-dev

  package:
    stage: package
    image: python:3.9
    script:
      - cd backend
      - poetry build
    artifacts:
      paths:
        - backend/dist/*

  deploy:
    stage: deploy
    image: python:3.9
    script:
      - pip install twine
      - cd backend
      - twine upload --repository-url $PYPI_REPOSITORY_URL -u $PYPI_USERNAME -p $PYPI_PASSWORD dist/*
    only:
      - main

Not perfect, but catches a lot more nuance:

- Uses python as base image, but adds the node to it (not a big fan of installing tools during build, but at least took care of that set-up)

- Took care of passing the artefacts built by the frontend; explicitly navigates to correct directories (cd frontend ; ... ; cd ../backend)

- --no-dev flag given to `poetry install` is a great touch

- Added "artifacts: " for good troubleshooting experience

- Gave "only: main" qualifier for the job, so at least considered a branching strategy

- Disabled virtualenv creation in poetry. I'm not a fan, but makes sense on CI

I would typically also add more complexity to that file (for example using commitizen for releases) and I only feel confident that gpt4 won't fall apart completely.

EDIT: Yes, gpt4 did ok-ish with releases. When I pointed out some flaws it responded with:

  You're correct on both counts, and I appreciate your attention to detail.

Links:

- [1] https://www.phind.com/agent?cache=clsye0lmt0019lg08bg09l2cf

- [2] https://chat.openai.com/share/67d50b56-3b68-4873-aa56-20f634...

May you please. PLEASE

post as to how the chat option was polluting stuff, and the pipeline of whatever made that happen.

Make this less opaque. (actually just post how pollution happens, as well as a definition to pollution as pertains to such.

Diminishing trust is at stake.

> We love the open-source community and will be releasing the weights for the latest Phind-34B model in the coming weeks. We intend to release the weights for Phind-70B in time as well.

I don't understand the utility of this comment?

Reply on news.ycombinator.com