kevinlu1248

We're building a webhook services on FastAPI + Celery + Redis + Grafana + Loki and the experience with setting up every service incrementally was miserable, and even then it feels like logs are being dropped and we run into reliability issues. Felt like something like this should exist already but I couldn't find anything at the time. Really excited to see where this takes us!

The new Sweep assistant is also interactive! You can request for a plan, approve the plan to have Sweep run through each of the proposed files to edit. We found that it was really difficult to iterate with an agent that only responds every 20 min.

It probably makes it easier for many companies to move off of OpenAI since they won’t need to drastically alter their codebase.

The other problem is with latency: Deepseek 34B on A100s seem slower than GPT-4 but perhaps it will be better on H100s.

We have tried many of the current open source models but unfortunately the only model whose capability is close to GPT-4 is Deepseek and unfortunately Deepseek can’t follow our specified format and is very sensitive to prompt changes.

I feel like at least for me, when I want to build fast, writing the implementation and having AI generate the unit tests to double check and catch edge cases is easier. I also don't use the unit tests as a verification but just to ensure the util functions don't deviate.

Writing unit tests is also more boring so I prefer writing the implementation.

They say it's semantic search but it looks more like lexical search

How has your experience with function calls been? We tried doing code generation and making ChatGPT generate diffs and it seems to perform worse than the March edition.

This makes me believe simulation theory even more tbh. Quantum mechanics exist to fuse operations, altogether making simulating our universe more computationally inexpensive.

Might be wrong but I think it started with Jupyter/MatplotLib + Python's permissive license. Heard it started all with a physics professor unable to use Matlab for teaching.

This is amazing! I think for production and rapid development use-cases though we just use XML for information extraction. It's extremely easy to parse with regex and rarely do the models make mistakes since the start and end tokens are uncommon. At least this is just for the OpenAI model which are different from the use cases in this ShowHN.

There's a few tools out there like AgentGPT (https://github.com/reworkd/AgentGPT, although it's a more conversational interface), and (https://github.com/logspace-ai/langflow) and others. I think most developers definitely prefer a code-first interface though like a library but haven't found one that's great yet. We've used them in the past but didn't have the best experience so would love to hear if anyone has worked with a library they found really flexible.

Aider's awesome and we're using a few of the ideas from the aider blogs like ctags and the search-and-replace editing prompt. I'm wondering, is anyone experimenting with fine-tuning Llama 2 to get it reply in this search-and-replace format? Excited to see open-source models catch up to GPT-4 on code editing.

Thanks for the suggestion! CLI tools can be simple to deploy and setup but are limited in the long run. Running the logic server side allows us to perform way more validation, such as GitHub Actions, linting, and pre-commit hooks. You can do this in your terminal, but it's not as easily parallelized and the process is not as easily persisted. There are definitely some tradeoffs to be made.

Also, the onboarding flow for GitHub Apps is significantly smoother and is a familiar interface for many people. We built a CLI-installed tool previously and there were a ton of versioning and environment-related problems etc, with 3 OS's to support. Unfortunately, it becomes harder to focus on building the best tool for our users if our focus is too broad.

To your last point, you can pry into our codebase and our blogs to see what we do under the hood; "Watch the magic happen" is just referring to the simplicity of our installation process. Also, we only store the logs for debugging and they only get persisted 30 days (which is also how long OpenAI stores them for). We gave a lot of thought to designing our search engine that doesn't store any code in plaintext. Check it out here: https://docs.sweep.dev/blogs/search-infra

Are you referring to the installation.md? It's a bit of a misnomer, we intended for it to be for post-installation instructions for redirect after you install.

I think for teams that want to move fast in a non-critical environment (health, finance etc.) something that works 90% of the time is fine. Getting to 95% takes twice the amount of time but does not provide twice the value. When the 5% difference becomes the difference maker we can fix it later.

Further, we're adding better test systems to Sweep. For now, you can just comment to get Sweep to cover the edge cases and write tests. Happy to take any other feedback.

Yup, it's a bit frustrating since it's a problem with LLM, RLHF and fine-tuning for chat. In fact, we also added in the prompt to not say that it did testing. I find that in general it seems really difficult to tell a language model (especially 3.5) to not do something.

The self-review generally catches stuff like this since we tell it that this code is written by an inexperienced developer, so that Sweep becomes more critical.

Just changed the color to royal blue via https://github.com/sweepai/sweep/pull/932

I think they meant that now that an AI can write the tests they can bring themselves to write enough to hit the 100% coverage. And I think importance of coverage just depends on if you want to build fast or have better maintenance, but I could be wrong since I usually only write e2e tests at most.

Thanks, and I'm glad to hear! I'm Will's co-founder btw. Just wondering, what's the PR about?

Yup! We used to use https://synclinear.com/ but you can also use Zapier to automatically redirect Linear issues to GitHub. It's a nice experience since we also had a Discord to Linear hook.

This is interesting, I'll take a look. My main concern with running this in production is that there is more text data in the world than code. Further, a pure tree-manipulation model has less explainabilities since you can always ask GPT-4 what it's thinking.

For editing code there's a decent chance of syntax errors or undefined variables since it's only modifying a subset of the code.

Ya I think forcing correct syntax at the generation level likely will not be extremely beneficial. At Sweep, we iterate the language models on linters and type-checkers using GitHub Actions and it yields better results.

Thanks! The repo is just the backend that runs the GitHub webhooks. We used to have a "chat with your code" client but stopped supporting it. Now it's only the GitHub interface with creating tickets and comments.

Yup it's based on the aider blogs. They're perfect for our use case and are very reliable compared to our old attempts.

Feel free to use the algo! Happy to help and reach out if you need help with implementation!

For others in this thread, this algo is now publicly accessible in LlamaIndex at https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89b...

Update: this algo is now publicly accessible in LlamaIndex at https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89b...

Here's some of the papers: https://arxiv.org/abs/1911.09983 and https://aclanthology.org/2021.findings-acl.384.pdf