No-Code Coding with Git, Xcode CLI and GPT4
github.com
github.com
The goal is "no-code coding". The developer here (a friend ;) ) doesn't know how to write swift, and I doesn't build mobile apps professionally. Can they use ChatGPT 4 to help write code, and how frustrating is that? The goal is to automate as much development as possible by utilizing a Large Language Model (LLM).
The demo “agent” starts with an existing Hello, World IOS app made with the XCode template. It develops iteratively, where each iteration consists of the following steps: One turn creates the prompt, obtain the LLM response, parse the response (to be just code changes), updates the code files, print the code diff to developer's CLI for approval, run the app simulator for approval, then commits everything at the end if the developer so chooses.
Starting out, the objective was to use the LLM to edit a single file passing the single code file into the prompt along with a feature request and some boilerplate about how the output is to be structured. That was possible. However, that wasn't very useful because IOS apps are currently organized as a set of a bunch of files and it's rare that any single feature would edit a single file. So the objective became to edit multiple files. The key challenges there were:
- Parsing the output so that the correct code can be routed to the correct file (Seems mostly possible, especially with gpt4)
- Sometimes chatgpt does not return full code files as a way of saving tokens
- How to prompt the model better to get better structured output? Boilerplate prompt...
- How to post-process the output more effectively
The current demo shows editing multiple files, adding files that don’t currently exist e.g. new views. It’s not shown in this video, but the system was used to go from Hello, World! to an app that allows for editing and saving edits to selected files.
Key Takeaways:
- Prompting LLM is hard because you have a moving target. The output is dependent on the prompt (prompt text, hyperparams, boilerplate). Evaluating the input prompt is done by analyzing the output you get. So how do we move forward? Systematize the prompt strategy, hold certain aspects constant, including the output information and data structure. Make small systematic changes to observe the output.
- Several LLM prompt templates are desirable: feature request, exception, feature iteration request.
- Working optimally within GPT 4's 8K context window is important. Finding ways to compress the code project while maintaining enough information to make progress feels icky. But does it work?
- How do other models do? LLama? Claude2? Bard? If anyone is at Anthropic, I would love to get access to evaluate Claude2.