Problem solving across 100,633 lines of code – Gemini 1.5 Pro Demo [video]
youtube.com
youtube.com
A system for converting a natural language specification document into a formal specification would be interesting.
The codebase is 100k lines, but the tasks it gave it seemed to be focused on just hundreds of examples. The examples are probably largely independent, so it doesn't seem like this is really flexing anything a relatively simple RAG approach with a much smaller context window couldn't handle. The prompts said "the demo that ...", so it's a matter of identifying the demo in question and looking at just that code, which is a much smaller necessary context. There was the "use the GUI approach from other examples" task, which kind of gets there, but that's kind of another distinct little bit of code.
In other words, while the codebase has lots of lines, the actual inference across them seemed to use relatively few of them, and identifying the relevant lines didn't seem that hard to me based on the tasks given. That means it could be done with some retrieval and a much smaller context window.
From the title, I thought it would be loading 100k into the context and then asking some deeper questions like "find the bug" that spans several function calls or something like that. Something that wouldn't be trivial to accomplish with current techniques.
I simply don't believe the model is that good. Otherwise, maybe try to compete with OpenAI directley?
For Microsoft it doesn't make a difference. They are taking their own cash, investing it in OpenAI, and then turning right around and booking it as revenue. As a bonus it makes Google look wrong-footed. But fundamentally Microsoft doesn't care how much money they torch doing this.
The code changes are the most common tutorials you can find on the web. Adding a speed slider, the terrain tutorials are literary called "height maps" and focus on making it taller or flatter.
This one seems to be super straightforward about timeliness and capabilities, but the examples might be a bit simpler than people think. This is pretty amazing but like someone else said you could achieve similar results from rag due to the lack of novelty in these questions and the fact that each dealt with pretty independent examples as opposed to using custom code developed elsewhere in the codebase.
Meta, but the speaker sounds eerily close to Mark Zuckerberg.
But the input was still divisible into self-contained little bits -- so this is still somewhat different from dumping the full source code for a database engine into it, and having it answer questions about, say, where foreign key constraints are implemented -- or, more dramatically, how several different parts of the codebase work together to implement, say, transaction isolation levels.