If I were running product at Sentry I'd be working on an agent that opened pull requests to fix your Sentry errors. This would work today with GPT-4 on a nontrivial number of errors/codebases.
If you have a "can't access key 'foo' of undefined" error, adding a check if the object is undefined or not is not a fix. It's simply silencing the error.
The bug might be way, way down in the stack, maybe even in the backend or somewhere else in the client that made the backend store invalid data my mistake.
I highly doubt that the current breed of models are capable of this.
It is certainly not enough for every case, but it is enough (and the models are smart enough) to be useful.
It really isn't.
I just did a test with a random TypeScript file with 400 lines of code which translates to almost exactly 3000 tokens in the GPT-4 token counter app.
This means that we are talking about being capable to analyse a maximum of 43 small / medium sized files.
Each analysis will also cost $1.28 without any additional intermediary costs.