Were LLM providers prepared for agentic programming tools like Cline?
imgur.com
imgur.com
For those unfamiliar, Cline is an “Autonomous coding agent right in your ID”, open source and available for free on GitHub. It’s very similar to GitHub edits, but does a much better job of sequential reasoning (in my experience, at least). You give it an API key and your code base, and it handles putting together the instruction prompt with the relevant portions of your code. This tends to work better with smaller codebases that dont bump up against the context window, but there’s nothing stopping anyone from using it with a much larger codebase. And from the numbers, it’s seems like that’s happening quite a bit.
This means that these agentic coding apps are now responsible for monthly token usage that has to be orders of magnitude larger than the numbers that were getting hit prior to their release. (Maybe I’ll see if I can find historical openrouter data to back this up)
So my question is - how well are OpenAI and Anthropic handling this, and can they continue to scale at the pace that these tools are growing in popularity? Obviously both companies offer models on GitHub Copilot, and cursor, Sourcegraph Cody, etc have been around for a while, so it’s not like Cline popped up overnight, but still - but given the economical and supply chain challenges with scaling AI data centers, there’s no way this isn’t putting pressure on their poor ops teams.