Do you have an estimate for typical token usage for a developer who'd use it as part of their workflow? I'd imagine the costs can rack up fairly quickly if you're not careful.
Two things we're doing this:
- right now our API response contains more than what's required for autocompletion, so there is room for improvement there. And since we focus on team alignment, the goal is to boost the suggestion acceptance rate compared to alternatives. So in the end, fewer calls and lower token consumption.
- since we're working on fully migrating to hostable OSS models of reasonable size, the financial aspect of token consumption should be mostly moved out of the picture to focus on latency.