Show HN: Tokensift, an open-sourced token-efficiency linter for LLM prompts

github.com

6 pointsritenv3 comments

Hello, author here.

Most prompts include personalizing with a user's data. These prompts grow, and there is quite a bit of extra content that can be removed. UUIDs, pretty-printed JSON, ISO timestamps, repeated instruction blocks, etc., and all of it tokenizes to cost a lot.

tokensift is a linter for that. It tokenizes the prompt with a real encoder the specific model uses: exact BPE counts for OpenAI and a calibrated estimate for Claude with the measured error shown. Then, it runs a set of rules against it and reports where the tokens are going, along with the dollar cost per finding.

Works as a library or a CLI, and runs in CI.

Current state: 20 rules, OpenAI and Claude supported, no Gemini yet.

Does it work also on multi modal LLMs?

It does work partially. TokenSift can analyze the textual parts of prompts sent to multimodal models, but doesn’t calculate or optimize image/audio/video token usage. Non-text payload calculation is something I’d like to add.

Reply on news.ycombinator.com