Comment by gpugreg | Hacker News Reader

Comment by gpugreg | Hacker News Reader

Parent Full thread

Agentic workloads are somewhere around 1%/0.5%/98.5% input/output/cached tokens. Cached tokens are pretty much free for inference providers (if they implement sparse and compressed attention properly) and throughput for input tokens is much higher.

Lets assume that you've got 2 million input tokens, 1 million output tokens and 98.5 million cached tokens to process. That would cost 2 * $0.14 + 1 * $0.28 + 98.5 * $0.0028 = $0.8358 with DeepSeek API pricing.

For comparison, it would take 2M / 8000 + 1M / 800 = 1500 seconds to process this amount of tokens with the linked framework, which is about $0.83 when we assume $2/hr for one MI300X.

However, other inference providers have 10 times higher prices for cached tokens, which results in a comfortable margin.

And we should not discount that DeepSeek also gets paid in data, which is probably more valuable to them.

And I believe that this framework still has some room for optimization for generation with high batch sizes.

Your math is a bit funny if you're assuming the 1/0.5/98.5 ratios: you doubled input and output tokens but not cached. If you double cached tokens to match your original ratio it works out to around $1.11, and if you 10x the cached token cost it's around $6.08.

Based on your $0.83 estimate, the margin isn't great. This is within shooting distance of "at cost" which is probably pretty close to what DeepSeek is operating with, ignoring the value of the data they're collecting of course.

> And I believe that this framework still has some room for optimization for generation with high batch sizes.

If that optimization can bring this scenario closer to $0.50 then it gets pretty compelling, otherwise I'm not confident.

Oh, I messed up. Half-way through, I thought it would be a good idea to double the numbers so I don't have to deal with half millions, but forgot to also double the 98.5. Unfortunately, I can not edit it anymore.

I think the margins of DeepSeek may be a bit better than with this vibe-coded framework here, since they had the liberty of optimizing their models for their own hardware.

For DeepSeek V3, they claimed a cost profit margin of 545%: https://github.com/deepseek-ai/open-infra-index/blob/main/20...

At the time, open frameworks were not anywhere close to achieving that number. Not sure whether they caught up. The software wizards at DeepSeek are quite skilled.

> should not discount that DeepSeek also gets paid in data, which is probably more valuable to them

That's agentic feedback loops for training, right? Any more detail on this, such as how they actually tell whether that data is good or not? That seems like a very hard problem, and like the value of that data is low compared to just building their own, controlled RL gyms.

Agents usually start with ingesting the existing code base, and DeepSeek can use those code bases for pretraining. And they will have filters on top of that to throw out garbage.

I am not sure how they are using the data for post-training, but there probably are ways to get signal out of it, e.g. sentiment analysis when the user begins cursing at the agent, or checking whether the user continued another session with the generated code, or started a new session with the same starting point as before, i.e. they git-stashed.

Generally, you can train on data that is quite bad (e.g. the entire internet). It will still work, but take much longer compared to clean data.

> e.g. sentiment analysis when the user begins cursing at the agent, or checking whether the user continued another session with the generated code, or started a new session with the same starting point as before, i.e. they git-stashed.

Thank you for elaborating, that's already useful. Anywhere I can learn more about this? I'm very interested in it!

To learn about sentiment analysis, I'd look for related datasets and then look at recent code, e.g. here: https://www.kaggle.com/datasets?search=sentiment+analysis

For more LLM-specific stuff, you can pick some agent trace dataset on https://huggingface.co/datasets?format=format%3Aagent-traces and check out what people are doing with it (usually linked on the right when you click on a dataset).

And of course https://scholar.google.com/ for research papers.

Reply on news.ycombinator.com