Comment by entrope | Hacker News Reader

Comment by entrope | Hacker News Reader

Parent Full thread

A lot of the article focuses on problems induced by a 512-token input limit. For example, one needs a lot more chunks with such a small input, especially with overlap. I realize that some embedding models do have input contexts that small, but 8K and 32K are fairly widely supported and reduce chunking-related problems.

For languages like English, there's also usually a lot of redundancy within a text, so 512 tokens might not give a very clear indication of the context. Lots of documents have similar introductions (like "#include <foo.h>\n") that make short contexts and truncation particularly harmful.

Also, "Nothing in the document past that point can ever be retrieved, and nothing anywhere told you." This is user-hostile behavior, even if they didn't want to admit to users that the auto-embedding support was poor.

Finally, the paragraph later on about truncation being "what you already have" reads like Claude talking to the developer, not like a vendor talking to users. But sure, maybe this is a good default for a database searching page titles, chat logs and Xeets?

"Chunking" has a bad history in IR going back through the history of the TREC conference. For instance you might index the whole document or index sections or paragraphs and somehow aggregate the paragraph hits up into document hits. It's one of the many things that doesn't help performance even if you think it did, and it may hurt.

The answer is "make the context window as large as you reasonably can" because you have to have enough (con)text in the window for the system to decide what the words mean and if you don't have enough of it you won't get it right.

Author here. Fair critique, thanks.

On the 512 tokens: that's the window of the model we benchmarked with (all-MiniLM-L6-v2), not a claim about embedding models in general. The article does mention text-embedding-3-small's 8,192 window, and the same setup works with 32K models like Qwen3-Embedding. If your documents fit the window, the advice stands: keep truncate.

A bigger window makes chunking easier, not irrelevant. max_tokens defaults to the model's own limit, so with an 8K model you get fewer, larger chunks and overlap matters a lot less.

Two reasons we still chunk even when the document would fit:

1. One vector per document is a summary of the whole thing, so a short, highly relevant section gets averaged away by everything around it. One vector per chunk turns the question into "does this document contain something close to the query?", with the doc scored by its best chunk. Your #include example is exactly that case: the first N tokens of every file look alike, and what distinguishes them is further down. That's the "deep content" split in the benchmark — truncate got 55% recall@5, recursive got 83%.

2. Cost. Transformer embedding time grows superlinearly with input length, so pushing a whole 8K or 32K document through a local model on CPU costs far more than embedding it as 512-token chunks. Remote APIs bill per token either way.

That said, you're right that our numbers only show the effect against a 512 window. We should rerun the same benchmark with an 8K and a 32K model. I'd expect the gap to shrink but not disappear, and that's worth measuring rather than assuming.

On "nothing anywhere told you": agreed, silent truncation is bad behavior. That line describes what Manticore used to do (and what most embedding pipelines still do by default), not a defense of it. truncate is still the default because multi-vector output needs a different column type, so it has to be opt-in.

On "what you already have": fair, that sentence reads badly. It means "the old default, unchanged", not "good enough for you". We'll reword it.

"If your documents fit the window, the advice stands: keep truncate." "Two reasons we still chunk even when the document would fit:"

Which advice do you stand by? Obviously, very short content doesn't need chunking, so let's consider a document that fills 75% of the input context.

When chunking, your cost overhead (per token) goes up as the number of new tokens per chunk goes down. That's an argument for longer chunks, although the averaging/smearing point argues for not going too long.

Embedding calculations are effectively prefill: on my cheapo local inference system (32 GB AMD R9700 + 8 GB AMD RX 7600), the older 8 GB card goes about 80% as fast as the bigger card for Qwen3-Embedding-4B (a bit over 19 chunks/second on my usual corpus, blog posts+comments that are mostly well under 32K tokens). So I would suggest that anyone who is limited by CPU embedding models could benefit from even a small local GPU.

For your blog post, I would suggest an explanation of the chunking modes, either in the blog post or as a hyperlink to the docs about them. "truncate" and "sentence" are fairly clear, whereas the others are not. (If "mean" just computes the mean of the embeddings, that seems like a poor choice. The arithmetic at https://www.johndcook.com/blog/2026/09/16/coffee-milk-latte/ might work for single words, but seems likely to break down at the document level. "recursive" and "fixed" are opaque, at least to me.)

If/when I index my team's documents, I will consider a content-aware chunking that fits as many sentences, paragraphs or sections as possible into each chunk, with overlap determined by the level at which the chunk finishes. Content-agnostic chunking is easier to code and more generic, but indexing should respect a document's internal structure.

Just so you know, your comment was automatically hidden (“dead”) until I vouched for it now. Same for most of your recent submissions. Actually it’s probably because of your (exclusively self-promotional) submissions that your comments and submissions get hidden.

Their comments are also AI-generated, for some reason.

Did you really need to use an LLM to write this comment?

Reply on news.ycombinator.com