Hacker News Reader

Any timeframe when it will be released to the public?

We are in the middle of developing and app and we are not able to do it with the limited context window of Open Ai. We already submitted the request of access.

View on HN

There are tricks you can do to better utilize the smaller context window, such as sub-summaries and attention tricks. That's how there are already products on the market that consume entire big PDF's and let you query them. Granted, a larger context window would still work better, but it's possible to do.

it's using "overlapping chunking" methods and it usually works for generic PDFs. It really falls apart on technical documents, SOPs and research articles where you need to get context from chunks way above. Using vector DBs also doesn't work well bc you have to twiddle around with window size / overlappy-ness, which changes depending on what kind of paper you're uploading. It's a mess and takes too long

The problem is that making a summary of a text of 100k token costs 2$ using Davinci.

What are the commercial applications of mega context window LLMs at current prices? I would guess mainly legal. And what strategies would you rely on to reduce the accumulating costs over the course of a session?

Reply on news.ycombinator.com