Hacker News Reader

Comment by benjaminwootton | Hacker News Reader

AWS Bedrock is fairly easy. You can do it in 5 or 6 clicks.

You have to upload your documents to S3, create a “Knowledge Base” then sync your documents into a vector database like OpenSearch or PineCone. You are then good to go via their playground or the AWS API.

I made a video here describing the process, check around 14 minutes in:

https://ensembleanalytics.io/blog/introducing-bedrock-knowle...

Bedrock is a decent product I think. All of the models in one place (apart from the big dogs from OpenAI) and a common API across them.

View on HN

Bedrock is cool, but I found it prohibitively expensive for hobbyists and small companies. On a first glance, it would cost me something like $ 5,000 per month for a simple trained model.

I suspect most of the cost in the OpenSearch vector database?

I agree it is high and it is not exactly serverless. You pay by the hour.

Bedrock itself is charged based on tokens exchanged.

I think Pinecone is a cheaper database for hobby and small business projects, though I haven't looked into it.

No, if you use a custom model (trained with your data), you'll pay around $20 per hour, minimum. That equates to ~$15,000.

You can reduce a lot by committing to a 6-month contract, but it won't get cheaper than about ~$5,000/mo.

That's prohibitively expensive for small projects.

Fine tuning GPT 3.5 is much cheaper.

To be fair, Bedrock is more flexible than OpenAI's fine tuning. It's nice to have at least the option to pay big bucks for this. But it's big bucks nonetheless.

Don’t use OpenSearch it’s way overpriced. Use a cheap-o RDS with pgvector. Everything else is charged like any other LLM by token use.

This has nothing to do with OpenSearch. Using off the shelf models is cheap and pay as you go. Using custom models is what's expensive.

Is there a limit? Could I create a knowledge base with 10,000 documents? 100k? 1M?

https://docs.aws.amazon.com/bedrock/latest/userguide/quotas....

I’m sorry, I don’t understand those limits. It uses a lot of unfamiliar terms like “batch inference” and “modality”. I just want a nice UI that I can give my hard-drive to and then ask it questions.

That’s probably unrealistic at this time

It's doable with Amazon Q, but it's in Preview phase now. https://aws.amazon.com/q/

Is Q the answer to this whole thread? People are talking about AWS Bedrock but that seems like something a startup would build upon and offer something like Q, eh?

I’m going to try it this week so maybe we have a follow up thread??

The documents are encoded as vectors and stored in a database, so I suspect it would be effectively unlimited. You would just pay for storage and compute.

AWS OpenSearch has fairly good integration so you could look up costs for that. It’s not the cheapest AWS service to run and not exactly serverless as you pay by the hour.

Even if you could, the problem is that these documents are first chunked into smaller pages, and then embeddings are created. When you ask a question, the algo searches for relevant chunks and passes them to the LLM's overall prompt. If there are too many chunks, or too many chunks with similar content, the search also coupled with the LLM's limited context window would mean only 1-3 chunks get passed.

This isn't the same as the training data the LLM is trained on. As a result, it doesn't take advantage of the entire document set. So, if you have a billion documents, only 1-3 chunks will be picked for the final answer. When you know a question spans many many documents, the answer is never going to cover that.

You could make a recursive also where you parse all the chunks, generate summaries of those and then pass them to the next chunk sequentially and so on. But you can imagine how expensive and slow that will be. It might still work for you but this is a very lossy approach.

Easy: if > n chunks returned query "too much information, ask user to clarify".

If you have that much data in your vdb, and the user is querying a slice, it just won't do to have them ask something too generic, they need to be asked to be specific.

Ie "what was that thing from last year?" "Sorry, could you be more specific?" "Oh uh the thing to do with the financial reports" "Financial reports, yes, what about them?" "It was something to do with the way we generate them" "Ah yes, the generation of financial reports; last year in December your team leader said on Slack...etc"

How does bedrock satisfy the non-hallucinating requirement?

It doesn't. You need to try and reduce hallucination as much as possible with your prompts, and then benchmark it.

Reply on news.ycombinator.com