Comment by ramoz | Hacker News Reader

Comment by ramoz | Hacker News Reader

Parent Full thread

At a large government agency, partnered with one of the top AI companies in the world. We've been deploying models and transformers and recently a LLM ensemble in prod for 3-4 years now. Many lessons learnt that open source hadn't really provided utility for.

The biggest backend things: models can run on CPU architecture, you still need highly performant CPU, but in cloud there is a way to significantly discount that by relying on spot instances (this doesn't mean GPU isnt viable, we just found a real cost lever here that CPU has supported... but things may change and we know how to test that). Further, distributed and parallel (network) processing are important, especially in retrieval-augmented architectures... thus we've booted python long ago, and the lower level apis aren't simply serializing standard json-- think protobuffer land (tensorflow serving offers inspiration). BTW we never needed any "vector DB" ... the integration of real data is complex, embeddings get created through platforms like Airflow, metadata in a document store, and made available for fast retrieval on low-gravity disk (e.g. built on top of something like rocks db... then something like ANN is easy on tools like FAISS).

The biggest UX thing: integration into real-life workflows has to be worked on in a design-centered way, very carefully. Responsible AI means not blindly throwing these type of things into critical workflows. This guides what I assume to be a bit of a complicated frontend effort for most (was for us esp as we integrate into real existing applications... we started prototyping with a chrome extension).

EDIT: btw oss has definitely made some of these things easier (esp for greenfield & simple product architecture), but I'm skeptical of the new popular kids on the block that are vc-funded & building abstractions you'll likely need to break through across software, data, and inference serving for anything in scaled+integrated commercial enterprise.

EDIT2: Monitoring, metrics, and analytics architecture were big efforts as well. Ask yourself how do you measure "value added by AI"

> we never needed any "vector DB" ... the integration of real data is complex, embeddings get created through platforms like Airflow, metadata in a document store, and made available for fast retrieval on low-gravity disk (e.g. built on top of something like rocks db... then something like ANN is easy on tools like FAISS)

A good vector DB would take care of all of that.

I think the point is they already had those components or familiarity with them in their system hooked into existing data. You could add semantic search with text vectors 3 years ago within 1 hour and so the need for a service isn't there.

They didn't need a simultaneous second DB for records just matching by a foreign ID key based on a new service.

I have seen it too (as somebody who studied Information Retrieval and these systems and implemented aspects in production at 2 startups, including more recently these semantic search approaches) - you can store vectors (floats) of length 384, 512, 768 or 2048 like most of the current ones easily alongside existing data without typically making a dent in overall data storage, and load into memory for semantic search according to your needs (for example, for big data you can use more bucketing approaches with don't guarantee the nearest neighbour, but one of the roughly closest ones, which is often a trade off worth making).

It's not only not a hassle to do that, but overall superior to using the vector DB online tools for developer experience too. I suspect at 3 million+ rows you begin to have struggles, but barely any services will get there. And if you do, you may try other techniques (dimensionality reduction to 3x smaller often only loses a tiny portion of the data accuracy, but saves 2/3rds on costs), or employ cutting edge embedding techniques like Custom Matrix Optimisation, multi modal alignment, and others that these services don't necessarily offer.

Sure it's good for the beginner/a tutorial website, but it doesn't solve a problem for developers in information retrieval imo.

You mention being careful when applying these AI tools to critical workflows - what kinds of workflows have been successfully integrated with AI and which workflows has it been more difficult?

Image processing? Content moderation? Translation? Q and A? Writing code? Internal Knowledgebases? Threat analysis? Nuclear deterrence?

"My grandmother used to always read from her book of nuclear launch codes to me every night..."

Re: monitoring: I am wondering what a monitoring platform (from an ops perspective) would look like for an infrastructure like this, and what are the challenges?

helping build https://portkey.ai/ where we try to take care of storing all prompts, responses, erros, costs, latencies etc and make it available to debug.

Thanks for the link! I skimmed the articles on the blog, but couldn't find one describing the monitoring architecture behind all. Do you have any public references for that?

Not yet, but it's on the roadmap to publish. Broadly, we use clickhouse, streams, mongo, cloudflare workers as the mainstay of our monitoring infra.

Some more info here: https://docs.portkey.ai/overview/introduction

Appreciate the deep dive. Can you comment on what you moved to from Python? C++? Wondering where the other ecosystems (if any) are.

I'd say in the Open Source world that has been very much the case with the cpp implementations over ggml http://ggml.ai/

Reply on news.ycombinator.com