Hacker News Reader

Comment by pjot | Hacker News Reader

Can you talk about how you implemented search-as-you-type? Doing so with semantic search seems tricked given the roundtrips needed to compute embeddings on the fly (assuming the use of OpenAI embeddings)

View on HN

sure - implementing a search-as-you-type experience with an ai-powered feature was what i wanted to do as well. it doesn't use embeddings at the moment. when you type a short query like 'openai,' it simply runs a basic query using Typesense. however, if you enter a question-like query, such as 'how to llimit api cost,' it transforms it into multiple queries, like 'budget' and 'limit.'

in the self-hosted version, it use the CHAT_COMPLETION_MODEL env variable for selecting the llm model. in our cloud version, we use a fine-tuned version of 4o-mini that we will eventually move to a smaller model like llama8b or even 1b.

Got it! I saw this in the code and assumed you were using embeddings def evaluate(input: shared.EvaluationInput): ds = Dataset.from_list(input.dataset) metrics = [metric_map[metric] for metric in input.metrics]

    llm = ChatOpenAI(
        model_name=shared.LANGUAGE_MODEL,
        base_url=os.environ["OPENAI_API_BASE"],
        api_key=os.environ["OPENAI_API_KEY"],
    )

    embeddings = OpenAIEmbeddings(
        model=shared.EMBEDDING_MODEL,
        base_url=os.environ["OPENAI_API_BASE"],
        api_key=os.environ["OPENAI_API_KEY"],
    )

that piece of code is for llm response evaluation, but we are not really using it at the moment.

Reply on news.ycombinator.com