Hacker News Reader

Comment by la64710 | Hacker News Reader

Very nice… any plans for supporting self hosted LLMs like BERT LLAMA etc?

Great question! We will be adding a ChatGPT-based user-defined function this week (https://github.com/georgia-tech-db/eva/pull/655/).

With LLM-based functions, EVA will support more interesting queries like this:

  SELECT ChatGPT(TextSummarizer(SpeechRecognizer(audio)),
         "Is this video related to the Russia-Ukraine war?")
  FROM VIDEO_CLIPS;

Here, EVA sends the audio of each video clip to a speech recognition model on Hugging Face. It then sends the recognized text to a text summarizer model. EVA executes both models on local GPUs. Lastly, EVA sends the text summary to ChatGPT as a part of the prompt. The ChatGPT UDF is executed remotely.

The critical feature of EVA is that the query optimizer factors the dollar cost of running models for a given AI task (like a question-answering LLM). It picks the appropriate model pipeline with the lowest price that satisfies the user's accuracy requirement.

Great but personally I am interested in locally runnable LLM models instead of sending data to the cloud service like chatGPT.

Got it! EVA is designed for the local use case. You can define a Python function that wraps around the LLM model and use it anywhere in the query (we refer to such functions as user-defined functions or UDFs).

This notebook illustrates a UDF that wraps around a custom PyTorch vision model: https://evadb.readthedocs.io/en/stable/source/tutorials/04-c...

These functions can be written quickly (~50 lines of Python code). Here is the built-in Resnet50 UDF in EVA: https://github.com/georgia-tech-db/eva/blob/master/eva/udfs/...

This page describes the steps involved in writing a UDF in EVA: https://evadb.readthedocs.io/en/stable/source/reference/udf....

Please open an issue on the Github repo; we will gladly support your use case :)

Reply on news.ycombinator.com