jarulraj

:)

Agreed, I just updated it to "Elements of System Design".

I added it now :)

Yes :)

Exactly. It is intentionally open-ended: new "principles" can emerge, split, or retire as the taxonomy evolves. This is just version 1. The grid is a visual index and the fun part is mappin systems to "molecules" in different domains like OS, databases, computer architecture, distributed systems, programming languages, networking, and more..

Yes, I just borrowed the periodic table metaphor. "Elements of System Design" is a better name.

Author here, great question :) If principles are the elements, we can think of each system as a "molecule" with some imagination. For example, an SQL database system has many principles:

1. Abstraction Lifting (Al) + Policy/Mechanism Separation (Pm): SQL states high-level intent with precise semantics, and logical operators are decoupled from physical operators.

2. Equivalence-based Planning (Ep) + Invariant-Guided Transformation (Ig): We apply algebraic rewrites that preserve semantics (e.g., join reordering, predicate pushdown) under stated invariants.

3. Cost-based Planning (Cm): We choose concrete physical operators and join orders using a cost model and so on..

Totally fair.. I am not claiming periodicity here :) I just wanted to use the "periodic table" as a visual metaphor. The goal is to outline a mostly orthogonal set of system design principles and illustrate cross-domain connections to students so that it is easier to compare trade-offs and discuss designs more precisely.

Author here, appreciate the share :) I was not expecting this to get so much attention.

To clarify: this is indeed just a taxonomy of classic system-design principles. The periodic-table styling is a familiar metaphor; there is no claim that principles repeat periodically. The goal was to outline a mostly orthogonal set of design principles and highlight cross-domain connections across computer systems so it is easier to discuss designs precisely. Thanks for all the thoughtful feedback!

It is based on the feedback shared by an undergraduate student -- pretty subjective. For instance, it is possible to graduate with a CS degree without even taking an introductory course on systems.

It is currently not possible to get rigorous summaries of paper chunks using GPT-4.

Wouldn't more semantically related neighbors be retrieved by just increasing K?

Cool project! Can you elaborate on "scalable" AI deployment? We are exploring the data + AI space in EvaDB and would love to exchange notes [1].

[1] https://github.com/georgia-tech-db/evadb

Very cool project, @MatthausK!

What are your thoughts on reducing LLM cost?

We are also exploring LLM-based data wrangling using EvaDB and cost is an important concern [1, 2, 3].

[1] https://github.com/georgia-tech-db/evadb

[2] https://medium.com/evadb-blog/stargazers-reloaded-llm-powere...

[3] https://github.com/pchunduri6/stargazers-reloaded

Nice :)

What were the interesting problems you faced in processing the survey data?

If possible, can you share the prompt?

As we do not have ground truth, we only qualitatively checked for accuracy -- no quantitative metrics. We did find a significant drop in accuracy with GPT 3.5 as opposed to GPT 4.

Are you measuring accuracy with data wrangling prompts? Would love to learn more about that.

Yes, there is definitely a human-in-the-loop element here.

It would be great if you could share an example of the inconsistent output problem -- we also faced it. GPT-4 was much better than GPT-3.5 in output quality.

Great question! We iterated on the prompt for several days and manually verified the results for ~100 users.

The results were pretty good: https://gist.github.com/gaurav274/506337fa51f4df192de78d1280...

Another interesting aspect was the money spent on LLMs. We could have directly used GPT-4 to generate the "golden" table; however, it's a bit expensive — costing $60 to process the information of 1000 users. To maintain accuracy while reducing costs significantly, we set up an LLM model cascade in the EvaDB query, running GPT-3.5 before GPT-4, leading to a 11x cost reduction ($5.5).

Query 1: https://github.com/pchunduri6/stargazers-reloaded/blob/228e8...

Query 2: https://github.com/pchunduri6/stargazers-reloaded/blob/228e8...

Thanks for your kind words, @skeptrune! It was certainly a fun project.

We found some interesting insights. In the Langchain community, ~40% of the stargazers are from India. In the GPT4All community, we found that web developers love open-source LLMs -- more so that machine learning folks :)

Curious if there is any reason why you would not use it?

Thanks for your kind words, @treebeard5440! :)

Hi Andrey. Thanks for your feedback. We should have better emphasized that we are benchmarking Qdrant in local mode. We have updated the post to clarify that Qdrant is being evaluated in local mode. We plan to next evaluate the server mode.

We went with the local mode as several Python AI apps are using Qdrant in that mode based on the suggestion here: https://qdrant.tech/documentation/quick-start/.

We also believe in open-sourced benchmark code. Please find the code here: https://github.com/jiashenC/vectordb-benchmark-and-optimize/....

Neat AI app!

1. What feature extractor is used to derive code embeddings?

2. Would support for more complex queries be useful inside the app?

   --- Retrieve a subset of code snippets
   SELECT name 
   FROM snippets
   WHERE file_name LIKE "%py" AND author_name LIKE "John%"
   ORDER BY
      Similarity(
         CodeFeatureExtractor(Open(query)),
         CodeFeatureExtractor(data)
      )
   LIMIT 5;

The discord link seems to be not working. Just a heads up.

The YOLO example on your Github page is super interesting. We are finding it easier to get LLMs to write functions with a more constrained function interface in EvaDB. Here is an example of an YOLO function in EvaDB: https://github.com/georgia-tech-db/evadb/blob/staging/evadb/....

Once the function is loaded, it can be used in queries in this way:

  SELECT id, Yolo(data)
     FROM ObjectDetectionVideos
     WHERE id < 20
     LIMIT 5;

  SELECT id
      FROM ObjectDetectionVideos
      WHERE ['pedestrian', 'car'] <@ Yolo(data).label;

Would love to hear your thoughts on ChatCraft and a more constrained function interface.

Can LLMs help in cleaning up the logistics database -- like handling different variants of an address? Here is an example: https://medium.com/evadb-blog/augmenting-postgresql-with-ai-....

Nice writeup! Here is another example of using LLMs is to augmenting existing software that we are exploring -- specifically SQL database systems. We are using LLMs inside a SQL query to power a "soft join" between SQL tables for when a correspondence is only implied (e.g. different address formats, etc.).

   --- Create a reference table that maps neighborhoods to zipcodes using ChatGPT
   CREATE TABLE reference_table AS
   SELECT parkname, parktype, 
          ChatGPT(
          "Return the San Francisco neighborhood name when provided with a zipcode. The 
   possible neighborhoods are: {neighbourhoods_str}. The response should an item from the 
   provided list. Do not add any more words.",
       zipcode) 
   FROM postgres_db.recreational_park_dataset;


   --- Map Airbnb listings to park
   SELECT airbnb_listing.neighbourhood
   FROM postgres_db.airbnb_listing
     JOIN reference_table ON airbnb_listing.neighbourhood = reference_table.response;

More details on LLM-powered joins and EvaDB: https://medium.com/evadb-blog/augmenting-postgresql-with-ai-..., https://github.com/georgia-tech-db/evadb

You might be interested in another way to use LLMs inside a SQL query. For example, LLMs can be used to power a "soft join" between SQL tables for when a correspondence is only implied (e.g. different address formats, etc.).

More details here: https://medium.com/evadb-blog/augmenting-postgresql-with-ai-...

Thanks for sharing that observation on customer chatbots.

1. Will that query look like this:

  SELECT LLM("{user_question}", order_info)  
  FROM postgres_data.order_table  
  WHERE user_id = “101”;

2. How will a feature store, like Hopsworks, help in this app?

Shameless self-plug: We are building EvaDB [1], a query engine for shipping fast AI-powered apps with SQL. Would love to exchange notes on such apps if you're up for it!

[1] https://github.com/georgia-tech-db/evadb

Yes, it should be "positive%". That's an interesting LLM-level optimization.

Doesn't this token sampling optimization require using a locally-running model like Llama?

I am presuming that OpenAI doesn't provide direct access to token probabilities in its API.

Interesting observation :)

I guess this is the SQL query you have in mind that uses the LIKE operator:

  SELECT ChatGPT("Respond to the review with a solution to address the reviewer's concern", review)  
    FROM postgres_data.review_table  
    WHERE ChatGPT("Is the review positive or negative?", review) LIKE "%positive%"
        AND location = “waffle house”;

From a query processing standpoint, both queries should have equivalent performance -- unless we build an index over the output of the ChatGPT query in EvaDB, in which case the former query would be faster than this one.

Congrats on the launch, @ukuina! Neat idea. Would love to learn more about how you are querying the summaries/comments database and how you are planning to extend the queries in this app.

Shameless self-plug: We are building EvaDB [1], a query engine for shipping fast AI-powered apps with SQL. Here is an illustrative query for analyzing food reviews stored in Postgres and generating responses for negative reviews:

  SELECT ChatGPT("Respond to the review with a solution to address the reviewer's concern", review)  
  FROM postgres_data.review_table  
  WHERE ChatGPT("Is the review positive or negative? Only reply 'positive' or 'negative'.", review) = "negative"
      AND location = “waffle house”;

It would be interesting to learn about the queries needed for supporting HackYourNews application. Would love to exchange notes on this if you're up for it!

[1] https://github.com/georgia-tech-db/evadb