Hacker News Reader

Comment by ed | Hacker News Reader

Congrats on the launch, it looks like you worked very hard on it.

But I’m an engineer, I read the README and the website, and I still don’t know what Super-duper is.

Is it just a python library? Does it have its own persistence (it must)? It doesn’t appear to be a set of plugins for various DB’s but I could be wrong.

As such I don’t know how I’d use it. It might be helpful to describe the product in more concrete terms.

One of the phrases used in YC is: ACME makes soup taste better. We do it with a seasoning that chefs add to their broth.

Maybe that’s helpful. Explaining a product can be hard!

View on HN

All things aside, it's a framework for building data workflows in Python.

Like taking the data from that source (e.g., SQL), processing them (e.g., pytorch or openai), and storing the results somewhere (e.g., data on Mongo metadata on SQL).

It actually consists of the following: 1. nifty abstractions for Data (e.g., sources, encoders, listeners), Metadata (e.g., vector indexes), Compute (e.g., sync, async, parallel). 2. gluing engine that transparently handles the interaction between components 3. out-of-the-box integrations with established tools (databases, AI models and APIs, compute engines)

This way, you can build customized data layers that sit on top of your database and save you from moving the data to dedicated systems (e.g., vector databases or MLops tools)

For further discussion, feel free to join our slack https://join.slack.com/t/superduperdb/shared_invite/zt-1zuoj...

Ah, it’s a langchain competitor, possibly with better DB support.

One of the nice things about langchain is the code examples, making it easy to get simple services up and running. And because it’s a toolkit I can take what I need and leave the rest.

However, the ecosystem around langchain is really exploding, is there some way you can retool what you have to extend langchain with better DB support, rather than build your own thing?

Indeed both frameworks support model chaining.

However, achieving goals like "training your LLM" or enabling "real-time inference" requires more than just pipelines. For that, we have invested in enhancing compatibility with databases and facilitating parallel computing.

About your last point, I 'm not sure I fully understand. Do you mean to write a guide for moving lang-chain models to superduper? Or to create superduper wrappers for langchain ? Or to move the core functionalities of superduper to langchain ?

The guide, is something have in our immediate plans. The wrappers are under discussion. The latter I don't think it's possible due to architectural differences. For example, superduper is designed with multi-node environments in mind.

By connecting AI models with the data's source (the database) we make it very easy to bring AI to your end-user-facing applications.

SuperDupeDB is really an end-to-end AI development and deployment framework wrapping and integrating your existing data infrastructure. It replaces MLOps entirely as it covers inference and model training.

Reply on news.ycombinator.com