ianpurton

I use dbmate and for a deployment I package up the migration files into a docker container which runs and then applies the changes.

Firstly I use a devcontainer for development so I know my versions line up. dbmate uses .sql files which also makes things a lot easier.

You can see my setup here https://github.com/bionic-gpt/bionic-gpt

Have a look at the CONTRIBUTING.md to get an idea of the dev workflow.

Are some of these models/techniques open source?

It would be nice to integrate them into on prem solutions.

Is it possible to buy Groq chips and how much do they cost?

So, what would be the hardware setup for this?

Can it run on 1 GPU and swap between experts.

Then put that in a docker container for people like me who are too impatient to read 10 step documents.

Sort of. I'm working on https://bionic-gpt.com/

There are a lot of moving parts. So for example we had someone trying to use the software and it didn't really work for their use case. https://github.com/bionic-gpt/bionic-gpt/issues/106

It may work for your use case.

Rust on nails. https://rust-on-nails.com/

Apparently there's a hole in the wall and the police are looking into it.

The problem with nuclear is that their is so much FUD around it that it becomes not just expensive but very difficult to build.

Mainly due to new regulations meaning projects have to re-engineer themselves before they are even complete. Leading to delays and cost overruns.

The first thing they should work on is their landing page.

Because for sure I'm their target market but I've got no idea what it does.

I spent 10 seconds trying to figure it out then I was gone.

The typography plugin which comes out of tailwind gives a clean readable experience in my opinion. https://tailwindcss.com/docs/typography-plugin

Running LLama 2 70b a Mac Studio gets 13 tokens per second.

2 x 4090 is more like 38 tokens per second.

But it's harder than that. As the 2 x 4090 is using https://llm.mlc.ai/ which is highly optimized so perhaps the Mac Studio inference could be better when tuned.

Then a lot of production inference engines use continous batching, which means they perform really well when processing multiple prompts at a time.

I've just finished writing an article on hardware required for LLM's in production https://bionic-gpt.com/blog/llm-hardware/

Would be interesting to see how this performed running Llama 2 7b.

We're working on an open source chat gpt that runs on kubernetes. https://github.com/bionic-gpt/bionic-gpt

And you're right many companies are looking at running LLMs in house.

SOC2 is a process, i.e. become compliant then get an audit.

So become compliant and put the label on your website. Don't get an audit.

With any customers when asked always mention that you have followed all the procedures and are waiting for an audit.

40,000 a year for a support contract.

https://bionic-gpt.com/services/

Rust supports more software engineering best practices than Go.

Some people say the learning curve for Rust is steep. But, in a way you're learning how to write safer code using the latest in language techniques.

As an architect working on LLM applications I have these criteria for a database.

- Full SQL support

- Has good tooling around migrations (i.e. dbmate)

- Good support for running in Kubernetes or in the cloud

- Well understood by operations i.e. backups and scaling

- Supports vectors and similarity search.

- Well supported client libraries

So basically Postgres and PgVector.

> my co-founder wants to pocket these speaking engagements and not run them through our business.

I can't imagine this would generate a lot of income for the CEO. I've no idea but I guess speaking engagements are in the low thousands or often you do it for free just to get some exposure for the company.

I would let this play out and see what happens.

I think the first Rust ML framework to gain massive traction will be the one that supports quantisation out of the box.

At the moment there's no Rust ML library that does what llama.cpp or ggml can do.

This will run on a PC with 16GB of ram (possible less) It's a 7b model quantized.

docker run -it --rm ghcr.io/purton-tech/mpt-7b-chat

I'm currently writing up a deployment architecture for LLM's and the API question is answered here https://fine-tuna.com/docs/choosing-a-model/model/

Basically you can get a Docker container that will publish an Open AI API compatible end point. You can then choose the model that sits behind that API.

As deployment will be in Kuberenetes we will clusters with GPU resources to maxz out performance but we're not there yet.

Really nice.

I would take the d out of the -itd, otherwise I get no logs and can't see that it downloads pytorch_model before giving me access.

Otherwise, great job.

From you home page, I couldn't figure out exactly what you do.

Can you parse PDF into embeddings for a vector database?

Did you mean user interface or API?

Anyway for UI you could look at chainlit, for API some of the models are already getting wrapped up in an open ai compatible rest interface.

See https://github.com/go-skynet/LocalAI

> Meanwhile, they'll happily pay Deloitte or Accenture 3x as much as what I charge, and then take the misguided ramblings of a junior-sold-as-senior at face value, not knowing the difference.

This is so true.

It's also very difficult to stop as the consultancy gets the project and the client doesn't get to interview the people who actually turn up.

Web components are already built into the browser. Lit is like a wrapper around those.