Show HN: Bodhi App – Run Open Source/Weights HuggingFace LLMs Locally

Hi HN,

I'm excited to share Bodhi App, a tool designed to simplify running open-source Large Language Models (LLMs) locally on your laptops. While we currently support M2 Macs, we plan to support other platforms as our community grows.

# Problem

To use LLMs, you typically need to purchase a subscription from providers like OpenAI or Anthropic, or use OpenAI API credits with compatible Chat UIs. These options can not only burden you financially, but also raise data security and privacy concerns.

Many laptops are capable of running powerful open-source LLMs, but for non-tech users, setting them up can be challenging.

# Solution: Bodhi App

Bodhi App allows you to run LLMs on your own hardware, ensuring data privacy and cost savings. Our goal is to bring the power of LLMs to everyone.

Built with non-tech users in mind, Bodhi App ships with a simple Chat UI, making it easy to start conversing with an LLM. It also exposes OpenAI-compatible APIs, enabling other apps to use LLM services without relying on external providers.

# Features

Bodhi App currently supports:

1. Running GGUF format open-source LLMs from the Huggingface repository.

2. An in-built Chat UI to quickly start conversations with an LLM.

3. A powerful and familiar CLI to download and configure models from the Huggingface ecosystem.

4. Exposing LLMs as OpenAI-compatible APIs for use by other apps.

--

# Feature Comparison: Bodhi App vs. Ollama

## Bodhi App

- Targeted at non-tech users.

- Includes a simple Chat UI to get started quickly.

- Integrates well with the Huggingface ecosystem:

  - Use Huggingface repo/filename to run a model.

  - Use tokenizer_config.json for chat templates.

- Currently only supports Mac M2.

## Ollama

- Requires some technical insight.

- No inbuilt Chat UI.

- Requires baking the model using a custom process:

  - Modelfile

  - Golang template to specify chat templates.

- Supports various OS platforms.

Bodhi App leverages the Huggingface ecosystem, avoiding the need to reinvent the wheel with Modelfile etc., and making it easier to get started quickly.

# Quickstart

Try Bodhi App today by following these simple steps:

```bash

brew tap BodhiSearch/apps

brew install --cask bodhi

bodhi run llama3:instruct

```

# Documentation and Tutorials

- Technical docs on GitHub (README.md, docs folder): https://github.com/BodhiSearch/BodhiApp

- YouTube playlist covering Bodhi App features in detail: https://www.youtube.com/playlist?list=PLavvg7KIktFI1ZaFc2nLe...

# Conclusion

We would love for HN to try out Bodhi App and provide feedback. You can reach us through:

- Raising an issue on GitHub: https://github.com/BodhiSearch/BodhiApp

- Connecting with the developer on Twitter: https://twitter.com/AmirNagri

- Leaving a comment on our YouTube tutorials: https://www.youtube.com/playlist?list=PLavvg7KIktFI1ZaFc2nLe...

Please show your support by starring the repo on Github.

Thank you for your time!

Best, The Bodhi Team