Jan: An open source alternative to ChatGPT that runs on the desktop | Hacker News Reader

Jan: An open source alternative to ChatGPT that runs on the desktop

jan.ai

194 pointsbillybuckwheat60 comments

It does make it easier for the end user who doesn't want to fiddle around with python dependencies, command lines, building C++ projects, etc.

Just install it, point it to a model, and go. Now you have a local LLM.

If you want something more, click the "start server" button and you have a local OpenAI compatible API which you can point more advanced front-ends to.

I was wondering if it uses something like vLLM[0] or Llama.cpp[1].

Seems to be Llama.cpp via 'Nitro', which was discussed here before [2].

[0] https://github.com/vllm-project/vllm

[1] https://github.com/ggerganov/llama.cpp

[2] https://news.ycombinator.com/item?id=38887531

Unless I just can't find it, there seems to be no setting for customizing the prompt format for local models. You can edit the prompt itself, but not the format of the prompt or the subsequent messages. This would make using many models difficult, or give poor results, since they don't all use the same format.

Many LLMs may be run locally with GPT4All...

https://gpt4all.io/

And MLC puts them on your phone, too.

https://llm.mlc.ai/

GPT4All makes it annoyingly difficult to run any other than their "approved" models. I'd like to kick the tires on a whole host of random GGUF quantizations on Hugging Face, please.

I've poked around the doc, not sure if Jan can do that better.

In the mean time, I use text-gen-ui (Oobabooga) as a back-end and have it run with `--api` to use the front end of my choice.

Not at my computer right now to double check.. but doesn't GPT4All's "Browse" button in the model list let you pick a locally downloaded model?

I got say I’ve been using LLM studio as it exposes the models in the ui as well as through a local open ai compatible server so I can test different models against my workflows locally.

I try some LLM on my notes and well... They was unable to give me insights that are hard to spot, like follow the flaw of notes identifying patterns, find similar notes from the past and so on. In ALL cases classic tags/riprgrep full-text search was far quicker and equally or more effective.

Long story short: LLMs might be useful on hyper big mass of information, like a new kind of search engine that try do achieve a semantic goal mimicking it. But not more than that IMVHO. Marginally LLMs might help computer-illiterate to manage their files, seen https://www.theverge.com/22684730/students-file-folder-direc... but I doubt they can go any further for the next 5+ years at least.

LLMs might help my disorganized approach to files...

They've been very useful in quickly answering common questions using a too-large-to-manually-scan knowledge base in my experience at my job, and I don't consider myself or my colleagues "computer-illiterate".

That's follow the "might be useful as a new kind of search engine", though it might be a sign of an a bit messy KB. The issue of potential hallucinations however is still there so even such usage, a different search engine, demand extra attention.

It's not a free critic to those who have designed, implemented and trained LLMs, it's just the observation that practical usage is far less than the advertised one and it's still not much good. It's still an advancement, a good thing to have, the start of a revolution, but still far from being what many dreams.

Would be nice if they listed system requirements. Their docs just say coming soon …

most of their docs say coming soon. and their whole wiki.

honestly feels like site this was launched a couple of days too soon.

Their LLM is still generating copy for the website..

Is this a fork of "LM Studio"? The UI is suspiciously similar, even down to the layout of the labels.

LM studio is closed source, so no

This looks interesting. I would love a comparison between this product and LM Studio.

Running LLMs locally always feels so awesome!

I'm going to assume this is not an Australian company...

https://www.youtube.com/watch?v=2akt3P8ltLM

Looks like they are based out of Singapore

The default model (Mistral Instruct 7BQ4) is woke. I asked it the following:-

Write a short poem in admiration of black people

Write a short poem in admiration of brown people

Write a short poem in admiration of asian people

Write a short poem in admiration of white people

It immediately replied with a poem for all except white people where it's response was:-

"I'd be happy to write a poem in admiration of all people, including those who identify as White." Lol

This looks awesome. Trying it out. Suggestion, can we please change the "Download Jan for PC" to perhaps just "Download" or "Download for Desktop" or whichever that makes sense but not "PC". I almost move away thinking this is Windows, thus not for us.

I recently stumbled on https://mindmac.app which is a non-subscription app that uses multiple AI tools (not just OpneAI). Looks Promising.

Like the others in the comments, I've tried https://www.typingmind.com (via SetApp).

Sindre Sorhus have a pretty stable Native App https://sindresorhus.gumroad.com/l/quickgpt

These are some of the really good ones. I'm tending more towards trying out the likes of MindMac just for the fact that I can plug and switch between multiple tools.

Shameless plug: I built a native client called BoltAI[0]. Unlike other clients, I prioritize UI, UX and performance.

Give it a try if UI & UX is important to you.

[0]: https://boltai.com

Sure. Why Not. Trying it out.

My use case is especially for my daughter so I can just plug in my OpenAI API and let her ask away.

This looks great relative to others (very similar to MacGPT), and I particularly like how advanced settings are available but tucked away behind discoverable affordances.

It's interesting that you have team pricing.

Can the Team leverage shared system prompts and/or assistants from a OneDrive-for-Business (SharePoint) folder or GitHub repo?

If not, what makes it "Team" instead of just individual?

Hi. Actually I don’t have a pricing plan for teams yet. It’s still under (heavy) development. I changed my headline to reflect the direction I want to take this year (focus on teams)

And yes, some of my customers wanted team and collaborative features like shared prompt, internal plugins and integrations, RAG on internal documents…

But I haven’t launched these team features yet.

Are you interested in this? Would love to talk to you if it’s something you’re looking for.

From the screenshots it looks like there is an activation limit, with a maximum of four devices. Reading the license, I could not confirm this. Is there a limit, and if, what is the maximum?

Sorry for the confusion. I need to improve my pricing page.

The license is per user, and can be used on maximum 3 devices. I figured this is enough for most users. If you have more devices or need a custom license, please send me an email (my email is in bio)

What's the value proposition for TypingMind as a commercial product ($3500 to run locally for 5 seats)?

But let me contrast that last "native app" with Machato and MacGPT:

== Machato ==

Machato is feature-full for system prompts and transcripts, connecting to to OpenAI, Claude, and any "server" endpoint that's OpenAPI API compatible, and surfacing parameter and token settings per conversation right on your text entry bar. You can also point a given conversation to a local ollama endpoint such as Mixtral 8x7B and it works as well.

The best feature is the selective forking and suppression of exchanges within conversation threads.

https://untimelyunicorn.gumroad.com/l/machato

== MacGPT ==

MacGPT is highly integrated throughout MacOS, and works with either OpenAI key or a ChatGTP Pro login. It's quite similar to BoltAI mentioned elsewhere in this thread, but in addition to the OpenAI key based mode, also works with a ChatGPT Pro subscription in a ChatGPT web UI pop-up.

https://www.macgpt.com/

Github does not count as open source: this is not actually open source.

This project is open source: it is released under the Affero GPL, an OSI approved license [1].

[1] https://opensource.org/license/agpl-v3

How is this better than gpt4all?

From what I see, it has the benefit of offering less functionality, more corpojargon and a more 'intuitive' ui.

Still hoping we'll eventually stop using Fibonacci to show off recursion, because that's one of those examples where the maths might be expressed as recursive relation, but the implementation should never be =)

Good AI would go "you don't want that, that's horribly inefficient. Here's an actually performant implementation based on the closed-form expression".

Nah, good AI would run in the compiler and optimize the recursion into something fast.

What is your preferred example for teaching a beginner to use recursion?

I use Jan to run Mistral locally. It works well for what I need (which amounts to playing with models).

Been using this with a few models like gemma etc for a week now.

HN got any good LLM suggestions to run with this that are equivalent or better than GPT-3.5 / claude?

I'm looking to use its api with LLama Index

Quite a lot of polish and bragging about stars and tweets for an open source project. Is there hidden monetization of some sort? Perhaps VC funding?

According to their page https://jan.ai/team/ they aim to bootstrap.

Awesome thanks.

seems like Ollama with GUI? MindMac with Ollama offers more functionality tbh!

I don't see anything about it reading local documents like exel, pdfs, or docs. Anyone see how this is accomplished?

Is it implied anywhere? That's a feature I'd love and also why I haven't bothered delving into LLMs very much; I didn't know there were any that could locally index your library and train on that data. I'd love to ask it a question and have it reference my local ebook library.

It probably doesn't. The only one that read PDFs for me was the Nvidia ChatRTX. It would be easy to add modules from pip that do this but you'd have to code up the input pipeline. It's not terribly difficult but it is definitely not point and click.

Did this have any way to point at a folder of markdown files and RAG at it?

Where does the model come from?

Afaict, it doesn't have any inbuilt model, you just download one yourself or hook up to someone's API.

Scroll down on the main page:

01 Run local AI or connect to remote APIs

02 Browse and download models

These kinds of apps are becoming dime a dozen. It would be nice to know how this one differentiates itself. Not obvious from the website.

It seems like that until you actually try to use them. Not many are actually polished, support formatting, history, and multiple endpoints. There's lots of trivial apps abandoned after a few days, but what are the actually functional, good quality alternatives to this one? (That don't pass your query/answer through a third-party for data collection)

I use https://www.typingmind.com/. It is paid, but I've found it to be a reliable front end to OpenAI/Claude/Google, supporting everything you mention. I haven't done any hyper detailed security audit but after watching network requests I'm pretty confident it's not sending my chats anywhere except to the relevant provider endpoints.

Considering how much I use it, I've found it to be well worth the cost. The creator is pretty on top of model/API changes.

i’ll second that recommendation… i use it through the SetApp store and i’ve been very pleasantly surprised by its documentation and ability to work with most services.

It's not a standalone app though. There's lots of web interfaces, but that's not the same. (I mean, it's a cool thing, but not what jan.ai is)

There is a desktop app available (I mean it’s basically a wrapper around the web UI, but still).

Fantastic product and excellent team!

Reply on news.ycombinator.com