Hacker News Reader

Potential solutions:

1. AI video watermarks that carry over even if a video of the AI video is taken

2. Cameras that can see AI video watermarks and put an AI video watermark on the videos of any AI videos they take

Umm, customary, tongue-in-cheek reference to McCarthy's proposal for a 10 person research team to solve AI in 2 months (over the Summers)[1]. This was ~70 years ago :)

Not saying we're in necessarily the same situation. But it remains difficult to evaluate effort required for actual progress.

[1]: https://www-formal.stanford.edu/jmc/history/dartmouth/dartmo...

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Yes, this seems to be a common concern. We're trying to see how best to crisply address it. But yes, Khoj can be used even with your internet turned off

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

We're still trying to figure out the right balance between configurability and ease of use/maintenance.

The 7B version was a decent enough starting point in terms of what it can answer (and way fewer folks can run a 13B on their machine).

If you really want you can just replace the 7B model file with the 13B one under the ~/.cache/gpt4all directory on your device and it should just work.

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Yes, of course

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

It should work but not sure how fast. Best way to find out would be to try it.

We also have some fixes and perf improvements we plan to release later today in version 0.10.1

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Thanks for the feedback. Does your machine have a GPU? 32GB CPU RAM should be enough but GPU speeds up response time.

We have fixes for the seg fault[1] and improvement to the query speed[2] that should be released by end of day today[3].

Update khoj to version 0.10.1 with pip install --upgrade khoj-assistant later today to see if that improves your experience.

The number of documents/pages/entries doesn't scale memory utilization as quickly and doesn't affect the search, chat response time as much

[1]: The seg fault would occur when folks sent multiple chat queries at the same time. A lock and some UX improvements fixed that

[2]: The query time improvements are done by increasing batch size, to trade-off increased memory utilization for more speed

[3]: The relevant pull request for reference: https://github.com/khoj-ai/khoj/pull/393

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Khoj provides exactly that; it runs on your machine, none of your data leaves your machine and it has a web interface to chat

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Firstly, my previous statement needs quite a lot of clarification:

Khoj works more like an incremental, natural language version of org-agenda-search (or projectile) rather than isearch.

You've to configure which files it should index first. You can then use natural language to search those files with a search-as-you-type experience.

This is not the same as isearch that just searches the current file for keyword matches.

Second, khoj doesn't index source code and the default search models don't work well with code files.

But yeah someone should implement a natural language isearch as a standalone tool (as suggested by parent comment). It'd be super-useful

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Khoj exposes a local API. Hopefully that makes it easy to integrate with Raycast, Alfred (or even Spotlight)?

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Yeah, this would be ideal for Mac users. Just need to look into what is required and how much work it is

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Thanks for the feedback! Someone else mentioned this issue the other day as well. I'll fix this issue on the landing page soon

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

To use Chat with non-english sources you'll need to enable OpenAI. Offline chat with Llama 2 can't do that yet.

And Search can be configured to work with 50+ languages.

You'll just need to configure the asymmetric search model khoj uses to paraphrase-multilingual-MiniLM-L12-v2 in your ~/.khoj/khoj.yml config file

For setup details see http://docs.khoj.dev/#/advanced?id=search-across-different-l...

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Thanks! :)

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Buggy for setup? We've done some improvements and have desktop apps (in beta) too now to simplify this. Feel free to report any issues on the khoj github. I can have a look.

Yes, we don't do optimizations on the query encoding yet. So SBERT just re-encodes the whole query every time. It gets results in <100ms which is good enough for incremental search.

I did create a plugin system, so that a data plugin just has to convert the source data into a standardized intermeditate jsonl format. But this hasn't been documented or extensively tested yet.

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Khoj chat with Llama2 will not work with non-english languages though. You'll have to enable OpenAI for that

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Search should work with Slavic languages including Russian and 50+ other languages.

You'll just need to configure the asymmetric search model khoj uses to paraphrase-multilingual-MiniLM-L12-v2 in your ~/.khoj/khoj.yml config file

See http://docs.khoj.dev/#/advanced?id=search-across-different-l...

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

You do not need to break the bank to use Khoj for local chat, 16Gb RAM should be good enough

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Oh yeah, I used to use Google Desktop too. It was awesome for it's time

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Khoj uses Llama 2 7B, 4bit quantized. So it just needs 3.5Gb of RAM (GPU or System) [1].

Khoj and your other apps need more RAM themselves, so practically 8GB of System or GPU RAM should suffice.

Khoj has been tested with CUDA and Metal capable GPUs. So Nvidia and Mac M1+ GPUs should work. I'm think it'll work with AMD GPUs out of the box too but let me know if it doesn't for you? I can look into what needs to be done to get that to work.

[1]: The calculation is [params] * [bytes] GB RAM, so 7 * 0.5 = 3.5Gb

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

That would be pretty awesome. Building a daily web history summarizer as a browser extension shouldn't be too much work. I bet there's something like that already out there.

Having something that indexes all your digital travels and makes it easily digestible will be gold. Hopefully Khoj can become that :)

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Oh neat, thanks for sharing that! Having a 32K offline model is pretty promising. Let me test out how it performs

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Ah, you're right, forgot to mention that. We use the Llama 2 7B 4 bit quantized model. The machine requirements are:

Ideal: 16Gb (GPU) RAM

Less Ideal: 8GB RAM and CPU

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

1. I hear you on going back to lightweight native apps. Unfortunately the Python ecosystem is not great for this. We use pyinstaller to create the native desktop app but it's a pain to manage.

2. The web UI isn't required if you use Obsidian or Emacs. That's just a convenient, generic interface that everyone can use.

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

That.would.be.awesome! Khoj isn't their yet, but that actually shouldn't be too far away if you give it a voice interface and terminal access.

Of course, having it be stable enough to not `rm -rf /` soon after is definitely not part of the warranty

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Thanks for the pointer, yeah the website content has gone stale. I'll try update it by end of day

Khoj is using the Llama 7B, 4bit quantized, GGML by TheBloke.

It's actually the first offline chat model that gives coherent answers to user queries given notes as context.

And it's interestingly more conversational than GPT3.5+, which is much more formal

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Yeah being able to search and chat with PDF files is quite useful.

Khoj can index directory of PDFs for search and chat. But it does not currently work with scanned PDF files (i.e not with ones without selectable text).

Being able to work with those would be awesome. We just need to get to it. Hopefully soon

110on Show HN: Khoj – Chat offline with your second brain using Llama 2

Wow that's an intense use-case. I don't know how but we'd love to be able to support this.

If you can collate your notes into markdown or some such, then messy notes can be handled, at least using Khoj with GPT3.5+.

Do let us know how we can help out and what your current biggest pain-points are?