Makes me curious on how useful a tiny AI model based just on wikipedia would be on mobile phones.
Especially if it could also cite sources.
Especially if it could also cite sources.
You would have an hallucinating make-up-some-fake-facts machine on your phone.
I don't think that's definitely true, if you build the system around it well. It would look something like this:
1. User asks a question. LLM extracts key concepts from that question to use as search terms.
2. LLM triggers a search of Wikipedia, getting back snippets of pages, each with their page identifier.
3. LLM is fed those snippets along with the user's question and instructions to cite pages that it uses content from.
4. LLM generates a response which includes formatted citations. This response may be complete garbage, but...
5. Your code can at least confirm that the citations correspond to the pages that you fed into the LLM.
I've seen this approach work well with larger models. The open question is if it could work with smaller ones.
The 7B models (Mistral and its variants in particular) are getting VERY effective. I'm confident they could mostly-work for the above sequence... and you can just about run a 7B model on a phone.
The bigger question for me is if you could get this to work with a 3B model, since those are much more mobile-device friendly than 7B.
LLMs on mobile phones can be surprisingly capable. I've been having fun trying out this one on my iPhone: https://llm.mlc.ai/#ios
Getting that to work with a local copy of Wikipedia - where it can essentially run a "search_wikipedia(...)" function any time it needs extra information - would definitely be feasible, using this pattern: https://til.simonwillison.net/llms/python-react-pattern#user...
I think it would need to use regular text search though - you could try calculating embedding vectors for the whole of Wikipedia but even with a small embedding model I doubt you could fit those on the device, they'd take up too much space.
Maybe impractical, but seems like it could potentially be more seamlessly integrated. Keeping in mind:
- Many models already include WP in their training, at least parts of it.
- The parts that are missing or a subset of what’s missing could be supplemented via finetuning.
- Data is commonly compressed about 100x during training (it’s lossy) so the total size might still run on a phone.
You’d then have a fully capable LLM, but enhanced to be comprehensively encyclopedic.