Sidekick: Local-first native macOS LLM app
github.com
github.com
Amazingly generous that it’s open source. Let’s hope the author can keep building it, but if they need to fund their existence there is precedent - lots of folks pay for Superwhisper. People pay for quality software.
In a past tech cycle Apple might’ve hired the author, acquired the IP and lovingly stewarded the work into a bundled OS app. Not something to hope for lately. So just going to hope the app lives for years to come and keeps improving the whole way.
I’m bullish on an AirPods-with-cameras experience
It'd be great if the app would only have read access to my files, not full disk permission.
As an end-user, I'm highly concerned that files might get deleted or data shared via the internet.
So ideally, Sidekick would have only "read" permissions and no internet access. (This applies really to any app with full disk read access).
Also: why does it say Mac Silicon required? I can run Llama.cpp and Ollama on my intel mac.
I'm running it right now and macOS didn't ask for any permissions at all which afaik means that it cannot access most of my personal folders and definitely not the full disk. Am I missing something?
This is only true if it uses App Sandbox, which is mandatory for apps distributed through the App Store, but not necessarily everything else
Optionally, offload generation to speed up generation while extending the battery life of your MacBook.
Screenshot shows example, mentions OpenAI and gpt-4o.
anythingllm https://github.com/Mintplex-Labs/anything-llm
openwebui https://github.com/open-webui/open-webui
lmstudio https://lmstudio.ai/
OpenWebUI a little bit more harder to setup (need docker), but interface very similar to OpenAI
Somewhat related, one issue i have with projects like these is it appears like everyone is bundling the UX/App with the core ... pardon my ignorance, "LLM App interface". Eg We have a lot of abstractions for LLMs themselves such as Llama.cpp, but it feels like we lack abstractions for things like what Claude Code does, or perhaps this RAG impl, or whatever.
Ie these days it seems like a lot of the magic in a quality implementation is built on top of a good LLM. A secondary layer which is just as important as the LLM itself. The prompt engineering, etc.
Are there any attempts to generalize this? Is it even possible? Feels like i keep seeing a lot of good ideas which get locked behind an app wall and no ability to switch them out. We've got tons of options to abstract the LLMs themselves, but i've not seen anything which tackles this (but i've also not been looking).
Does it exist? Does this area have a name?
They are pluggable across more than just LLM itself.
• B2C: wants vertically-integrated tools that provide "middleware" plus interface. Doesn't want to dick around. Often integrates their own service layer as well (see e.g. "character chat" apps); but if not, aims for a backend-integration experience that "knows about" the quirks of each service+model family, effectively commoditizing them. The ultimate aim of any service-generic app of this type is likely to provide an "subscription store" where you purchase subscriptions to inference services through the app, never visiting the service provider itself.
• B2B (think "using agents to drive pseudo-HFT bots for trades in fancy financial instruments that can't be arbitraged through dumb heuristics"): has a defined use-case and wants to control every detail of both "middleware" and backend together. Vertically integrates their own solution — on-prem inference cluster + custom-patched inference engine + business logic that synthesizes the entire prompt at every step. Doesn't bother with the "chat" abstraction other than as part of several-shot prompting.
• B2B2C: wants "scaling an inference cluster + engine + model deployment" to be Somebody Else's Problem; thinks of an "app agent experience" as the deliverable they want to enable their business customers to achieve through their product or service; and thus thinks of "middleware" as their problem / secret sauce — the thing they will build to enable "app agent experiences" to be created with the least business-customer effort possible. The "middleware" is where these B2B2C businesses see themselves making money. Thus, these B2B2C businesses aren't interested in paying some other middleman for a hosted "generic middleware framework as a service" solution; they're interested in being the only middleman, that captures all of the margin. They're interested in library frameworks they can directly integrate into their business layer.
---
For an analogy, think of the "middleware" of an "easy website builder" service like Squarespace/Wix/etc. You can certainly find vertically-integrated website-builder services; and you can also find slightly-lower-level library components to do what the "middleware part" of these website-builder services do. But you can't find full-on website-builder frameworks (powerful enough that the website-builder services actually use them) — let alone a white-labelable headless-CMS + frontend library "website builder builder" — let alone again, a white-labelable headless-CMS "website builder builder" that doesn't host its own data, but lets you supply your own backend.
Why?
Because B2C businesses just want Squarespace itself (a vertically-integrated solution); B2B businesses don't want an "easy website builder", they want a full-on web-app framework that allows them to control both the frontend and backend; and B2B2C businesses want to be "the Squarespace of X" for some vertical X, using high-ish-level libraries to build the highest-level website-building functionality, while keeping all of that highest-level glue code to themselves, as their proprietary "secret sauce." (Because if they didn't keep that highest-level code proprietary, it would function as a "start your own competitor to our service in one easy step" kit!)
---
The only time when the "refined and knowledge-enriched middleware abstraction layer -as-a-Service — but backend-agnostic!" approach tends to come up, is to serve the use-case of businesspeople within B2B orgs, who want to be able to ask high-level questions or drive high-level operations without first needing to get a bespoke solution built by the engineering arm of said org. This is BI software (PowerBI), ERP software (NetSuite), CRM software (Salesforce), etc.
The weird / unique thing about LLMs, is that I don't think they... need this? The "thing about AI", is precisely that you can simply sit an executive in front of a completely-generic base-model chat prompt, and they can talk their way into getting it to do what they want — without an engineer there to gather + formalize their requirements. (Which is not to say that the executive can get the LLM to build software, correctly, to answer their question; but rather, that the executive can ask questions that invoke the agent's inbuilt knowledge and capabilities to — at least much of the time — directly answer the executive's question.)
For LLMs, the "in-context learning" capability mostly replaces "institutional knowledge burned into a generic middleware." Your generic base-model won't know everything your domain-specialist employees know — but, through conversation, it will at least be able to know what you know, and work with that. Which is usually enough. (At least, if your goal was to get something done on your own without bothering people who have better things to be doing than translating your question into SQL. If your goal is to work around the need for domain expertise, though... well, I don't think any "middleware" is going to help you there.)
In short: the LLM B2C use-case is also the LLM "B2Exec" use-case — they're both most-intuitively solved through vertical integration "upward" into the backend service layer. (Which is exactly why there was a wave of meetings last week, of businesspeople asking whether they could somehow share a single ChatGPT Pro $200/mo subscription across their team/org.)
Hoping that these kinds of tools will run well in these scenarios.
As a suggestion to the author, please try to make it verifiably local only with an easy to set option.
Maybe the author could make a note of that in the README.
That said, it seems built toward "Cheat on your homework" and doesn't reliably surface information from my notes, so I uninstalled it.
The readme says:
> Give the LLM access to your folders, files and websites with just 1 click, allowing them to reply with context.
…
> Context aware. Aware of your files, folders and content on the web.
Am I right in assuming that this works only with local text files and that it cannot integrate with data sources in Apple’s apps such as Notes, Reminders, etc.? It could be a great competitor to Apple Intelligence if it could integrate with apps that primarily store textual information (but unfortunately in their own proprietary data formats on disk and with sandboxing adding another barrier).
Can it use and search PDFs, RTF files and other formats as “experts”?
One of the screen shots shows a .xlsx in the “Temporary Resources” area.
Also: I haven’t checked, but for a “Local-first” app, I would expect it to leverage Spotlight text importers from the OS, and run something like
mdimport -t -d3 *file*
(I wrote one of those Apple API SDKs)
I had a project in the past where I had hundreds of PDF / HTML files of industry safety and fatality reports which I was hoping to simply "throw in" and use with Open WebUI, but I found it wasn't effective at this even in RAG mode. I wanted to ask it questions like "How many fatalities occurred in 2020 that involved heavy machinery?", but it wasn't able to provide such broad aggregate data.
I'm not sure there's a way to get what a lot of people want RAG to be without actually training the model on all of your data, so they can "chat with it" similar to how you can ask ChatGPT about random facts about almost any publicly available information. But I'm not an expert.
But I get this far:
/Users/aa-jv/Development/InterestingProjects/Sidekick/Sidekick/Logic/View Controllers/Tools/Slide Studio/Resources/bin/marp: No such file or directory
It seems there is a hand-built binary resource missing from the repo - did anyone else do a build yet, and get past this step?
I'm immediately suspicious of such binary resources, however.
A single file seems to finish quickly, but folders (even with just a few files) seem to be very slow.
... so image generation is not fully offline?
This tool looks like it could be worth a try to me, but only if I'm sure I can run it into a mode that's fully offline.
Only want it for some code so it looks like it can be fully offline, but it's worth being paranoid about it.
The only safe option is if it's guaranteed to not leave my machine, so for this app, to disable anything that has a chance of exfiltrating data.
Or to rephrase: would you go to court with the contents of that link as evidence that you haven't inadvertently published someone else's proprietary data in some external database?
Right now only code editors and Claude support MCPs, but we'd love to see more clients like Sidekick