netsroht

Sure, just consumer grade products: an ~8yr old 16 core AMD Threadripper CPU + 64 GB RAM, Samsung NVMe drive(s) and obviously an nvidia RTX 4090 (watercooled). Only the GPU is important though.

I have been looking for a good local setup for a while now. Qwen 3.8 27b is really good for a dense model of this size IMO. I already had an RTX 4090 and I forked ninfer [0] with the obsession to squeeze everything out of this card for this model. Results: 149 tok/s decode speed (aggregate with concurrency about ~270 tok/s) with prefill speeds faster than 2500 tok/s. And all of this with full 262k albeit quantized context. Fast prefill speed is really important when launching multiple clients such as opencode or pi at the same time and especially if they launch subagents. This is why I also implemented a caching tier so computed contexts can be faster loaded from RAM (or disk). Speeds feel almost like with official SOTA openai or anthropic models.

Im currently measuring a pareto front in J/tok in order to set power limits of this card without sacrificing too much performance. Since we are talking about full power draw of ~480W which is fine during the day (with solar panels) but during night when the sun doesn't shine (even with a battery) I'd like to limit this a little bit.

[0] https://github.com/tensorninja/ninfer-4090

Remember when openai was too afraid to release the full GPT-2 model (this one had only 1.5B params) because humanity apparently wasn't ready for it. Look where we are just a couple of years later. I really admired them back in the day for openai gym and PPO etc.

Gadgetbridge added support for Garmin watches recently [1]. All data is stored on your Android phone with no internet connectivity required and you can even export the sqlite DB so you own your sensor data. The UI isn't as nice as Garmin's but it does its job.

[1] https://gadgetbridge.org/basics/topics/garmin/

Always cool seeing stuff in this space.

Regarding "zeitgeist", about a year ago I built something similar called https://zeitgaist.ai which also incorporates other sources like Mastodon, Bluesky, some subreddits etc.

Thanks for the links. Using a disposable email with crypto payments and occasionally generating a new account to unlink from previous searches could be a viable intermediate solution.

Also, I found this link [1] in the thread you mentioned. They seem to have implemented something like that.

[1] https://metager.de/keys/help/anonymous-token

I get your perspective. A lot of us just want a search engine that serves the user first, not advertisers, especially at the results level. It's about function over strict privacy for many--everyone has their own privacy threshold.

But it's also about digital data autonomy. It's not just about avoiding surveillance over sensitive searches, but having control over our data's destiny. Even mundane data, in aggregate, can sometimes be used in ways we can't predict.

I'm not a cryptography expert, but from my research, shouldn't it be possible to verify quota on ZKPs server-side? Essentially, the server doesn't need to know the specifics of the user's identity, just that they possess a valid token and haven't exceeded their quota.

You can use search engines like Google without being logged in. When combined with tools like uBlock Origin and Cookie AutoDelete, it becomes more challenging for them to build a singular profile about a user, especially one tied to payment methods such as credit cards.

I genuinely appreciate what Kagi is doing, and I'd absolutely be willing to pay for their service, because if you're not paying for a service, you're the product. I trust companies to uphold their privacy promises, but "Trust is good, but proof is better." ;)

Being logged in while making search queries in search engines poses significant privacy risks. The searches can paint a comprehensive profile of the user, and these data often remain stored for extended periods. There's a chance this information might be shared with third parties. Coupled with other user data, these logged-in searches can pave the way for targeted advertising, sophisticated predictive analysis, and potential exploitation by governments or malicious entities. In the event of data breaches, the user's logged-in search histories can be exposed. Furthermore, users typically don't have clear insight into how their data is utilized when logged in.

I hope Kagi introduces an anonymous access feature. For instance, it could incorporate zero-knowledge proofs (ZKPs). These are cryptographic techniques where one party (the prover) can confirm to another (the verifier) that a claim is accurate without disclosing any additional information. This is especially beneficial for authentication scenarios where it's essential to avoid sharing extra details.

To implement zero-knowledge authentication for quota API access:

1. Token Creation:

- Each month, users receive a token tied to their identity and quota.

- The token can be split for use on multiple devices using cryptographic methods.

2. API Access:

- Clients present a zero-knowledge proof (ZKP) to confirm they have a valid token and haven't used up their quota. The server verifies this without seeing the exact details.

3. Client Synchronization:

- Each client tracks its quota usage.

- Synchronization can be peer-to-peer or through a centralized, encrypted server to prevent double spending of the quota.

4. Quota Renewal:

- Monthly, old tokens expire, and new tokens are issued.

Challenges:

- ZKPs can be resource-intensive.

- Token security is crucial; there should be a way to handle lost or compromised tokens.

- The system should prevent quota "double-spending" across devices.

- If a centralized server is used for synchronization, it should operate with encrypted data.

This way Kagi would only know who their customers are but not what kind of searches they make.

Thanks for your input. This website should neither compete with nor replace regular journalism. What I try to achieve here is to be able to break free from social media silos where usually people are in kind of bubble. No one can read this many comments and people usually tend to read only comments / conversations where they align with their believes. Hence, I try to highlight different view points along with contrasting opinions (across several different social media platforms) to get an overview--not necessarily fact-based. These stories aren't supposed to push any agenda down anyone's throat.

Since this project just went live I'm still figuring out how to communicate that.

Because I intentionally process them with a very rudimentary "cartoonizer" in order to distinguish from a regular news articles and to emphasize that these stories are not written by humans. I don't know yet whether this helps.

Is my new project [0] also part of the problem? I'm still unsure myself because LLMs also allow us to process data in unprecedented ways. In my specific case, I auto generate stories to highlight different view points based on what people are saying about hot controversial topics on social media.

What's your opinion?

[0] https://zeitgaist.social

I regularly get these infinity captchas on Firefox as well. A couple of days ago I noticed that switching to a different Firefox container let's me pass the captchas.

When I started working on Zeitgaist [1] I immediately recognized that biased information is the main problem. I'm currently thinking about attaching additional information to the sources I present like ground [2] is doing with political spectrums. I really like your idea for the news algorithm. If you or someone elsw wants to build something like that on top of Zeitgaist with me, don't hesitate to contact me.

[1] https://zeitgaist.ai [2] https://ground.news/

I think he's just emphasizing that OpenAI is in fact not open, thusly it's crossed out.

AI is a very broad term. Most search engines use some kind of ML for building their search index and also for ranking because it works very well.

> Do you want to avoid LLMs answering your search? I have not seen that widely adopted at all.

It's starting to get more though. The brief answers that sometimes show up right beneath the search term will most likely get improved by leveraging LLMs.

I agree that showing the prompts will break the usability flow. I'm currently thinking about a way that let's users see the reasoning behind the AI agent - maybe in form of prompts if they explicitly enable it - for my current project [1].

Unlike Bing chat etc., I at least show the detailed sources with contents from web searches and social media comments that have been used to generate the answers.

[1] https://zeitgaist.ai

LangChain is a great workaround for that. [1]

> how to work with a memory module that remembers things about specific entities. It extracts information on entities (using LLMs) and builds up its knowledge about that entity over time (also using LLMs).

[1] https://python.langchain.com/en/latest/modules/memory/types/...

Can and will you really read all the sources that you find with Google? What about topics people are talking about on all the different social media platforms? Will you really read all the comments?

I think these tools will help us break out of local bubbles. I'm currently working on a Zeitgeist [1] that tries to gather the consensus on social media and on the web on general.

[1] https://foretale.io/zeitgeist

This is why I prefer the term "weak AI". Weak AI is specifically trained to solve tasks whereas strong AI can teach itself to solve new tasks.

Whether humans are able to create strong AI is a philosophical question: While some argue that's not possible (can we be Gods?), others argue that this is the next logical evolutionary step.

Let's see if we can at least mimic strong AI when we let LLMs connect to external systems (internet, money, more energy, etc) and specifically allow themselves to fine-tune or train new NNs in general.

Time will tell.

That's why more research should be poured into homomorphic encryption where you could send encrypted data to the API, OpenAI would then run computation on the encrypted data and we would only decrypt on the output locally.

I would never send unencrypted PII to such an API, regardless of their privacy policy.

Wow, a context of 32K tokens. I'm excited to see what new capabilities that will have! Up until now and depending on the task by hand, I usually broke a larger context down into several contexts. For example to summarize multiple websites and/or long social media posts, on a recent task [1] I fell back to making several requests each with its own (isolated) context and then merging these summarized contexts into a new context. That worked remarkably well, though.

[1] https://foretale.io/zeitgeist

I use this method to answer questions about historic and real-time social media comments.

https://foretale.io/toolbox/Social_Media_QA