Comment by voytec | Hacker News Reader

Comment by voytec | Hacker News Reader

Parent Full thread

Those are my exact feelings since testing Kagi. I did not found it appealing as a plain search engine, nor private.

But they have recently shown a tech preview of an "Universal Summarizer"[1] which I have found far superior to ChatGPT (cGPT is worse than some freely available pre-trained models in this regard).

I have asked Kagi and cGPT to summarize my own post[2] which Kagi did rather well (I don't have a screenshot), while cGPT spilled utter nonsense[3], mentioning hardware and software that isn't mentioned in the post and which - in some cases - I have never used.

While Kagi is unappealing to me at the moment, I will happily try it out the future again.

tl;dr: I wrote about using: FreeBSD, Linux, Fail2ban, blacklistd, Terraform, Ansible, Postfix, Dovecot, Kubernetes, Jenkins, ArgoCD, Git and nginx.

ChatGPT stated that I wrote about MacBook Pro, iPhone, AirPods, Kindle, Chrome, iTerm2, VSCode, Vimium, Adblock Plus, Dropbox, 1Password, Trello and Slack.

[1] https://labs.kagi.com/ai/sum

[2] https://wojteksychut.com/posts/work-tech-i-use-privately/

[3] https://wojteksychut.com/pic/chatgpt_post_summarization.png

> I have asked Kagi and cGPT to summarize my own post which Kagi did rather well (I don't have a screenshot), while cGPT spilled utter nonsense

> https://wojteksychut.com/pic/chatgpt_post_summarization.png

ChatGPT currently has no capability to access external sources, and its training data cuts off at the end of 2021, so there's no way for it to have ever succeeded at this task. It was essentially asked to summarize an article it has never seen, so it hallucinated the entire summary based off of the words in the URL as that is the only information the model was provided. For a more equivalent comparison you'd need to copy the text of the article into the chat input.

Of course that doesn't change the fact that it responded to your query with totally fabricated nonsense, which is a horrible failure mode that points to why it can be so hard to trust LLMs' responses, but I do believe the disclaimer popup that appears when you launch ChatGPT notes this particular limitation.

It spelled my last name correctly in 4th paragraph (although could have got it from domain name). This domain was registered in 2023 so it could not have data from 2021; but maybe has been fed from Bing later?

It currently states: "As an AI language model, I don't have direct access to real-world events that have occurred after my knowledge cutoff date of September 2021. However, I can access information and data from websites created after September 2021, as long as they are publicly available on the internet."

> it responded to your query with totally fabricated nonsense, which is a horrible failure mode

Agree. I would be fine with getting "I cannot summarize this post as I don't crawl external sources" but not this.

Reply on news.ycombinator.com