Show HN: Interactive Graph by LLM (GPT-4o) | Hacker News Reader

Show HN: Interactive Graph by LLM (GPT-4o)

columns.ai

45 pointscaoxhua50 comments

From a previous submission of this domain (May last year): https://news.ycombinator.com/item?id=35798482

> After trying some queries, I found ChatGPT is good at finding public data, sometimes it may not be accurate, but overall, pretty good.

GPT-4o is no more reliable for this than GPT-4 or GPT-3.5 before it.

One of the many unintuitive things about LLMs is that they are terrible tools for looking up specific facts about the world - which is weird, because that's traditionally one of the things computers have been best at.

If you want to build a working version of a system like this the trick would be to provide it access to tools for looking up data in citable, reliable data sources - things like the CIA World Factbook.

Then display the charts along with a link to that underlying reliable data.

GPT-4o has been the most aggressive one-shot hallucinator of the GPT-4 class models for me.

4-turbo should have a knowledge up to December 2023. But it doesn't know anything about the cli tools you created before then (except shot-scraper).

Compare 4-turbo to 4o

llm -m 4t "Tell me about these cli tools and link to them: - llm: - ttok - symbex - shot-scraper " Among the tools you've listed, "shot-scraper" is the only one that I could provide detailed information about based on widespread use and documentation available as of the last update. As for "llm," "ttok," and "symbex," there isn't readily available, widely recognized information on tools precisely matching those names, so it's possible they might be lesser-known tools, newly released, or possibly regional or specialized tools not widely documented in widely-used platforms. (Truncated)

llm -m 4t "Tell me about these cli tools and link to them: - llm: - ttok - symbex - shot-scraper " As of my last knowledge update in September 2023, here is the information on the CLI tools you are asking about:

1. *llm*: - I couldn't find a specific command line tool named "llm" that is widely recognized or used. It could be a lesser-known tool, a new tool, or potentially a typographical error. Please verify the tool's name and provide additional details if possible.

2. *ttok*: - As with "llm," there isn't any widely known CLI tool named "ttok." (Truncated)

llm -m 4o "Tell me about these cli tools and link to them: - llm: - ttok - symbex - shot-scraper " Sure! Here are brief descriptions of the CLI tools you mentioned along with links to their official repositories or webpages:

### 1. llm `llm` is a command-line interface (CLI) tool to interact with large language models like OpenAI's GPT-3. It allows you to query these models directly from the terminal, making it easy to integrate AI capabilities into your workflow or automate tasks.

- *Link*: [llm GitHub Repository](https://github.com/simonw/llm)

### 2. ttok `ttok` is a CLI tool for working with tokenizers. It's a tool under development aimed at helping users understand how tokenization processes text, which is essential for preparing data for machine learning models, especially in NLP applications.

- *Link*: [ttok GitHub Repository](https://github.com/simonw/ttok)

### 3. symbex `Symbex` is a CLI tool for performing symbolic execution on Python bytecode. It can analyze Python code to find potential bugs, vulnerabilities, or logical errors by exploring all possible execution paths.

- *Link*: [symbex GitHub Repository](https://github.com/simonw/symbex)

(Truncated)

llm -m 4o "Tell me about these cli tools and link to them: - llm: - ttok - symbex - shot-scraper " Sure, here is a brief overview of each CLI tool along with links to their respective repositories or documentation:

1. *llm (Language Model CLI)* - *Description*: The `llm` command-line tool leverages large language models, such as OpenAI's GPT-3, to make it easier to incorporate AI functionalities into your command-line tasks. You can use it to generate text, answer questions, and assist with coding or other language-based tasks directly from your terminal.

    - **Link**: [llm on GitHub](https://github.com/simonw/llm)

2. *ttok* - *Description*: `ttok` is a simple command-line utility that provides tokenization for text, particularly useful for breaking down input for language models. It helps to preprocess text, count tokens, or manage tokens more efficiently from the command line.

    - **Link**: [ttok on GitHub](https://github.com/simonw/ttok)

3. *symbex (Symbolic Execution)* - *Description*: `symbex` is a CLI tool designed for searching codebases. It employs symbolic execution to analyze code paths and find functions, variables, or other code elements. This can be particularly useful for developers looking for higher-precision searches in large codebases.

    - **Link**: [symbex on GitHub](https://github.com/paulgb/symbex)

(Truncated)

Clearly, openai did a lot more web scraping for this model.

Cute, but can I see sources on the data? I tried some query on some belgian statistics and it gave me just... very weird numbers that i do not trust, but zero way to check them.

This is pretty cool, it's snappy, it seems to work, but there's no way I trust it without a way to check the LLM's work.

Yeah I just tried it with numbers I knew a bit and it seems totally made up. The generated chart showed a linear downwards trend while in reality there isn't one and the numbers seem way off.

This is a great point. It's easy to spin up an LLM data analysis demo, but it's really hard to trust the outputs.

I think we've (https://www.definite.app/) done a decent job here. Once the LLM generates results, you can inspect the query all the way back to the source (SQL).

https://www.youtube.com/watch?v=Nu2Tal3hm7s

Unless it's very "public common", otherwise, the numbers can be used for trustworthy work. In general, I think this is just for "fun play" most of the time.

When prompted for

'daily unique visitors to openai.com by month since 2022 to 2024'

It gave a graph with 1) time axis decreasing left to right, 2) visitor numbers which can't be real (near prefect linear trend) 3) points in the future, going out to Dec 31, 2024.

https://columns.ai/chatgpt/Fq3fID0AVvWWIM

Inspired by that I tried to ask it for “Price of bitcoin from year 1900 to year 2300”

I had to press the “refine” button several times which turned the query into “Bitcoin price 1900-2300”

The result was a bar chart with five data points.

All of the five data points were labeled “2K”, and had differing heights.

The dates along the X axis were in a short span of time – just a few days – and were seemingly randomly ordered. Or like, sorted with even dates in descending order followed by odd dates in ascending order. Very.. peculiar.

2023-10-04

2023-10-02

2023-10-01

2023-10-03

2023-10-05

Also the link copy button is not working for me in Safari on iOS, and the url doesn’t change from the op link either, so I can’t share a link to the graph.

this is what I saw, https://columns.ai/chatgpt/jMTu9QXU9nbeCs "no fact check", not sure if LLM would give source cite while giving those numbers.

Hmmm this doesn't look right? : https://columns.ai/chatgpt/AXoHKTTYYkeT3x

I was surprised, but it does look right https://en.wikipedia.org/wiki/Demographics_of_Ukraine

To be fair, it does not look right.

The graph in that page is completely different (e.g. the sharp drop in 2015). This seems to be just linearly interpolating between start and end populations.

If I click the `Y-axis`, then set the low-range to 0, then it looks better https://columns.ai/chatgpt/WoGchDeDpKNKWl

Looks about right https://columns.ai/chatgpt/Gajr7yJ9O5x57o

Would you mind explaining a little more on what is happening in the background?

- What is your core technical value add?

- Do you have the data sets in your own database and you are using OpenAI to query them?

- Looks like you have your own home built database?

- Are you using LLM Agents?

- I saw that you have Airtable integrations, are you able to do the same for any datasource including Airtable?

So it's a pretty simple wrapper of LLM model in use (currently gpt-4o), it does not add much technical stuff in it.

It does not use database for any "random search", but yes, columns.ai is a data analytics tool that allows you to connect supported live data sources like Google Spreadsheet, Airtable, Notion Database to create visual stories.

The analytics engine is home built (https://github.com/varchar-io/nebula) but it is not a database. And I don't use LLM agents, just build logic how to purify data returned by LLM, and fit them into an optimized visualization.

Hope I answered your question!

Nice one, though I can't scroll the list horizontally thus can't read columns beyond screen bounds (iOS 17 Safari, tried "best ingredients that go with bacon" query and the result was a table view)

Yeah, not mobile friendly UI. because the visualizer really needs some space to render, so overall I didn't pursue to optimize it for mobile.

Anything is clickable and styleable on the canvas though.

I can't believe there are no sources for the data. A few charts I tried seem off, but even in the cases where it looks like it might be right, trusting it without a source is a big no.

Trusting is a problem, to be honest, mostly for fun. If you want to trustable source, feel free to copy text paragraph and let it parse for you, there is another mode for that.

How is this in any way different than an RNG? The numbers are completely made up, does slapping an AI label on a RNG somehow make this unique or interesting?

It's like russian roulette but the bullet is a RNG. The number of bullets is also a RNG. The gun is a RNG too.

hahaha~

I don't why it's flagged, it's just a free service for people to play with.

GDP in Germany for the last 10 years

4k each year but the bar chart goes up

Sounds about right

I always click Y-axis to modify the axis range, usually modify low-value to 0 and it looks better - https://columns.ai/chatgpt/uUN2oYp7Yusstk

I mean kudos, the UX here is so insanely good it feels like magic, but I don’t trust magic. Where’s the data coming from?

Never ask a woman her age, a man his salary, or an LLM it’s data sources.

hahaha~

I get that free will means you can do evil. But, given the choice, why would you not only do it this baldly, but then go out of your way to show it off?

Where do the numbers come from?

It's numberwang

I think what the world really needed right now was an LLM making up fake statistics, presented in a convincing high quality way.

There is just no way this could ever be a problem. Surely everybody knows that if presented with some data they need to do a deep dive into the actual sources instead of blindly trusting a graph.

I seriously want to know what was going on in the mind of the person who made this.

I still have a faint hope that proliferation of AI generated content will make it all so absurd that people will turn back to traditional editorially reviewed content. But I know that's probably too naive - I recently had someone bring up "here's what Chatgpt told me" as an argument in a discussion, in manner like Wikipedia was quoted before

It also made sense that being able to quickly look up evidence online should have ended nonsense conspiracy theories… it did not quite work out like that.

There are three kinds of lies: lies, damn lies and statistics.

genius, haha~

This might be a fun project, but it's going to cause extreme misunderstanding for many people. People trust graphs, and users of this site are going to unwittingly spread falsehoods to for example r/dataisbeautiful or even less reliable social media pages.

I'd urge you to take this site down as it will be net negative to the world.

As an example that even simple queries result in false information, here is one repeated.

https://imgur.com/a/4pm8sFb

The correct data, collected from the central statistical bureau, is almost the opposite, with the winter as the most wind time (thankfully, as that's when Sweden needs the power) https://imgur.com/a/OX1608V

Wow, this is fantastic!! Thanks! Currently writing my PhD and this can be really useful

https://columns.ai/chatgpt/3Gk20NKW6D4vTt

Okay, but this one is really accurate: https://columns.ai/chatgpt/90ECu7cRj1pL82

This one is even better: https://columns.ai/chatgpt/3Gk20NKW6D4vTt

Careful, their IQ is slipping https://columns.ai/chatgpt/0qvrhRcljw6Xwh

The political bias in the results is astonishing and shows why these models should not be used for educational purposes. Just ask it a "contentious" question and notice it give biased and sometimes nonsensical results - 'crime rate by political orientation' shows three 'republican' states coming out on top with three 'democratic' states filling out the bottom. When necessary it just seems to make up data to get the desired results, e.g. 'murder rate by political orientation' talks about 'Country A´ and 'Country B' with, of course, 'right' being far more murderous than 'left'. It claims that 'democratic New York' has the lowest crime rate. 'IQ by political party' is another interesting example. Compare this to 'trust in LLM output by political party' and maybe those 'dumb republicans´ (who do not trust these models, together with the supposedly super-smart independents by the way) suddenly seem to be a lot smarter than all those 'bright "democrats"'.

This site is a harbinger of the agitprop factories which are about to flood the 'net due to the general availability of LLMs.

To the downvoters: instead of just pressing that downvote button on things which you do not agree with it would be beneficial for the discourse if you laid out your disagreement.

https://columns.ai/chatgpt/90ECu7cRj1pL82

Reply on news.ycombinator.com