chriddyp

It's a valid concern. We're not releasing the dataset, the answers, or the full set of questions to help prevent this. At the same time, I like to share where it gets things wrong in a bit more detail, which involves sharing a bit of the exam. I expect that we'll create a new benchmark with a new dataset in 6 months.

Ran our DataAnalyticsBench benchmark on it: https://plotly.com/blog/claude-haiku-5-5-plotly-data-analyti...

9x cheaper than Haiku 4.5 and 2 letter grades better. It's also now the fastest model (using the default speeds, not trying any of the other models "Fast" mode) to complete the exam.

Similar ballpark to Luna in price, cost, and accuracy. These are very cheap models: $0.38 to answer 40 in-depth data analytics questions (compared to $15 for Opus 5.5 or $20 for Astra).

Overall very good at data analysis - handling all of the straightforward data analytics questions correctly. It fell short answering some of the questions that required some deeper statistical analysis like looking into other variables. In other words, it's not as persistent as other models in its analysis, which I think we'd expect from how they're positioning the model.

Compared to OpenAI: GPT-6 Luna did a bit better and was about 30% the cost of Haiku 5.5. GPT-6.1 Sol got all answers correct, but was 10x more expensive.

That’s correct. About the same in price, but Qwen does get more answers correct in this particular benchmark.

Just ran this through our data analytics benchmark (I work at Plotly).

It's 10x cheaper than Mistral Medium 3.5 from April and goes from 58% to 74% correct. Definitely a generational shift.

It's not on the Pareto curve yet, but it's good enough for data analytics, and at this rate I suspect it'll be excellent in another few months.

Full write up: https://plotly.com/blog/mistral-large-4-plotly-data-analytic...

This is a really cool idea. So much context is lost within agent conversations these days, and it requires far too much discipline to have the conversation between you and your agent in a shared place like GitHub (plus, thee conversation isn't always just a comment and response - it's... an actual conversation).

I'm very glad to see folks innovating in this space.

congrats! the paper that is published alongside this (Cordis) is super interesting. has anyone on the team given a talk or published a talk about this? would love to hear the authors break this down

While this is true, there is also a layer in the harness between the output of _any_ tool output (eg stdout or hand-rolled tools) and the LLM. A tool could read the file but then the agentic harness could redact the output before returning it back to the llm if any of the contents matched the file contents. We do something similar in Plotly Studio where we check the entropy of strings in the user input and flag & redact any high entropy strings to the user as “potential credentials” thay the user might have inadvertently copied and pasted into the prompt before sending to the llm.

There are ways around this - the llm can always be clever by invoking tools to read the file contents in a different way than the direct file contents - but this is all to say that the agentic harness layer _does_ allow for deterministic logic in between tool output and the LLM requests.

This is really cool. And timely! Check out the recent paper by Google et al re "Societies of Thought": https://arxiv.org/html/2601.10825v1. It goes into how different conversational behaviors (raising questions or just say "but wait..."), perspective shifts, conflict of perspectives, tension, tension release (jokes!), asking for opinions) and different personalities (planner, expert, verifier, pragmatist) is both a sign of and can result in much higher performance reasoning.

So I'd be curious to see if encouraging certain conversational behaviors might actually improve the reasoning and maybe even drive towards consensus.

The issue goes beyond access and into whether or not the data is "trusted" as the malicious prompts are embedded within the data. And for many situations its hard to completely trust or verify the input data. Think [Little Bobby Tables](https://xkcd.com/327/)

Curious to see Apple using real black as background on their home page but rgb(29, 29, 31) for their black text.

FYI re Dash & HTML - If you aren't using your own stylesheets with HTML, then I'd recommend dash_bootstrap_components.

Here's an example that uses almost entirely higher level components: https://dash-bootstrap-components.opensource.faculty.ai/exam...

We've also been working on `dash.templates`, which provide opinionated, prebuilt UIs - no layout code required: https://community.plotly.com/t/introducing-dash-labs-dash-2-...

Plotly's 3D viz is built with WebGL & SVG, using libraries like regl & stack.gl. SVG is used for axes & text and WebGL for the high performance rendering of points and surfaces. Surfaces, lines, points, and subplots are all supported. See https://plotly.com/python/#3d-charts.

For more complex 3D objects, Dash users can use dash-vtk. This includes things like point clouds, CFD simulations, 3D mesh, or 3D images.

Hello HN! Nice to see this up here. Chris here, cofounder of Plotly.

Falcon is open source and works without an internet connection or a Plotly Chart Studio account. Falcon wires together our graphing library plotly.js (https://github.com/plotly/plotly.js/), the plotly.js chart editor (https://github.com/plotly/react-chart-editor), Electron, and some open source NPM packages for connecting to databases.

Just FYI - As a company (Plotly), we're spending most of our development effort these days on Dash Open Source (https://github.com/plotly/dash) and Dash Enterprise (https://plotly.com/dash). Truth be told, we found that most companies we worked with preferred to own the analytical backend. We also heard many stories of organizations running into roadblocks with off-the-shelf SQL or BI tools (Falcon included!). Our approach with Dash is to provide the visualization and application primitives so that you could build your own tailor-made dashboards, analytical apps, or yes, even SQL editors.

If you want to read more about where we're at, here's an essay we wrote last week on Dash: https://medium.com/plotly/dash-is-react-for-python-r-and-jul...