chenglong-hn

for agent to generate, but also easy for human to edit (especially with UI) :)

Some composite charts are quite annoying to be generated well (like bullet, waterfall etc), their Vega-Lite equivalent can be quite long if just starting from scratch.

The intention here is that Flint is a simpler abstraction to get basic setups right and any followup edits can be done on top of the first compiled outputs (thus not limiting expressiveness). It also makes it easier for user to manipulate (like swapping axes, click to change something, which can be very hard if LLM generates a complex chart spec upfront).

But for many basic stuff your intuition is completely right.

we are considering also reliability, interactivity besides expressiveness. Simpler spec with good expressiveness comes handy when you want the agent to be reliably for non-expert users and with small models.

right, in fact many small models still struggle with following json, some new forms are also needed

For AI agents and Humans :)

That's awesome!

I find that besides training better models, designing new language for agents is also a super viable paths to improve their performance!

Vega was a high-level language in the past for human, but now they can be a bit too low-level for AI agents! AI agents have to write a lot of low-level params just to make charts looking good, and the result is that programs are hard to write reliably for AI agents.

Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good.

So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past.

This is fun! We started thinking it would just be an engineering task in the beginning, but doing a solid intermediate language turned out to be a research project (the paper will be out soon).

Also, I find NTChart very fun, maybe we should add NT chart to the list of compilation backend for Flint so it works in the library. Putting a reminder here: https://github.com/microsoft/flint-chart/issues/45

In fact, Json as a common language for human in visualization has been around for a while! The benefit of declarative grammar is that users can effective manipulate specs through UI (drag and drop, clicks).

Btw, Flint is intentionally designed to allow agent skip low-level params like scale, axe, zero, step size etc (which are extremely crucial for "GOOD-looking") and they are dynamically optimized by the compiler. So AI agents can have a easier time.

It's more like a simple high-level spec to make it easier. The idea is that you don't have to fill position / axes details just to make the chart work. The compiler has a bit of magic of using semantic types to optimize what parameters will be set in ECharts.

In some composite chart examples, the good-looking echart spec is like 5x longer than the simple Flint one!

Project page: https://microsoft.github.io/flint-chart/

MCP setup: https://microsoft.github.io/flint-chart/#/mcp

Zenquery is super cool! Data Formulator is mostly designed for data visualization and not as flexible for general QA -- we might be able to find some collaboration, Data Formulator is open source: https://github.com/microsoft/data-formulator

It's here! https://github.com/microsoft/data-formulator/tree/main/py-sr...

When install Data Formulator locally, it's possible to connect DF to databases with connection parameters in UI. To add more data loaders, there is a common template.

This is incredibly cool! A lot of times the user query can be ambiguous enough to make it consistent across runs. The semantic layer is essential to reduce ambiguity, either built by AI or engineers.

Feel free to submit requests in github for any customization needs!

That's something we are building! We hope to enhance the report generation as a dashboard builder. Instead of automatically compose an article out of the exploration, we could add more instructions and UI to allow user to specify how different components (vis, data, questions) should be put together to "codify" into a live document to share.

That's a good suggestion :)

I'm tempted to add a button to support connecting data from/to ODBC/JDBC but at same time worried about security stuff...

maybe there could be some intermediate layer to support this :)

yes! chat is still a necessary component, since it's sometimes the only way for us to communicate unstructured information to the system.

you find it! you can't imagine how often I'm annoyed when see April being ranked before March due to alphabetic order...

Thus the sorting agent, and now running by default in the background!

That's a cool example! You are right, GPT-4o is much more powerful than we allow it to perform in Data Formulator, and our current design is to restrict it to a point that the model behavior is more or less reliable.

While we design it targeting more end-user analysts scenarios (thus much simpler UI and function support), we see the big value of "freeing" GPT-4o for advanced users who would like to have AI do complex stuff. I guess a starting point could be having an "interactive terminal" where AI and the user can directly communicate about these out of the box concepts, even having the user instruct AI to dynamically generate new UI to adapt to their workflow.

this is a big challenge! the more complicated the code generated by AI, the more likely is going wrong and harder to verify. (more magic == more risk)

I'm curious how to really restrict what AI is generating each step so it is simple enough to verify and edit, yet not making it seems too verbose and slow

1. when started out, we have Tableau/PowerBI users as the main audience in mind, hoping to grant them the power of data analysts who program in python to create charts requiring data transformation. but as we are building the tool, data formulator more or less are most powerful for people work with python data as you mentioned, since they can more easily issue instructions, verify results and followup.

from our user study, it seems like experience with data analysis (i.e., know how to explore is the most important skill) either they know programming or not, software engineers without data analysis experience sometimes struggle using it.

2. check out how it's build in our paper (https://arxiv.org/abs/2408.16119)! but generally data types and sample values

3. hallucinations is unfortunately a big challenge! our main strategy is to (1) restrict the code space that AI can transform data so it's less likely to go wild, and (2) provide much information to the user as possible (code + data output + chart) to let them discover and (3) let them easily refine using data threads

4. yes! we have some plans in the bag and about to figuring it out!

this is really really cool! directly working with table is sometimes the only way to clean the data as well :)

I wish multiple ways of interacting with data can co-exist seamlessly in some sort of future tool (without overwhelming users (?)) :)

this is constant challenge! code is the ultimate verification tool, but not everyone gets it.

sometimes reading charts help, sometimes looking at data helps, other times only code can serve the verification purpose...

I'm not very confident either -- some of users in our user study also reports the same. :)

We are recently thinking about how to have AI provide more information in ways we can interpret. For example, might adding some feature to hint if AI's computation use a specific formula that users didn't mention. I feel working with AI is like the first week working with a someone we never met before.

potentially! I myself as an experienced programmer finds directly chatting with GPT-4o much easier, since I have full control. But Data Formulator only transforms data in the back and GPT-4o don't do anything with charts (sometimes we want it!)

We are wondering how to really achieve flexible chat experience with UI at the same time, right now it's quite a research challenge!

Wow, that's pretty cool! I think there are potential -- current LLMs are not that good on VegaLite when I ask it to edit the script :)