Improvements to data analysis in ChatGPT
openai.com
openai.com
The demo they show has an excel sheet source being transformed into another sheet by grouping by a specific column. I don't like how it does a transformation step, sends a new transformed file and then switches the display to this new file. I worry that this loses the history of what exactly has happened to the file, so you're letting a black box run an unknown transformation and trusting the result to be correct.
A better implementation would require a custom UI for this, similar to existing data wrangling tools where every action is logged so that it's clear what has happened to the file and the steps can be checked transparently and rolled back if needed. As it exists now I would find it hard to trust it
You'd want deterministic steps, like it would need to have a limited list of functions to choose from that do specific things eg "group_by()" "filter()" etc so that the code it runs is always the same for it.
In fact, this is probably more accurate than custom UIs that log actions you take, since to build that UI, engineers need to make sure each action is logged (and it's easy to forget to log one!).
Here's an example where I recently used aider and GPT-4o to plot a graph.
https://aider.chat/2024/05/13/models-over-time.html
The graph itself is kind of interesting. It shows how LLM code editing skill has been changing over time as new models have been released by OpenAI, Anthropic and others.
One of the nice things about human programmers is that you can derive intent and create responsibility. Sometimes we have to encode that in a commit message or ticket, but it's there. When you find out that subtly the program was piping all data into devnull without realizing it you at least have a human to figure out how you got here. Another example of this is the xz debacle.
What are you supposed to do when that's a thousand layers deep? What about when your next generation of programmer's ability to do things is exceptionally stunted by this effort?
The obvious answers: record the human's intent in the form of their prompt, and record the LLM's raw output (if you use a conversational LLM out of the box, it almost always includes this even if you explicitly prompt it not to, lol). Of course, depending on your UX this may or may not work. For autocomplete there is no obvious user intent.
There are additional approaches which I'm exploring that require more intentional engineering, but essentially involve forcing "structure" so that more intent gets explicitly specified.
Think about this as well: You're creating processes that have no reasonability. I am 100% responsible for all code I write, even if I wrote it wrong, but if the code your tool generates is wrong it's not my fault, I didn't write it. Multiply this by the hundreds of thousands of times this will happen in a given year and by each employee.
Frankly, you should re-evaluate if you even want your product in the world. What kind of future hellscape are you enabling?
I think we're closer to what you're describing. All transformations in Definite[0] are done with SQL which you can easily inspect. The transformations are stored in your data model so everyone else can reuse them.
The setup we normally see at a company is one (or more) data people control the model which can be used by the rest of the company thru our AI assistant. Quick demo here: https://youtu.be/p6BoqX0cYnU?si=hg2V_KM8ScUzwrDx&t=147
> "...analyzing customer data, which has become too large and complex for Excel. It helps me sift through massive datasets..."
The emphasis here is on the size of the data, but I think that's a red herring. Excel can certainly handle way more data than you're going to want to repeatedly pipe through an LLM. I'm betting it's the diversity of---or even lack of---schemas. Big piles of tables and unstructured data you'd normally have to normalize and join.
I spent years doing data analysis slowly with Python (pandas), and it took me a long time to accept that getting better at Excel was just more efficient. Hard to accept that code isn't always a task's best interface. Automating some Excel work sounds even better.
So this will handle millions of rows of data quite happily. I've used it extensively with SQLite in the past - you can upload SQLite databases to it and have it run SQL against them. You can also ask it to save data to SQLite for you to download.
Maybe an analogy is doing image processing. One could write filters and other manipulations in code, or use something like Photoshop. It turns out Photoshop is a great interface for many of those tasks. I think the same thing is true for more data analysis than I'd realized because of my poor Excel skills.
Still want to improve my spreadsheet abilities though. There's a fun video called You Suck at Excel by Joel Spolsky I'm halfway through [1]. Would love to know of more like this.
OpenAI has some network effects baked into its product suite, but it’s not even in the same league as Microsoft’s bundling strategy (setting aside Google for a moment). Microsoft’s history is littered with competitors who had a better product, a faster adopted product, who were snuffed because of Microsoft’s superior distribution capabilities.
I’m on the train at the moment, but the big exception is the mobile market, and in hindsight that makes perfect sense - most of the phone market consumer grade, where bundling productivity software isn’t much of a value proposition.
It’s far more likely that OpenAI remain where they are on the value chain, because it’s easier to capture and integrate AI startups into their products. If they were to compete on productivity software, they need to differentiate with entirely new modalities of AI-informed user interfaces, or they will be yet another slightly better program that got blown out of the water by MSFT enterprise distribution
edit: to be clear my point is, why would you become a bit player in an established market when you can dominate a new market you created yourself?
Consumer-focused wrapper startups like jenni.ai, research helpers, or math tutors I doubt they'll focus on, since most of the revenue is in enterprise.
https://the-decoder.com/sam-altman-explains-why-openai-might...
> OpenAI CEO Sam Altman has a clear message for startups developing products based on OpenAI's GPTs: They should assume that the models will improve drastically with each new release, rather than relying on the current state of the technology.
> Altman uses GPT-4 as an example: Any company that builds something based solely on GPT-4 is likely to be surpassed by GPT-5 if it is as big a leap over GPT-4 as GPT-4 was over GPT-3. Those companies will be "steamrolled" by OpenAI, he says. "Not because we don't like you, but because we have a mission."
what a phrase, will be borrowing it
if it is, maybe time to reconsider.
https://techcrunch.com/2023/07/17/microsoft-lost-keys-govern...
And the trend has been towards standardising on Kubernetes to allow for the hybrid model to work.
Going on a tangent, the other day a friend of mine was explaining how he uses Google Drive to copy files from an iPad to a Windows laptop. He is trying hard to make sure three letter agencies have an extra copy of his data, just in case.
https://learn.microsoft.com/en-us/azure/confidential-computi...
https://cloud.google.com/confidential-computing/confidential...
Hypervisor cannot read its guests data.
Is there some kind of "backdoor" to enable the cloud provider to grant law enforcement/intelligence/etc access? Or not?
From what I understand, theoretically with something like AMD SEV it could be configured so no such "backdoor" exists. However, is it actually set up that way in practice?
The rest is vaporware, basically. Obviously it will get a lot better, but this is refreshing.
Hopefully Microsoft has something better to show at Build that isn’t a bunch of half baked tools that will… maybe be better in two years.
As far as I know, this is the first integration of other cloud products in ChatGPT that's done as a 1st party integration (vs. "GPTs", which are all 3rd party)?
Completely agree, chat is not going take over every UI that's been developed over the past decades.
One example: are you going to build a "Company Dashboard" in ChatGPT?
ChatGPT is good for "last mile" adhoc analysis (e.g. you already have all your data in a single file and you want to ask a few questions). But you're not going to build dashboards and standard reports in it just like you wouldn't run your CRM in it.
I'm not even a fan of them, and I dislike the hypocrisy behind the name; and still, ChatGPT has replaced Google for 90% of my queries. It's starting to feel like a burden to have to open Google when ChatGPT doesn't give me what I want for some reason. As soon as they're able to "ground" results, 99% of my queries will go there.
Anyone that doesn't understand this is just coping hard with this new reality.
And yet they don't!
Talk is cheap.
Anthropic have just hired a Chief Product Officer (Mike Krieger - co-founder of Instagram), so you may see more layered on top of the core AI if that is what you are referring to, although they seem to have a different focus (more corporate) to OpenAI, so don't hold your breath for an Anthropic AI girlfriend.
OpenAI seem to be trying a shotgun strategy of productizing GPT any way they can, but I'm not sure that's the best long-term strategy. Maybe better to build an ecosystem - build the OS/APIs, and let 3rd party's build the applications on top of that.
And Google showcased the project astra thing, which seems like the equivalent.
Remember Duplex [1]? Which then turned out to be fake. The more recent fake Gemini demo? What an embarrassment.
LLMs also aren't correct 100% of the time, far from it.
I love open source but their very nature suggests there is no one open source model that will have the critical mass to just be a straight in replacement for ChatGPT and its UI.
I use Ollama and choose models and they are close to GPT-4 but OpenAI is speeding up into making a useful chat product. All the rest are foundational model demos.
Big pivot.
Less budget, less power.
If you're looking for a version of ChatGPT Data Analysis, but want to offer it to your customers, take a look at us! https://www.lightski.com/ (disclaimer: I'm the co-founder.) We're building this feature of ChatGPT, but in a way that you can embed inside your app, so that your end-users can get an AI Data Scientist without needing to export data, then import it into ChatGPT or Julius.
Disclaimer: I work at Akkio, but think we have a really nice, value-add product and an excellent team behind it :)
Upload or import a CSV, Excel, Google sheet, etc. and also just launched Postgres and Snowflake connectors today!
ChatGPT is good for "last mile" adhoc analysis (e.g. you already have all your data in a single file and you want to ask a few questions).
If you're looking for data infrastructure with an AI assistant, try https://www.definite.app/. We give you a full data stack in one app:
1. Built-in data warehouse - We spin up a duckdb database for you
2. 500+ connectors (e.g. Postgres, Stripe, HubSpot, Zendesk, etc.) - You don't need to buy a separate ETL, it's also built-in
3. Semantic layer - Define dimensions, measures, and joins using SQL in one place. We have pre-built models for all the sources we support (e.g. the Stripe model already has measures for MRR, churn, etc.)
4. Simple BI / Dashboards - Build a table with the data you want and generate visuals off that table. Works like a pivot table, if you can use a spreadsheet, you can use Definite.
5. AI assistant to help you thru all of this and analyze your data
I'm mike@definite.app if anyone wants help getting set up.
Is anyone working on this?
That real world example also doesn't seem representative of the bulk of real world use of spreadsheets. Even in the announcement they're focusing on simple pivot tables and creating presentations. I don't see any reason why this wouldn't be able to handle asking questions on a data dump from quickbooks or the like. I get the feeling that you might be operating in a very unusual context, like in tech or finance, where you might even have professional data analysts who are hired to work with data. That's probably the biggest market revenue wise for this stuff but in terms of the number of users, spreadsheets are used EVERYWHERE
Spreadsheets ARE everywhere, but they're even harder to connect back to the source of the data because their proximal source is some e-mail message. There just isn't enough information in the sheet to tell the model what the data actually means.