Automating my job with GPT-3
blog.seekwell.io
blog.seekwell.io
It's deep learning based with a lot of augmentation. Going from the OP's article to actually being able to run queries on any schema is quite a bit more work. I'd love to see GPT3 handle arbitrary schemas.
p/s: the startup failed. deep research based startups need a lot of funds.
Translating between natural language and SQL is a reasonable idea. I was thinking about this as well, but I didn’t try anything as I don’t have an ML background. I spent some time looking at the SQL side of the problem, and it seemed quite a rabbit hole.
If you do manage to get it working up to a point where it’s usable by the average person, you can take it one step further: auto generate simple apps or websites in a no-code product.
This might bring some hate from the dev community as we are automating ourselves out of a job, but it would be a pretty impressive product if it worked.
It's been 5 years and I've been thinking a lot on this. This is a product with no good market fit. If you break it down by "kind" of sales, your basic branches are B2B and B2C.
B2C is mostly out because the person on the omnibus have no general need for a programming language, SQL or not (plus, outside of emacs, nothing consumers use are inherently "programmable"). So this program simply becomes a utility program that you reach out to occasionally like `cut` or `sed`.
We mostly targeted businesses and other startups. We wanted to empower the common employee to be able to query data on their own. That itself came with a huge amount of challenge. Turns out most employers don't like the idea of any Tom Dick and Harry having query access to databases. So we branched out, tried to allow querying spreadsheets and presentations (yes, the advertising arm of one big media corporation stored their data in tables in a .pptx file on a sharepoint server). The integrations are finnicky and break often.
Maybe we're not smart enough to figure this out. But one day, one day I shall be back.
But in the meantime, the failure of the startup spawned the Gorgonia family of deep learning libraries (https://github.com/gorgonia/gorgonia). Check it out if you want.
It's not remotely easy a business though. Read the uphill battle just in the HN comment thread on the Looker acquisition by Google.
My thinking would be to explicitly *not* ship a supported "product" of any kind, but rather to provide a way to put the raw materials in their current state "out there" from a research perspective and provide users a way to exercise the system and reason about its limits, without letting everyone run off with it. (For example pretraining would happen on the server from uploaded datasets.)
In this scenario, the idea would be to let the market come to you, by just putting this absolutely everywhere in its current raw form and *letting the market materialize/emerge*. Rationale being, you want to know what the correct next step is from where you stand right now. If the program actually runs, for an interesting definition of "runs", ie it can actually handle real-world work in a novel way, well, I say that directly translates to it being worth trying because you might get novel answers/ideas for where to turn next. IOW, I'm theorizing that the quality of the answers you would get may correlate with the irreducible complexity of how well the program stands up in practice.
Reviewing uploaded data and executed commands may also prove insightful and inspire new ideas for real-world integrations. (Something something view-but-not-share TOS clause)
Depending on resource usage, charging for certain API actions, as a way to further ratelimit and not necessarily to make money, may be reasonable - for example doing extensive training (or lots of iterations) or performing tasks that take a long time to complete or scan a lot of input data. (And of course there would be research exceptions to this as well...)
Also, tables-in-pptx is now filed in my head a few rows down from "email of a TIFF of a scan of a photocopy of a printout of a fax of another email" :) - that's terrible, haha
There needs to be a secondary break through in ML for progress or you need magnitudes of more data then is available on earth to make this even viable.
I am definitely impressed by the fact that it could get this close without knowledge of the schema, and that you can provide additional context about the schema. Seems like there is a lot of potential for building a natural language query engine that is hooked up to a database. I suppose there is always a risk that a user could generate a dangerous query but that could be mitigated.
Not related to the article but what exactly is "open" about OpenAI?
Granted training your own copy of GPT-3 would be beyond most peoples means anyway (I think I read an estimate that it was a multi-million dollar effort to train a model that big).
I do think it's a bit dodgy to not change the name though when you change the core premise.
I think OpenAI just chose a bad name for this for-profit initiative — "GPT-3" — that makes it sound like they were pivoting their company in a new direction with a new generation of tech.
Really, GPT-3 should have been called something more like "GPT-2 Pro Plus Enterprise SaaS Edition." (Let's say "GPT-2++" for short.) Then it would have been clear that:
1. "GPT-2++" is not a generational leap over "GPT-2";
2. an actual "GPT-3" would come later, and that it would be a new generation of tech; and
3. there would be a commercial "GPT-3++" to go along with "GPT-3", just like "GPT-2++" goes along with "GPT-2".
(I can see why they called it GPT-3, though. Calling it "GPT-2++" probably wouldn't have made for very good news copy.)
Sure, both models are using the same structures (attention layers, mostly), so it's a quantitative change rather than a qualitative change. But there's still a hell of a big difference between the two.
There was a pre-trained GPT-2 model offered for download. The whole "interesting thing" they were publishing about, was that models trained under the GPT-2 ML architecture were uniquely-good at transfer learning, and so any pre-trained GPT-2 model of sufficient size, would be extremely useful as a "seed" for doing your own model training on top of.
They built one such model, but that model was not, itself, "GPT-2."
Keep in mind, the training data for that model is open; you can download it yourself and reproduce the offered base-model from it if you like. That's because GPT-2 (the architecture) was formal academic computer science: journal papers and all. The particular pre-trained model, and its input training data, were just published as experimental data.
It is under that lens, that I call GPT-3 "GPT-2++." It's a different model, but it's the same science. The model was never OpenAI's "product." The science itself was/is.
Certainly, the SaaS pre-trained model named "GPT-3" is qualitatively different than the downloadable pre-trained base-model people refer to as "GPT-2." But so are all the various trained models people have built by training GPT-2 the architecture with their own inputs. The whole class of things trained on that architecture are fundamentally all "GPT-2 models." And so "GPT-3" is just one such "GPT-2 model." Just a really big, surprisingly-useful one.
That sounds like it would have been a reasonable choice for naming their research, but isn't the abbreviation "GPT" short for "Generative Pre-trained Transformer"? Seems like they very specifically refer to the pre-trained model, which I would also take from the GPT-2 paper's abstract: "Our largest model, GPT-2, is a 1.5B parameter Transformer[...]" [1]
--- [1] https://cdn.openai.com/better-language-models/language_model...
Those terms are so disseminated that I wouldn't be surprised if GPT-2 could suggest them.
But, by marketing it as a new version of the tech, "GPT-3", OpenAI gave journalists something they could actually report on without feeling like they're just shoving a PR release down people's throats. "The new generation of the tech can do all these amazing things; it's a leap forward!" is news. Even though, in this case, it's only a "quantity has a quality all its own" kind of "generational leap."
OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity.
Certainly makes that statement seem less credible.
"Oh, we said we're open but this is too dangerous to publicly release; we'll be licensing it exclusively to approved customers instead."
I'm really glad that weapons aren't open source. Imagine every dictatorship would get their hands on weapons. Luckily, it's hidden behind a paywall. /s
- Hershey is a public company. Most certainly NOT owned by either a charity or a non-profit. The only way a non-profit comes into the picture is that a significant portion of their 'Class B' stock is owned by a trust which is dedicated to a non-profit (the Milton Hershey School). (https://www.thehersheycompany.com/content/dam/corporate-us/d... pp 36-37)
- Charity vs non-profit: https://www.irs.gov/charities-non-profits/charitable-organiz...
> To be tax-exempt under section 501(c)(3) of the Internal Revenue Code, an organization must be organized and operated exclusively for exempt purposes set forth in section 501(c)(3) [CHARITY], and none of its earnings may inure to any private shareholder or individual [NON-PROFIT].
You can be a charity (albeit not tax exempt) without being a non-profit, and moreover you can be a non-profit without being a charity. (See also https://www.irs.gov/charities-non-profits/other-nonprofits ; and keep in mind that still other types of non-profits are not tax-exempt at all!)
- Trust "owning" Hershey's: If you look at the document I cited, you'll note that the trust (which is still neither a charity nor a non-profit!) owns only 5.5% of Hershey's common stock.
Nothing. At this point it's simply openwashing.
Microsoft's checkbook?
OpenAI's GPT-3 API is expensive enough (especially with heavy prompt engineering) that the time saved may not outweigh the cost, particularly if the output is not 100% accurate.
Does this mean that the model is still being improved? Or just that your access to it will somehow become better? Either way, I'm curious what that entails.
> Or just that your access to it will somehow become better? Yes
Increasing the amount of training data we can send would improve our results and that's what OpenAI mentioned they're working on.
1) Have a question
2) Figure out the answer
3) Have the AI figure out the answer
4) If the AI figured out your answer, be impressed, otherwise try again.
Think of how frustrating it is to be unable to talk to a human at Facebook or Google because their AI closed your account without explanation.
Now imagine this is how everything works.
I consistently have terrible experiences with by human operators over the phone, e.g. phone company and similar (in my case italy, but I guess it's a general problem). They routinely cannot address my issues and just say they are sorry but they cannot do anything about it, or that this time it will work.
Human operators are a solution only if they are not themselves slaves to an internal rigid automated system
Whatever error a human can cause, a machine can do as much or more damage many orders of magnitude faster and larger and be difficult to correct.
I feel like I'm back in 95, when I had to beg faculty staff to get a copy of VB on some lab computer, only to be able to use it 1 hour a day. Restricting knowledge like this, in 2021, feels odd.
The same applies to quantum computers. Models like GPT-3 are way too big for a consumer machine to handle and require something like a DXG-station [0][1] with 4x 80 GiB A100 GPUs to run properly.
So even if the model were available for download, you wouldn't be able to even run it without hardware costing north of $125,000.
It's less about restricting knowledge and more about the insane amount of resources required. It's not as bad as getting access to FMRI or particle accelerators, but it's getting there ;)
[0] https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-mo...
[1] https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...
Input: how much revenue have we received from users with an email ending in 'seekwell.io' in the last 3 months?
The output conditions on: users.signup_dt>= now() - interval '3 months'
But my interpretation would be to condition on charges.charge_dt
It reminds me of the criticisms of almost fully autonomous vehicles where the driver pays less attention. Now I'm curious if the queries written from scratch would be less likely to have errors than queries that use GPT3 as a starting point.
1. There is unlimited flexibility in the prompt.
Seemingly irrelevant changes to the prompt can change whether you get out correct SQL or not. Sometimes you can just repeat things in the prompt and get different and better results. "Write correct SQL. Write correct SQL"
For any one input question you may be able to tweak the prompt to get the correct answer out. But you need to do this tweaking for each question (and know the correct answer you need). Tweaking one prompt may break all other input-output pairs.
2. Real questions involve multiple large schemas.
I deal with tables with thousands or tens of thousands of columns. There is no way you can get GPT-3 to deal with that scale with a simple input as shown here. And of course you want to join across many tables etc.
3. Syntax
Natural language is more robust than SQL, you can get close and get the point across. Most language models trained on general corpora are fundamentally not suited to the symbolic manipulation of languages like SQL.
This isn't to say that GPT-3 couldn't be part of a solution to this problem, but please restrain your exuberance, it's not going to solve this problem out of the box.
Very cool work, I continue to be blown away by what GPT-3 can achieve.
----
"Nice! can you add a drop-down for regional statistics when in country view?"
----
"Just one last thing. Can you make the logo bigger?"
These systems solve none of those underlying questions, and in fact are more likely to make it worse. Because if you won't even bother to write SQL or look at your database, you are even less likely to make sure it's high quality.
(There is maybe an exemption here for queries that you don't really care of you get the real answer. I don't know what those questions would be, but that would be a good usecase.)
If I'm a non-technical user and I ask a plain-language question and the generated SQL is incorrect, it's likely going to give the wrong answer -- but unless it's terribly wrong ("Syntax error", truly implausible values) the user may not know that it's wrong.
So I see this as more of a tool to speed up development than a tool that can power end users' plain-language queries. But who knows? Maybe GPT-4 will clear that hurdle.
We've been ramping up our invites from the waitlist — our Slack community has over 18,000 members — but we still are only a small fraction of way through. We've been really overwhelmed with the demand and have been scaling our team and processes to be able to meet it.
We can also often accelerate invites for people who do have a specific application they'd like to build. Please feel free to email me (gdb@openai.com) and I may be able to help. (As a caveat, I get about a hundred emails a week, so I can't reply to all of them — but know that I will do my best.)
Turns out they were using PowerBI lol
As for the latter, it's no automation, it's just a nice "autocomplete" feature.
Isn't that the same for your new coworker Bob over in that corner of the room? Sure, his "failure modes" might look different, but how can you tell whether someone (or something) "understands" what you want to query? And how can you be sure they got it right if you don't double check?
In my eyes GPT-3 is akin to a freshly hired grad that can't operate google (so you have to send him some related stackoverflow examples for his task). You can't trust that the result is right in either case, but both might bring pretty decent domain knowledge and - when given work accordingly - could save you time in the end.
The ethics implications are... fascinating.
But something else humans can do is look at the data to construct patterns as another way to extract data. To some degree it requires some (bad) inductive reasoning approaches like “I’ll assume maybe most or all data in this column has this format.” “Oops, it didn’t, so let’s tweak it to look like this, to cover it sometimes also being empty string or null.”