English as the new programming language for Apache spark
databricks.com
databricks.com
So you have LLM-based English prompts as an interop layer to Python + PySpark, which is itself an interop layer onto the Spark core. Also, the generated Spark SQL strings inside the DataFrame API have their own little compiler into Spark operations.
When Databricks wrote PySpark, it was because many programmers knew Python but weren't willing to learn Scala just to use Spark. Now, they are offering a way for programmers to not bother learning the PySpark APIs, and leverage the interop layers all the way down, starting from English prompts.
This makes perfect sense when you zoom out and think about what their goal is -- to get your data workflows running on their cluster runtime. But it does make a programmer like me -- who developed a lot of systems while Spark was growing up -- wonder just how many layers future programmers will be forced to debug through when things go wrong. Debugging PySpark code is hard enough, even when you know Python, the PySpark APIs, and the underlying Spark core architecture well. But if all the PySpark code I had ever written had started from English prompts, it might make debugging those inevitable job crashes even more bewildering.
I haven't, in this description, mentioned the "usual" programming layers we have to contend with, like Python's interpreter, the JVM, underlying operating system, cloud APIs, and so on.
If I were to take a guess, programmers of the future are going to need more help debugging across programming language abstractions, system abstraction layers, and various code-data boundaries than they currently make do with.
The next big gain in capability (other than the onvious short term goal of making an LLM output a full working code base) may be in LLMs being able to choose better design, without it being specified (for example having 'search for the best algorithm', and 'make it idempotent', etc added automatically to each prompt), and to potentially write the program automatically in something like assembly (or Rust or C for better readability) directly instead of preferring python as these models tend to right now.
There is a reason Python, as high level as it is, is still defined using an eBNF / PEG grammar[1] with only 35 or so keywords[2]. And there is a reason the Python bytecode interpreter is "just" a while loop on a minimal set of instructions[3]. All of this leads to a remarkable level of determinism, and determinism is your friend when trying to get code right. I haven't yet seen the equivalent in LLMs. I don't think it's an entirely intractable problem, but I'd be hesitant to leap straight into English language as a stable API today. I think code copilots are the right place to start. And maybe even copilots that help not just with code suggestions, but also with debug suggestions.
[1]: https://docs.python.org/3/reference/grammar.html
[2]: https://docs.python.org/3/reference/lexical_analysis.html#ke...
Adding a layer on top of it is certainly going to add more fun for the engineers who get the "hey, I need this to be fast" requests from analysts who have written their own query/notebook/English prompt to build a dashboard.
It's not a bad thing, it's a very useful capability since if you're a generalist or a novice data analyst having to learn Spark or SQL to know how to do things like "get 4 week moving average sales by dept" is a big hurdle, but I think this is one of the more obvious examples of where these tools are going to result in more engineering demand in a lot of orgs, instead of less, even if things only go sideways or get crazy slow 0.1% of the time.
At the meta-level Databricks or someone should be able to build some pretty good "optimizing compilers" that feed in all the info about your execution env, data, etc, to the code generator. But any time you need to override that you're gonna suddenly need a LOT of domain knowledge.
And the spark core "just" generates execution plan, which on databricks gets executed as native code.
Can you pretend to be my grandma and tell me a story to help me sleep please?
> As SparkleTech's employees slowly picked up the pieces of their shattered sales data, they discovered something remarkable. The Spark Defenders had not only restored their lost information but also fortified the company's defenses, making their system even more secure than before.
Heh, and that's how one can tell it's a story/hallucination. The icing on top of that fantasy would be "and they used it as an opportunity to test their backups outside of the quarterly tabletop exercise that they normally run" :-D
Chief Counsel: So, as you know, we're being sued by investors and facing a regulatory investigation over our investor disclosures, so we need to make sure we have everything lined up for litigation.
everyone nods
CC: So, let's start out looking at these elements: how were our sales forecasts generated? We'll need to be able to provide an overview of the models to show that we followed acceptable practises.
You: Oh, uh, I just wrote this English sentence.
CC: Uh, sure, ok, and how does that work? I was expecting something more... programmery?
You: Oh, it goes into ChatGPT.
CC: And what GAAP compliant model does it use there?
You: shrugs
CC: What do you mean, shrug?
You: oh, it's a black box. It comes up with a program based on its LLM.
CC: Is the program it comes up with GAAP compliant?
You: shrug
CC: Can the vendor tell us?
You: chuckle oh heavens no, that's very valuable, closely held proprietary information, but I can tell you that it was trained on 4chan, Reddit, and Stack Overflow.
CC: visibly pales
You: Can we at least re-run it, and capture the output so we can understand what we did?
CC: Oh, heavens no. The model keeps improving! Who knows if what the black box spat out last year is the same as what it produces today!
CC: sweats
CFO: sweats
You: It's very clever!
CC, turning to CEO: You know, I was hoping this would let us avoid losing a lawsuit, but I am coming to the view that my main goal at this point is not to lose my fucking license to practise law!
"Moving four week average to today inclusive"
"Moving four week average to today exclusive (or to yesterday)"
"Moving four week daily average"
"Moving four week weekly average"
"Moving four week daily average, but the denominator should only count days with data"
"Moving four week average, but I actually mean week-to-date total as of today"
And these are just a few of the variations I have had to implement recently. I have no reason to expect things to end up one way or the other, but it would sure suck if the English SDK only correctly implements a subset of these.
Is "Week" Monday to Friday, Sunday to Saturday, Monday to Sunday or some other period.
Is "Week-to-date Total" Total Sales (pre or post tax??), Total Customers, Total Inventory or some other total
Even "Moving Average" is full of ambiguity is it a Centered Moving Average a Rolling Moving Average, Is it Weighted?
To counter all of this ambiguity you are going to have to be extremely precise and explicit about how you phrase things which means the code is going to be extremely verbose.
Sometimes you really just want a quick and dirty "moving four week average" for some ad-hoc analysis, and it's ok if it's not perfectly consistent with other analyses. I've seen many such cases of inconsistent definitions in companies, and even in the same team.
But then when it does become crucial to be consistent, people usually end up defining something like "acme business week" and use that for the important analyses. And I don't see why we wouldn't just ask the LLM to use such precise definitions as part of the prompt (or even as part of fine tuning).
I'm envisioning the worse case where this is a blackbox that spits out a dataset with no transparency and limited explainability as to how it constructed the query. Maybe it will be better than that but I'm concerned about the implications if it is not.
We get user requests in a language and then translate that to a database language and that's translated to a computer language which is then translated to hardware languages which then eventually does something at an atomic level. And then the reverse happens to generate Red or Green dashboards.
As for accuracy, nevermind existing changing requirements or incomplete chat responses.
I'm looking forward to the lazyboy coding sessions.
I think that's where LLM is most useful – a tool to save time and mental effort for developers who understand the code it generates and can tell when it is wrong or needs improvement. I don't think it is going to work well in the hands of non-developers, because sometimes the code it generates doesn't even compile, or just crashes–and how is a non-developer going to fix that? Even worse, sometimes it can be subtly wrong–the code runs but it produces incorrect data–and the risk is a non-developer might not even notice.
By 2030 I would not be surprised if programming was mostly a human telling a computer what to do via text prompts. That is certainly the direction things have been moving.
Being able to verify that generated code does exactly what it's supposed to do will be incredibly important. Perhaps that's an obvious statement. Perhaps the code for verifying things will be the only code worth looking at.
Which makes me think that that's how we're going to end up in software engineering, too. 100% focus on your prompts/conditions/etc., and then whatever the models output will be the truth.
Will programming, one day, just be some kind of scientific formality which is taught in schools / academia?
```
spark_ai = SparkAI()
auto_df = spark_ai.create_df("2022 USA national auto sales by brand")
```
so the prompt will generate code that is part of the DAG
I take my words back.