For example, let's assume you use duckdb or mysql or other standard sql for data processing, how do you deal with intermediate tables?
You may have to frequently use CREATE TABLE and INSERT or write gigantic WITH statement to avoid the hassle. I guess many people use both.
In Fugue SQL, you can use simple assignment to do the same thing (Fugue may translate the assignment to CREATE TABLE or do something more efficient)
-------
a = CONNECT duckdb SELECT * FROM table1
b = LOAD "/tmp/src.parquet"
c = SELECT * FROM a LEFT SEMI JOIN b ON a.key = b.key
OUTPUT c USING mysql_dumper(table_name="temp", mode="append")
d = SELECT * FROM b UNION SELECT * FROM c
SAVE d OVERWRITE "/tmp/tmp.parquet"
--------
Look at the above Fugue SQL example, it's self-explaining, but how much extra code you have to write to achieve the same semantic without Fugue SQL?
Fugue SQL can provide a unified and minimal way to express your logic in SQL-ish way, while the real executions are done but those wonderful database solutions you mentioned.
For example, with a python function
```python
# schema: , c:int
def add_new_col(df:pd.DataFrame) -> pd.DataFrame:
df['c'] = 1
return df
fsql(query).run()
# to run on Spark
fsql(query).run(SparkExecutionEngine)
```