Notebooks are great for invoking existing functions and exploring data.

Notebooks aren't ideal for creating functions (standard text editor features are lacking and testing is impossible).

Notebooks encourage an "order dependent variable assignment" programming style without abstractions. Here's what you'll commonly see in a notebook:

val df = spark.read.csv("some_data")

df2 = df.withColumn("clean_name", trim("name"))

df3 = df2.filter("clean_name" === "Mark")

I've found that notebooks are very useful if you write all the complicated code in separate GitHub repos and attach binary executables to the cluster. If you try to write all your logic in notebooks, you'll quickly struggle with order dependent, messy code.