SQL is fine.

But as an engineer, I much prefer getting a notebook over a 1000-line plate of SQL spaghetti.

Some of our data scientists prefer SQL and that's fine. We figure out how to speed it up and ship it.

But I've gotten a few of them onto the PySpark+notebook train and it is just a much more productive way of working, IMO. We can extract things into functions with docs & linting. We can easily look at intermediate sub-queries. Yes, you "can" do all the in SQL but the open-source tooling for Python is just really nice.