And they are tasks with standardized best practices. I knew that I wanted to write an internal web app that allowed users to upload a file to S3 using Lambda and storing the CSV rows into Postgres.
I just told it to do it.
It got the “create S3 pre-signed url to upload it to” right. But then it did the naive implementation of download the file and do a bulk upsetting wrong instead of “use the AWS extension to Postgres and upload it to S3”. Once I told it to do that, it knew what to do.
But still I cared about systems and architecture and not whether it decided to use a for loop or while loop.
Knowing that or knowing how best to upload files to Redshift or other data engineering practices aren’t knew or novel