Our team write complex code for Spark to sanitize/anonymize personal data before processing.
With this we might simplify the pipeline because we can keep the same S3 interface.
With this we might simplify the pipeline because we can keep the same S3 interface.