Ask HN: How to partition by event processing time in Airflow?
For an example, if I have a system with events entering some stream (e.g. Kafka or Kinesis) and then periodically the data is written to storage (e.g. S3 or other) which are then batch processed on some schedule (e.g. airflow), then there are multiple 'time' values to consider.
t1 -> time of the event occurring t2 -> time of the event entering the stream t3 -> time of persisting batch of events to storage t4 -> time of batch run (airflow) for further processing
What is considered the "event processing" time in this case? How is a partition generated so that at immutable block of data can be landed predictably? Presumably there must be some deterministic pattern for generating batch runs so that time partitions are immutable and so that backfill tasks can be generated.