We have a similar setup, but I've been generating UUIDs for each event at logging time to avoid any computational overhead for deduping.
https://www.elastic.co/guide/en/logstash/current/plugins-fil....
Edit: typo
(These are micro-optimizations anyway.)