This app prevents duplicate events from being ingested into PostHog. It's particularly helpful if you're backfilling information while you're already ingesting ongoing events.
How it works
The Unduplicator crafts an event UUID based on key properties for the event, so if the event is the same (see below for definition) it'll end with the same UUID.
When events are processed by ClickHouse, the database will automatically dedupe events which have the same toDate(timestamp), event, distinct_id and uuid combo, effectively making sure duplicates are not stored.
The app has two modes that define what's considered a duplicate event. Either mode will prevent duplicates within a project, though duplicates across projects are still permitted.
- Event and Timestamp. An event will be treated as duplicate if the timestamp, event name and user's distinct ID matches exactly, regardless of what internal properties are included.
- All Properties. An event will be treated as duplicate only if all properties match exactly, as well as the timestamp, event name and user's distinct ID.
Requirements
The Unduplicator requires either PostHog Cloud, or a self-hosted PostHog instance running version 1.30.0 or later.
Not running 1.30.0? Find out how to update your self-hosted PostHog deployment!
Installation
- Log in to your PostHog instance
- Click 'Apps' on the left-hand tool bar
- Search for 'Unduplicates'
- Select the Unduplicator app, press 'Install'.
- Once the app is installed, press the blue button to configure the app and select which of the de-duplication methods you want to use (described above).
Configuration
FAQ
Is the source code for this app available?
PostHog is open source and so are all apps on the platform. The source code for the Unduplicator app is available on GitHub.
Who created this app?
We'd like to thank former PostHog team member Paolo D'Amico for creating the Unduplicator. We miss you, Paolo!
Who maintains this app?
This app is maintained by the community. If you have issues with the app not functioning as intended, please let us know!
What if I have feedback on this app?
We love feature requests and feedback! Please tell us what you think! to tell us what you think.
What if my question isn't answered above?
We love answering questions. Ask us anything via our community forum, or drop us a message.