How Duplicate Detection Works

Duplicate Detection Overview

Duplicate Curation

Potential Duplicates

Do you manage tons of assets and find it difficult to find potential duplicates? Are some users confused about which asset to download because it appears multiple times after a search? Asset duplicate detection is here to consolidate duplicates and provide a clearer overview of your assets.

As you add assets to your Media Library, they will be processed asynchronously to determine their digital fingerprint, that we can use to compare them with other assets of the same Library. The digital fingerprint is a hash value created by our algorithm. The technical term for this is a perceptual hash (pHash). We currently review the following formats for this comparison; .dng, .eps, .gif, .jpg, .png, .psd, .tif, .ttf. and .webp.

Copies or duplicates will show the same fingerprint and thus be flagged by the algorithm. The potential duplicate status that an asset may receive if the hash values of two assets are identical will not disturb other workflows in your Library. With this hash value, we can display the existing duplicates to you in your Media Library and give you the options to review and manage them.

Duplicate Detection is based on the image itself and will detect images that are very close to each other visually. Hence even if your assets are in 2 different formats (png & jpg for example), are of different sizes, and have different metadata - they will still be detected as duplicates.

Editors will find the duplicate overview in the backend of the Library by navigating to it through the left sidebar duplicate icon.

The overview displays groups of potential duplicates in an inbox-style view. The list of duplicates groups on the left provides the option to jump quickly from group to group. You can also scroll continuously on the right side of the screen to navigate between groups on the asset level.

The right side of the screen displays the assets with key information. Clicking on the asset will open the asset detail view if more details are needed. Clicking on the Collection names and linked Guidelines from this info card will jump you to those pages for easier navigation.

After reviewing the potential duplicates within the group, you can form a decision on what to do with the assets.

From the duplicate detection overview, the most prominent action you can carry out is the deletion of a potential duplicate from a duplicate group. If only one asset remains in the duplicate group after deletion, then the group will be automatically resolved.

In some cases, you may be unable to remove the duplicate due to existing relations to other assets or connections to guidelines. These conflicts must be removed before the duplicate can be deleted from this overview.

You also have the option to resolve a duplicate group. Resolving will remove the duplicate group from the main duplicate overview. It makes most sense to resolve a duplicate group if you decide to keep the remaining assets of the group and plan no further action on their duplicate status.

Our perceptual hash intentionally not only displays exact duplicates but may also show potential duplicates where e.g. only a minor color difference can be observed between the images, but they are otherwise identical. No worries if you want to keep both, simply resolve the potential duplicate group.

Here is an example of a potential duplicate: