wenmin-wu/cv-phash-duplicate-grouping

Perceptual Hash Duplicate Grouping

Overview

Perceptual hashing (pHash) produces a compact fingerprint that is identical for visually similar images regardless of resolution or minor edits. Group items sharing the same hash to find near-duplicates without any model inference. Use as a cheap baseline or combine with learned embeddings for higher recall.

Quick Start

import pandas as pd

def phash_group_matches(df, hash_col='image_phash', id_col='posting_id'):
    """Group items by perceptual hash.
    
    Args:
        df: DataFrame with hash and ID columns
        hash_col: column containing perceptual hash strings
        id_col: column containing item identifiers
    Returns:
        Series mapping each item to its hash-group matches
    """
    hash_groups = df.groupby(hash_col)[id_col].agg(list).to_dict()
    return df[hash_col].map(hash_groups)

# Usage
df['phash_matches'] = phash_group_matches(df)

# Compute hash if not provided
from PIL import Image
import imagehash
df['image_phash'] = df['image_path'].apply(
    lambda p: str(imagehash.phash(Image.open(p)))
)

Key Decisions

• Zero compute cost: no GPU needed — hash lookup is O(1) per item
• High precision, low recall: only catches near-exact duplicates; combine with KNN for broader matches
• Hash function: pHash is robust to resizing/compression; use dHash for rotation invariance
• Hamming distance: for fuzzy matching, compare hash bit distance instead of exact equality

References

• Source: part-2-rapids-tfidfvectorizer-cv-0-700
• Competition: Shopee - Price Match Guarantee