sadjad6/data_deduplicator
src/skills/data_engineering/data_deduplicator/SKILL.md
Find and remove duplicate records
data-ai
Updated Apr 13, 2026
$ install --global
skillsauthnpx skillsauth add sadjad6/open-autonomous-research-lab data_deduplicatorInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: Apr 24, 2026, 8:44 PM1.9s1 file scanned
SKILL.md
- name:
- data_deduplicator
- description:
- Find and remove duplicate records
Data Deduplicator
Description
Find and remove duplicate records
When to Use
When data may contain duplicates
Workflow
- Identify key columns\n2. Find duplicates\n3. Remove or merge
Tools Used
pandas
Examples
from src.skills.data_engineering.data_deduplicator import Skill
skill = Skill()
result = await skill.run(SkillInput(data={"key": "value"}))
Best Practices
Consider fuzzy matching for near-duplicates
Related Skills
sadjad6/scatter_plot
tools
VerifiedTrustedCommunity
Generate scatter plot visualizations
SKILL.mdUpdated Apr 13, 2026
sadjad6/report_chart
testing
VerifiedTrustedCommunity
Generate publication-quality charts
SKILL.mdUpdated Apr 13, 2026
sadjad6/pie_chart
tools
VerifiedTrustedCommunity
Generate pie chart visualizations
SKILL.mdUpdated Apr 13, 2026
sadjad6/line_chart
tools
VerifiedTrustedCommunity
Generate line chart visualizations
SKILL.mdUpdated Apr 13, 2026