404kidwiz/csv-data-wrangler

CSV Data Wrangler

Purpose

Provides expertise in efficient CSV file processing, data cleaning, and transformation. Handles large files, encoding issues, malformed data, and performance optimization for tabular data workflows.

When to Use

• Processing large CSV files efficiently
• Cleaning and validating CSV data
• Transforming and reshaping datasets
• Handling encoding and delimiter issues
• Merging or splitting CSV files
• Converting between tabular formats
• Querying CSV with SQL (DuckDB)

Quick Start

Invoke this skill when:

• Processing large CSV files efficiently
• Cleaning and validating CSV data
• Transforming and reshaping datasets
• Handling encoding and delimiter issues
• Querying CSV with SQL

Do NOT invoke when:

• Building Excel files with formatting (use xlsx-skill)
• Statistical analysis of data (use data-analyst)
• Building data pipelines (use data-engineer)
• Database operations (use sql-pro)

Decision Framework

Tool Selection by File Size:
├── < 100MB → pandas
├── 100MB - 1GB → pandas with chunking or polars
├── 1GB - 10GB → DuckDB or polars
├── > 10GB → DuckDB, Spark, or streaming
└── Quick exploration → csvkit or xsv CLI

Processing Type:
├── SQL-like queries → DuckDB
├── Complex transforms → pandas/polars
├── Simple filtering → csvkit/xsv
└── Streaming → Python csv module

Core Workflows

1. Large CSV Processing

1. Profile file (size, encoding, delimiter)
2. Choose appropriate tool for scale
3. Process in chunks if memory-constrained
4. Handle encoding issues (UTF-8, Latin-1)
5. Validate data types per column
6. Write output with proper quoting

2. Data Cleaning Pipeline

1. Load sample to understand structure
2. Identify missing and malformed values
3. Define cleaning rules per column
4. Apply transformations
5. Validate output quality
6. Log cleaning statistics

3. CSV Query with DuckDB

1. Point DuckDB at CSV file(s)
2. Let DuckDB infer schema
3. Write SQL queries directly
4. Export results to new CSV
5. Optionally persist as Parquet

Best Practices

• Always specify encoding explicitly
• Use chunked reading for large files
• Profile before choosing tools
• Preserve original files, write to new
• Validate row counts before/after
• Handle quoted fields and escapes properly

Anti-Patterns

| Anti-Pattern | Problem | Correct Approach | |--------------|---------|------------------| | Loading all to memory | OOM on large files | Use chunking or streaming | | Guessing encoding | Corrupted characters | Detect with chardet first | | Ignoring quoting | Broken field parsing | Use proper CSV parser | | No validation | Silent data corruption | Validate row/column counts | | Manual string splitting | Breaks on edge cases | Use csv module or pandas |