0xDarkMatter/data-processing

Data Processing

Query, filter, and transform structured data (JSON, YAML, TOML) efficiently from the command line.

Tools

| Tool | Command | Use For | |------|---------|---------| | jq | jq '.key' file.json | JSON processing | | yq | yq '.key' file.yaml | YAML/TOML processing |

jq Essentials

# Extract single field
jq '.name' package.json

# Extract nested field
jq '.scripts.build' package.json

# Extract from array
jq '.dependencies[0]' package.json

# Extract multiple fields
jq '{name, version}' package.json

# Navigate deeply nested
jq '.data.users[0].profile.email' response.json

# Filter by condition
jq '.users[] | select(.active == true)' data.json

# Transform each element
jq '.users | map({id, name})' data.json

# Count elements
jq '.users | length' data.json

# Raw string output
jq -r '.name' package.json

yq Essentials

# Extract field
yq '.name' config.yaml

# Extract nested
yq '.services.web.image' docker-compose.yml

# List all keys
yq 'keys' config.yaml

# List all service names (Docker Compose)
yq '.services | keys' docker-compose.yml

# Get container images (K8s)
yq '.spec.template.spec.containers[].image' deployment.yaml

# Update value (in-place)
yq -i '.version = "2.0.0"' config.yaml

# TOML to JSON
yq -p toml -o json '.' config.toml

Quick Reference

| Task | jq | yq | |------|----|----| | Get field | jq '.key' | yq '.key' | | Array element | jq '.[0]' | yq '.[0]' | | Filter array | jq '.[] \| select(.x)' | yq '.[] \| select(.x)' | | Transform | jq 'map(.x)' | yq 'map(.x)' | | Count | jq 'length' | yq 'length' | | Keys | jq 'keys' | yq 'keys' | | Pretty print | jq '.' | yq '.' | | Compact | jq -c | yq -o json -I0 | | Raw output | jq -r | yq -r | | In-place edit | - | yq -i |

When to Use

• Reading package.json dependencies
• Parsing Docker Compose configurations
• Analyzing Kubernetes manifests
• Processing GitHub Actions workflows
• Extracting data from API responses
• Filtering large JSON datasets
• Config file manipulation
• Data format conversion

Additional Resources

For complete pattern libraries, load:

• ./references/jq-patterns.md - Arrays, filtering, transformation, aggregation, output formatting
• ./references/yq-patterns.md - Docker Compose, K8s, GitHub Actions, TOML, YAML modification
• ./references/config-files.md - package.json, tsconfig, eslint/prettier patterns