laurigates/nushell-data-processing

Nushell Data Processing

Expert knowledge for structured data processing using nushell, a shell that treats data as tables rather than text streams.

When to Use This Skill

| Use this skill when... | Use jq-json-processing instead when... | |---|---| | Performing multi-step transforms across JSON, YAML, CSV, TOML | The input is JSON only and a single jq expression suffices | | Aggregating, grouping, or visually exploring tabular data | The pipeline runs in CI on a minimal image without nushell | | Combining multiple files into one cross-file analysis | Stdin-piping small payloads from gh or curl |

| Use this skill when... | Use yq-yaml-processing instead when... | |---|---| | Working across YAML plus other formats in one pipeline | Editing YAML in place while preserving comments and order | | Aggregating across many Kubernetes manifests | Targeted updates to a single Helm values or workflow file |

Core Advantages Over jq/yq

| Feature | nushell | jq/yq | |---------|---------|-------| | Native tables | Yes - visual output | No - text-based | | Multi-format | JSON, YAML, TOML, CSV, XML | Single format each | | Type awareness | Rich type system | Limited | | Shell integration | Full shell capabilities | Filter only | | Complex transforms | Natural syntax | Complex expressions |

Essential Commands

Opening Data Files

# Auto-detect format from extension
open data.json
open config.yaml
open settings.toml
open report.csv

# Explicit parsing
"[1, 2, 3]" | from json
"name: value" | from yaml
"a,b,c\n1,2,3" | from csv

Format Conversion

# Convert between formats
open data.json | to yaml
open config.yaml | to json
open data.csv | to json

# Compact JSON output (for piping)
open data.json | to json -r

Table Operations

# Select columns
open data.json | select name email

# Filter rows
open data.json | where age > 18
open data.json | where status == "active"
open data.json | where name =~ "^A"  # regex match

# Sort
open data.json | sort-by name
open data.json | sort-by age --reverse

# Limit results
open data.json | first 10
open data.json | last 5
open data.json | skip 2 | first 3

Transformations

# Add/update columns
open data.json | insert full_name {|row| $"($row.first) ($row.last)"}
open data.json | update name {|row| $row.name | str upcase}
open data.json | upsert status "active"

# Rename columns
open data.json | rename old_name new_name

# Remove columns
open data.json | reject password secret_key

# Transpose (pivot)
open data.json | transpose key value

Aggregations

# Group by
open data.json | group-by category

# Group with aggregation
open data.json | group-by --to-table category | update items {|row| $row.items | length}

# Math operations
open data.json | get prices | math sum
open data.json | get scores | math avg
open data.json | get values | math max
open data.json | get values | math min

# Count
open data.json | length
open data.json | where active | length

Iteration with each

# Transform each row
open data.json | each {|row| $"($row.name): ($row.value)"}

# Parallel processing
open data.json | par-each {|row| expensive_operation $row}

# With index
open data.json | enumerate | each {|item| $"($item.index): ($item.item.name)"}

Reduce Operations

# Sum values
[1 2 3 4 5] | reduce {|it, acc| $acc + $it}

# Build string
["a" "b" "c"] | reduce {|it, acc| $"($acc),($it)"}

# With initial value
open data.json | get prices | reduce --fold 0 {|it, acc| $acc + $it}

Merging Data

# Merge records
{a: 1} | merge {b: 2}

# Deep merge (nested structures)
{a: {foo: 1}} | merge deep {a: {bar: 2}}

# Append to list
[1 2 3] | append [4 5 6]

Real-World Examples

Compare Manifest Versions

# Cross-reference versions between files
open .release-please-manifest.json
| transpose plugin manifest_ver
| each {|row|
    let pjson = $"($row.plugin)/.claude-plugin/plugin.json"
    let pjson_ver = if ($pjson | path exists) {
        (open $pjson).version? | default "NOT SET"
    } else {
        "MISSING"
    }
    let status = if $row.manifest_ver == $pjson_ver { "OK" } else { "MISMATCH" }
    {plugin: $row.plugin, manifest: $row.manifest_ver, plugin_json: $pjson_ver, status: $status}
}
| sort-by plugin

Process API Response

# GitHub API - list repos with stats
http get "https://api.github.com/users/octocat/repos"
| select name stargazers_count forks_count
| sort-by stargazers_count --reverse
| first 10

Analyze Package Dependencies

# List all dependencies with versions
open package.json
| get dependencies
| transpose name version
| sort-by name

Aggregate Log Data

# Count errors by type
open logs.json
| where level == "error"
| group-by --to-table error_type
| update items {|row| $row.items | length}
| rename error_type count
| sort-by count --reverse

Multi-File Analysis

# Find all package.json files and extract versions
glob "**/package.json"
| each {|f| {file: $f, version: (open $f | get version? | default "none")}}
| sort-by file

CSV Processing

# Read CSV with headers
open data.csv
| where amount > 100
| select date description amount
| sort-by date

# Write CSV
open data.json | to csv | save output.csv

Agentic Optimizations

| Context | Command | |---------|---------| | Compact JSON | nu -c 'open data.json \| to json -r' | | Quick filter | nu -c 'open data.json \| where status == "active" \| length' | | Version check | nu -c 'open package.json \| get version' | | Format convert | nu -c 'open config.yaml \| to json' | | One-liner | nu -c '<command>' |

Quick Reference

Common Flags

| Flag | Description | |------|-------------| | -c | Execute command string | | -r | Raw output (no formatting) | | --reverse | Reverse sort order | | --to-table | Output as table (group-by) |

Type Conversions

| From | To | Command | |------|-----|---------| | String | Int | "42" \| into int | | String | Float | "3.14" \| into float | | Int | String | 42 \| into string | | List | Table | [[a b]; [1 2] [3 4]] |

String Operations

"hello" | str upcase           # HELLO
"HELLO" | str downcase         # hello
"  trim  " | str trim          # trim
"hello world" | split row " "  # [hello, world]
["a" "b"] | str join ","       # a,b
"hello" | str contains "ell"   # true
"hello" | str replace "l" "L"  # heLLo

Path Operations

"/path/to/file.txt" | path basename    # file.txt
"/path/to/file.txt" | path dirname     # /path/to
"/path/to/file.txt" | path extension   # txt
"relative/path" | path expand          # /absolute/path
"/some/path" | path exists             # true/false

When to Use Nushell vs jq/yq

| Use Case | Tool | |----------|------| | Simple JSON field extraction | jq (faster startup) | | Simple YAML field extraction | yq (faster startup) | | Complex multi-step transforms | nushell | | Multi-format processing | nushell | | Visual data exploration | nushell | | Cross-file operations | nushell | | Aggregations/grouping | nushell |

Installation

# macOS
brew install nushell

# Verify
nu --version

Resources

• Official Book: https://www.nushell.sh/book/
• Command Reference: https://www.nushell.sh/commands/
• Cookbook: https://www.nushell.sh/cookbook/