basher83/python-json-parsing

Python JSON Parsing Best Practices

Comprehensive guide to JSON parsing in Python with focus on performance, security, and scalability.

Quick Start

Basic JSON Parsing

import json

# Parse JSON string
data = json.loads('{"name": "Alice", "age": 30}')

# Parse JSON file
with open("data.json", "r", encoding="utf-8") as f:
    data = json.load(f)

# Write JSON file
with open("output.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)

Key Rule: Always specify encoding="utf-8" when reading/writing files.

When to Use This Skill

Use this skill when:

• Working with JSON APIs or data interchange
• Optimizing JSON performance in high-throughput applications
• Handling large JSON files (> 100MB)
• Securing applications against JSON injection
• Extracting data from complex nested JSON structures

Performance: Choose the Right Library

Library Comparison (10,000 records benchmark)

| Library | Serialize (s) | Deserialize (s) | Best For | |---------|---------------|-----------------|----------| | orjson | 0.42 | 1.27 | FastAPI, web APIs (3.9x faster) | | msgspec | 0.49 | 0.93 | Maximum performance (1.7x faster deserialization) | | json (stdlib) | 1.62 | 1.62 | Universal compatibility | | ujson | 1.41 | 1.85 | Drop-in replacement (2x faster) |

Recommendation:

• Use orjson for FastAPI/web APIs (native support, fastest serialization)
• Use msgspec for data pipelines (fastest overall, typed validation)
• Use json when compatibility is critical

Installation

# High-performance libraries
pip install orjson msgspec ujson

# Advanced querying
pip install jsonpath-ng jmespath

# Streaming large files
pip install ijson

# Schema validation
pip install jsonschema

Large Files: Streaming Strategies

For files > 100MB, avoid loading into memory.

Strategy 1: JSONL (JSON Lines)

Convert large JSON arrays to line-delimited format:

# Stream process JSONL
with open("large.jsonl", "r") as infile, open("output.jsonl", "w") as outfile:
    for line in infile:
        obj = json.loads(line)
        obj["processed"] = True
        outfile.write(json.dumps(obj) + "\n")

Strategy 2: Streaming with ijson

import ijson

# Process large JSON without loading into memory
with open("huge.json", "rb") as f:
    for item in ijson.items(f, "products.item"):
        process(item)  # Handle one item at a time

See: patterns/streaming-large-json.md

Security: Prevent JSON Injection

Critical Rules:

1. Always use json.loads(), never eval()
2. Validate input with jsonschema
3. Sanitize user input before serialization
4. Escape special characters (" and \)

Vulnerable Code:

# NEVER DO THIS
username = request.GET['username']  # User input: admin", "role": "admin
json_string = f'{{"user":"{username}","role":"user"}}'
# Result: privilege escalation

Secure Code:

# Use json.dumps for serialization
data = {"user": username, "role": "user"}
json_string = json.dumps(data)  # Properly escaped

See: anti-patterns/security-json-injection.md, anti-patterns/eval-usage.md

Advanced: JSONPath for Complex Queries

Extract data from nested JSON without complex loops:

import jsonpath_ng as jp

data = {
    "products": [
        {"name": "Apple", "price": 12.88},
        {"name": "Peach", "price": 27.25}
    ]
}

# Filter by price
query = jp.parse("products[?price>20].name")
results = [match.value for match in query.find(data)]
# Output: ["Peach"]

Key Operators:

• $ - Root selector
• .. - Recursive descendant
• * - Wildcard
• [?<predicate>] - Filter (e.g., [?price > 20])
• [start:end:step] - Array slicing

See: patterns/jsonpath-querying.md

Custom Objects: Serialization

Handle datetime, UUID, Decimal, and custom classes:

from datetime import datetime
import json

class CustomEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        if isinstance(obj, set):
            return list(obj)
        return super().default(obj)

# Usage
data = {"timestamp": datetime.now(), "tags": {"python", "json"}}
json_str = json.dumps(data, cls=CustomEncoder)

See: patterns/custom-object-serialization.md

Performance Checklist

• [ ] Use orjson/msgspec for high-throughput applications
• [ ] Specify UTF-8 encoding when reading/writing files
• [ ] Use streaming (ijson/JSONL) for files > 100MB
• [ ] Minify JSON for production (separators=(',', ':'))
• [ ] Pretty-print for development (indent=2)

Security Checklist

• [ ] Never use eval() for JSON parsing
• [ ] Validate input with jsonschema
• [ ] Sanitize user input before serialization
• [ ] Use json.dumps() to prevent injection
• [ ] Escape special characters in user data

Reference Documentation

Performance:

• reference/python-json-parsing-best-practices-2025.md - Comprehensive research with benchmarks

Patterns:

• patterns/streaming-large-json.md - ijson and JSONL strategies
• patterns/custom-object-serialization.md - Handle datetime, UUID, custom classes
• patterns/jsonpath-querying.md - Advanced nested data extraction

Security:

• anti-patterns/security-json-injection.md - Prevent injection attacks
• anti-patterns/eval-usage.md - Why never to use eval()

Examples:

• examples/high-performance-parsing.py - orjson and msgspec code
• examples/large-file-streaming.py - Streaming with ijson
• examples/secure-validation.py - jsonschema validation

Tools:

• tools/json-performance-benchmark.py - Benchmark different libraries