anhvth/io-utilities

IO Utilities Guide

This skill provides comprehensive guidance for using the IO utilities in speedy_utils.

When to Use This Skill

Use this skill when you need to:

• Read and write data in various formats (JSON, JSONL, Pickle, CSV, TXT).
• Efficiently process large JSONL files with streaming and multi-threading.
• Automatically handle file compression (gzip, bz2, xz, zstd).
• Load data based on file extension automatically.
• Serialize Pydantic models and other objects easily.

Prerequisites

• speedy_utils installed.
• Optional dependencies for specific features:
  • orjson: For faster JSON parsing.
  • zstandard: For .zst file support.
  • pandas: For CSV/TSV loading.
  • pyarrow: For faster CSV reading with pandas.

Core Capabilities

Fast JSONL Processing (fast_load_jsonl)

• Streams data line-by-line for memory efficiency.
• Supports automatic decompression.
• Uses orjson if available for speed.
• Supports multi-threaded processing for large files.
• Shows progress bar with tqdm.

Universal Loading (load_by_ext)

• Detects file type by extension.
• Supports glob patterns (e.g., data/*.json) and lists of files.
• Uses parallel processing for multiple files.
• Supports memoization via do_memoize=True.

Serialization (dump_json_or_pickle, load_json_or_pickle)

• Unified interface for JSON and Pickle.
• Handles Pydantic models automatically.
• Creates parent directories if they don't exist.

Usage Examples

Example 1: Streaming Large JSONL

Read a large compressed JSONL file line by line.

from speedy_utils import fast_load_jsonl

# Iterates lazily, low memory usage
for item in fast_load_jsonl('large_data.jsonl.gz', progress=True):
    process(item)

Example 2: Loading Any File

Load a file without worrying about the format.

from speedy_utils import load_by_ext

data = load_by_ext('config.json')
df = load_by_ext('data.csv')
items = load_by_ext('dataset.pkl')

Example 3: Parallel Loading

Load multiple files in parallel.

from speedy_utils import load_by_ext

# Returns a list of results, one for each file
all_data = load_by_ext('logs/*.jsonl')

Example 4: Dumping Data

Save data to disk, creating directories as needed.

from speedy_utils import dump_json_or_pickle

data = {"key": "value"}
dump_json_or_pickle(data, 'output/processed/result.json')

Guidelines

1. Prefer JSONL for Large Datasets:

   • Use fast_load_jsonl for datasets that don't fit in memory.
   • It handles compression transparently, so keep files compressed (.jsonl.gz or .jsonl.zst) to save space.

2. Use load_by_ext for Scripts:

   • When writing scripts that might accept different input formats, use load_by_ext to be flexible.

3. Error Handling:

   • fast_load_jsonl has an on_error parameter (raise, warn, skip) to handle malformed lines gracefully.

4. Performance:

   • Install orjson for significantly faster JSON operations.
   • load_by_ext uses pyarrow engine for CSVs if available, which is much faster.

Limitations

• Memory Usage: load_by_ext loads the entire file into memory. Use fast_load_jsonl for streaming.
• Glob Expansion: load_by_ext with glob patterns loads all matching files into memory at once (in a list). Be careful with massive datasets.