nakane1chome/generator-coding

Template-Based Code Generation

Three inputs produce repetitive (typically interface) code:

• Data model: Standardized representation -- XML, Domain-Specific Language (DSL), or schema-enforced JSON/YAML.
• Template: Templating meta-language markup (Jinja2, ERB, etc.).
• Helper functions: Functions that reshape the data model for template consumption.

Data Model ──► Parser ──► Helpers ──► Template Engine ──► Generated Output
   (SVD,         (cmsis-svd,   (reshape,    (Jinja2,        (one file per
    IDL,          sqlparse)     filter)      ERB)            generation unit)
    DBML...)

Terms

| Term | Definition | |------|-----------| | Data Model | Input describing the structure to replicate (e.g. SVD file for hardware registers). | | Helpers/Plugins | Functions that transform/filter/restructure the data model for templates. | | Template(s) | Defines output structure in the target language with template markup for variable parts. | | Target Language | Language of the generated output (e.g. C++, Python, Verilog). | | Generated Output | Files produced by the generator. Must never be hand-edited (see: Rules). | | Generation Unit | Data model component a template operates on (e.g. peripheral, table, message). One output file per unit (see: Step 3). | | Data Bridge Template | Template converting the data model into target-language compile-time representations (enums, constants, macros). Consumed by in-language meta-programming (see: Meta-Programming Rules). |

Generator Types

| Type | Description | Template Location | Flexibility | |------|-------------|-------------------|-------------| | Fixed function | Template embedded in generator tool code | Inside the tool | Output format fixed; change requires modifying the tool | | Customizable template-based | Template separated from tool | User-editable files | User can modify/replace templates freely |

Directive: Prefer customizable template-based generators for agent use. The agent can read, understand, and modify templates without understanding the tool's internals.

Meta-Programming

| Type | Mechanism | When to use | |------|-----------|-------------| | In-language | C++ templates/constexpr/consteval, C preprocessor, Python decorators, Verilog generate | Data can be represented in target language compile-time constructs | | External template-based | Jinja2/ERB generating source files | Data cannot be naturally represented in target language (e.g. register maps, protocol definitions) |

Directive: Prefer in-language meta-programming -- the target toolchain handles validation, error reporting, and IDE support. Combine with external generation using the layered approach (see: Meta-Programming Rules).

Examples

Generator Tools

| Tool | Type | Templating | Domain | |------|------|-----------|--------| | Protobuf (protoc) | Fixed | Embedded | Serialization / RPC | | Cyclone DDS (idlc) | Fixed | Embedded | DDS/OMG IDL interfaces | | SWIG | Fixed | Embedded | Foreign Function Interface (FFI) wrappers | | Ruby on Rails | Template | ERB | Web scaffolding | | Cookiecutter | Template | Jinja2 | Project scaffolding | | development-utils | Template | Jinja2 | Hardware description (SVD, SystemRDL, DBML, Device Tree, IDL) |

Data Model Formats

| Format | Domain | Type | |--------|--------|------| | SQL DDL | Databases | Data exchange | | XML (SVD, ATML) | Hardware, general | Data exchange | | JSON / YAML / TOML | General (schema required) | Data exchange | | IDL (Interface Definition Language) | Distributed systems (CORBA, DDS, Protobuf) | DSL | | SystemRDL | Register descriptions | DSL | | DBML (Database Markup Language) | Database schemas | DSL | | AADL (Architecture Analysis and Design Language) | Safety-critical systems | DSL | | SysML | Systems modeling | DSL | | Device Tree | Linux/embedded hardware | DSL |

Templating Languages

| Language | Ecosystem | Logic | Strengths | |----------|-----------|-------|-----------| | Jinja2 | Python | Full (loops, macros, filters) | Most expressive; StrictUndefined; dominant for standalone generators | | ERB | Ruby | Full Ruby execution | Native to Rails | | Liquid | Ruby | Safe subset | Sandboxed; safe for user-supplied templates | | Mako | Python | Full Python execution | Fast; used by Alembic | | Mustache/Handlebars | Language-agnostic | Logic-less | Forces logic into helpers |

Process

Step 1: Identify the Data Model

Use the user's existing data model. If none exists, select a domain standard:

| Domain | Standard formats | |--------|-----------------| | Hardware registers | SVD, SystemRDL, IP-XACT (IEEE 1685) | | Communication protocols | Protobuf, IDL, ASN.1 | | Database schemas | SQL DDL, DBML | | APIs | OpenAPI/Swagger, GraphQL | | Hardware architecture | Device Tree, AADL |

Step 2: Understand the Interface Boundary

Identify the specification-defined interface:

| Boundary | Examples | |----------|---------| | Persistent data | Database tables, file formats | | Inter-Process Communication (IPC) | Shared memory, message queues, pub/sub | | Hardware/software | Memory-Mapped I/O (MMIO) registers, Direct Memory Access (DMA) descriptors | | Language interop | Java Native Interface (JNI), Python C extensions, FFI | | Network protocols | RPC definitions, serialization formats |

Step 3: Determine the Generation Granularity

| Granularity | Scope | Use case | |-------------|-------|----------| | Component-level | One output file per generation unit (peripheral, message, table) | Direct usage patterns; simpler templates | | System-level | One output file covering all components | Cross-cutting concerns (dispatch tables, device maps, schema migrations) |

Split templates by granularity. A single system-wide template that includes all components is difficult to maintain.

Step 4: Template Output Conventions

| Convention | Pattern | Example | |------------|---------|---------| | Template naming | <generation_unit>_<output_name>.<target_ext>.jinja2 | peripheral_regs.hpp.jinja2, table_model.py.jinja2 | | Output directory | Separate from hand-written source: generated/, gen/, build/gen/ | Never mix generated and hand-written files | | Version control | Add output directories to .gitignore | Exception: version output if users lack the generator toolchain |

Step 5: Write the Generator Pipeline

| Stage | Responsibility | Notes | |-------|---------------|-------| | Parser | Read and validate data model; produce in-memory representation | Reuse open-source parsers: cmsis-svd, systemrdl-compiler, sqlparse. Custom parser = last resort. | | Helpers | Reshape/filter parsed data for template consumption | Keep logic out of templates | | Templates | Emit target-language constructs; read like target source code | Minimal control flow; iterate over data | | Driver script | Connect parser + helpers + templates; write output files | Script or Makefile target |

Rules for Using Generators

| # | Rule | Detail | |---|------|--------| | 1 | Never edit generated output | Fix the data model, template, or helper instead. If output is wrong, fix the input. | | 2 | Interface changes go in the data model | New field/register/message = modify the data model source, not the template. | | 3 | Data reshaping goes in helpers | Filtering, renaming, restructuring = helper functions. Keep templates simple. | | 4 | Templates read like target source code | Template markup ({% for %}, {{ }}) is a thin layer over recognizable target code. | | 5 | Control whitespace explicitly | Jinja2: enable trim_blocks + lstrip_blocks. Use {%-/-%} for fine-grained control. Verify output formatting. | | 6 | One concern per template | Each template produces one logical output artifact. |

Rules for Meta-Programming with Generators

When combining external generation with in-language meta-programming:

| # | Rule | Detail | |---|------|--------| | 1 | Prefer target-language mechanisms | C++ constexpr/consteval, Python metaclasses, Verilog generate, C macros. | | 2 | Data bridge template | One template converts data model to target-language compile-time data (enums, constants, constexpr arrays). | | 3 | Separate runtime templates | Consume compile-time data via target-language meta-programming (e.g. C++ template class parameterized by generated enum). | | 4 | Layering principle | External generation = data declarations. In-language meta-programming = behavior. Each layer independently testable. |

Testing Generators

| Strategy | What it catches | How | |----------|----------------|-----| | Golden-file tests | Unintended output changes | Store known-good output as reference; diff against current output | | Compilation tests | Syntax errors, type mismatches | Compile (or lint) generated output after generation | | Round-trip tests | Semantic correctness | Verify generated code represents the data model correctly (e.g. register access returns expected values) | | Sample data models | Template edge cases | Maintain small test data exercising all control flow paths (optional fields, empty lists, single-element lists) |

Build System Integration

| Concern | Approach | |---------|----------| | Dependency tracking | Makefile/script declares deps on data model + templates + helpers. Any input change triggers regeneration. | | Makefile pattern | generated/%.hpp: data/%.xml templates/peripheral_regs.hpp.jinja2 helpers.py | | Incremental generation | For large data models, regenerate only changed generation units. | | CI verification | Run generator in CI; verify checked-in output (if versioned) matches generator output. |

Post-Generation Hooks

| Hook | Purpose | Example | |------|---------|---------| | Code formatters | Match project style without burdening templates | clang-format (C/C++), black (Python), rustfmt (Rust) | | Linters | Catch issues template engine cannot detect | Unused imports, naming violations | | Hook integration | Run in driver script after rendering | clang-format -i generated/*.hpp |

Error Handling

| Scenario | Action | |----------|--------| | Malformed data model | Validate against schema before rendering. Report errors with file location + field name. Do not render invalid data. | | Missing template fields | Use strict undefined handling (Jinja2: undefined=StrictUndefined). Fail loudly. | | Non-deterministic output | Same inputs must produce identical output. If not, there is a bug in the generator. |

Example Generator Repository

development-utils demonstrates:

| Aspect | Details | |--------|---------| | Parsers | Python-based: SVD, SystemRDL, DBML, Device Tree, IDL | | Output | C++17/C++20 MMIO register interfaces, SQLAlchemy/Pydantic models from DBML, RISC-V CSR access code (C/C++/Rust) | | Naming | <generation_unit>_<name>.<ext>.jinja2 | | Granularity | Component-level (per peripheral) + system-level (device maps) |