ADu2021/autoenv-environment-synthesis

AutoEnv: Automated Environments for Agent Testing

Creating diverse, realistic environments for training and testing agents is expensive and time-consuming. This skill demonstrates how to automate environment generation using a three-layer architectural abstraction combined with LLM coding agents that iteratively synthesize and repair implementation code. The system enables systematic creation of heterogeneous game environments with validated dynamics, observation functions, and rendering pipelines—all driven from high-level YAML specifications.

The core innovation is decomposing environment complexity into three manageable layers (dynamics, observations, rendering) and using coding agents to translate natural language specifications into executable Python code with built-in self-repair loops.

Core Concept

AutoEnv employs a three-layer abstraction architecture to decompose environments systematically:

1. BaseEnv: Implements core dynamics—state space definition, action effects, transition logic, and reward computation
2. ObsEnv: Specializes observation functions through configurable policies that determine what information agents perceive
3. SkinEnv: Applies rendering to convert observations into agent-facing modalities (pixels, text, features)

This layering enables each component to be synthesized and tested independently, with composition handled by the system automatically.

Architecture Overview

• YAML DSL Frontend: High-level environment specification with themes, goals, rules, state variables, and reward conditions
• Prompt Assembly Pipeline: Converts DSL specifications into detailed, contextualized prompts for coding agents
• LLM Coding Agent: Generates implementation code using patterns and examples from libraries
• Three-Stage Verification: Execution testing, level generation validation, and differential model testing
• Self-Repair Loop: Iterative syntax and semantic error correction through LLM-guided debugging

Implementation Steps

The environment synthesis process flows through specification, code generation, validation, and repair stages.

1. Parse and Validate YAML Specification

Convert high-level environment themes into detailed specifications with goals, mechanics, and state variables.

# Example: Simple grid-world specification
environment:
  name: "Treasure Hunt"
  theme: "exploration"
  goal: "collect 3 treasures before time expires"
  state_space:
    - agent_position: [int, int]
    - treasures: list
    - time_remaining: int
  actions:
    - move_up
    - move_down
    - move_left
    - move_right
    - collect
  reward_conditions:
    - collect_treasure: +10
    - step_taken: -0.1
    - timeout: -100

2. Generate Implementation with LLM Coding Agent

Use an LLM agent to convert YAML specs into three Python classes implementing the architecture layers.

def generate_environment_code(yaml_spec, example_patterns):
    """
    Generate BaseEnv, ObsEnv, and SkinEnv implementations from YAML.
    The LLM receives the specification and code patterns to follow.
    """
    prompt = f"""
    Given this environment specification:
    {yaml_spec}

    And these code patterns:
    {example_patterns}

    Generate three Python classes:
    1. BaseEnv: Implements state transitions and rewards
    2. ObsEnv: Implements observation policies
    3. SkinEnv: Implements rendering/modality conversion

    Return complete, runnable code with proper class definitions and docstrings.
    """

    code = llm_agent.generate_code(prompt)
    return code

3. Execute and Test Generated Code

Run the generated environment with brief agent rollouts to detect crashes and verify reward signals.

def verify_execution(generated_code, test_episodes=10):
    """
    Test generated environment for crashes and meaningful reward structure.
    Runs brief agent rollouts to validate dynamics.
    """
    try:
        # Execute the generated code
        env = exec_and_create_env(generated_code)

        # Run test episodes
        for episode in range(test_episodes):
            obs = env.reset()
            total_reward = 0

            for step in range(100):
                # Random action
                action = env.action_space.sample()
                obs, reward, done, info = env.step(action)
                total_reward += reward

                if done:
                    break

            # Verify non-zero rewards
            if total_reward == 0:
                raise ValueError("No reward signal detected")

        return True, "Execution successful"

    except Exception as e:
        return False, str(e)

4. Generate Levels and Validate Reachability

Test that generated levels are solvable—agents can reach valid terminal states.

def validate_level_structure(env, num_tests=20):
    """
    Verify generated levels have valid, reachable goal states.
    Tests agent ability to solve at least one episode per level variation.
    """
    success_count = 0

    for test_idx in range(num_tests):
        obs = env.reset()

        # Simple greedy solver for validation
        solved = attempt_greedy_solution(env, obs, max_steps=500)

        if solved:
            success_count += 1

    solvability_rate = success_count / num_tests

    if solvability_rate < 0.3:
        raise ValueError(f"Too few solvable levels: {solvability_rate:.1%}")

    return solvability_rate

5. Self-Repair Loop for Errors

When verification fails, pass error messages back to the LLM to fix implementation.

def repair_and_retry(yaml_spec, error_message, attempt=1, max_attempts=3):
    """
    Iteratively repair code errors using LLM feedback.
    Continues until code passes all verification stages.
    """
    if attempt >= max_attempts:
        raise RuntimeError(f"Failed after {max_attempts} repair attempts")

    prompt = f"""
    The generated environment code had this error:
    {error_message}

    Original specification:
    {yaml_spec}

    Fix the code to resolve this error. Return the corrected classes.
    """

    repaired_code = llm_agent.generate_code(prompt)
    success, error = verify_execution(repaired_code)

    if not success:
        return repair_and_retry(yaml_spec, error, attempt + 1, max_attempts)

    return repaired_code

6. Differential Model Testing

Verify generated environments produce consistent, non-random reward signals across multiple agent policies.

def differential_model_testing(env, num_models=3, episodes_per_model=5):
    """
    Test that multiple agent policies produce differentiated rewards.
    Validates that environment has meaningful, non-random dynamics.
    """
    policies = [
        RandomPolicy(env.action_space),
        GreedyPolicy(env),
        ExplorationPolicy(env)
    ]

    reward_distributions = []

    for policy in policies:
        episode_rewards = []

        for _ in range(episodes_per_model):
            obs = env.reset()
            total_reward = 0

            for step in range(100):
                action = policy.select_action(obs)
                obs, reward, done, _ = env.step(action)
                total_reward += reward

                if done:
                    break

            episode_rewards.append(total_reward)

        reward_distributions.append(episode_rewards)

    # Check that different policies get different reward distributions
    variance_across_policies = np.var([np.mean(r) for r in reward_distributions])

    if variance_across_policies < 1.0:
        raise ValueError("Policies not differentiated by rewards")

    return True

Practical Guidance

When to Use AutoEnv:

• Creating diverse test suites for agent generalization (10+ environments needed)
• Developing procedural game environments at scale
• Evaluating cross-environment transfer learning
• Rapid prototyping of new RL domains

When NOT to Use:

• Single, highly specialized environments with unique physics (handcraft instead)
• Environments requiring complex 3D graphics or physics engines
• Tasks where environment diversity is less important than fidelity

Key Hyperparameters:

• max_repair_attempts: How many times to retry on error (typically 2-3)
• verification_episodes: Brief rollouts for execution testing (10-20)
• solvability_threshold: Minimum fraction of levels that must be solvable (0.3-0.5)
• num_models_for_differential_testing: How many policies to test (3-5)

Cost Optimization:

• Batch YAML specifications into single prompts to reduce API calls
• Use smaller LLM models for self-repair loops (can achieve 70-80% success with smaller models)
• Cache verified code patterns to reduce regeneration

Integration Pattern: The generated environments are standard OpenAI Gym-compatible, enabling drop-in integration with existing RL pipelines. Set num_envs=K to run K environments in parallel for efficient cross-environment agent training.

Reference

Research paper: https://arxiv.org/abs/2511.19304