wenmin-wu/tabular-time-varying-reward-shaping

Time-Varying Reward Shaping

Overview

In resource-management games, the optimal strategy shifts over time: early game rewards fleet/structure building (assets), late game rewards resource hoarding. Time-varying reward shaping encodes this by linearly interpolating weights between assets and resources as a function of turn number, plus a terminal win bonus inversely proportional to game length.

Quick Start

import numpy as np

max_steps = 400
transition_point = 300

w_assets = np.concatenate([
    np.linspace(1.0, 0.0, transition_point),
    np.zeros(max_steps - transition_point)
])
w_resources = np.concatenate([
    np.linspace(0.0, 1.0, transition_point),
    np.ones(max_steps - transition_point)
])

def board_value(player, turn, ship_cost=10, yard_cost=50):
    val_resources = w_resources[turn] * player.kore
    val_ships = w_assets[turn] * ship_cost * player.total_ships
    val_yards = w_assets[turn] * yard_cost * player.total_shipyards
    return val_resources + val_ships + val_yards

def reward(board, prev_board, turn, done, won):
    r = board_value(board.current_player, turn) - board_value(prev_board.current_player, turn - 1)
    if done:
        bonus = (1 if won else -1) * (100 + 5 * (max_steps - turn))
        r += bonus
    return r

Workflow

1. Define linear weight schedules for assets and resources over the game horizon
2. Compute board value as weighted sum of resources, fleet value, and structure value
3. Reward = delta board value between consecutive steps
4. Add terminal bonus/penalty scaled by remaining turns (faster wins → bigger bonus)
5. Tune the transition point and weight slopes on validation episodes

Key Decisions

• Transition point: set to ~75% of max steps; too early and the agent hoards prematurely
• Asset valuation: weight structures more than ships (shipyards compound value over time)
• Win bonus scaling: remaining_turns * constant encourages decisive play over stalling
• Delta vs. absolute: delta reward (change between steps) gives denser signal than absolute value

References

• Reinforcement Learning baseline in Python