harsh040506/reinforcement-learning

Reinforcement Learning — Full-Stack RL Engine

Provides the complete framework for designing, training, and evaluating reinforcement learning agents, from MDP formalization through policy optimization to deployment in production control systems.

MDP Framework

Every RL problem is formalized as a Markov Decision Process (S, A, P, R, γ):

| Component | Description | Design Considerations | |---|---|---| | S (State Space) | Observation the agent receives | Ensure Markov property; normalize to [0,1] or z-score | | A (Action Space) | Controls available to the agent | Discrete → DQN/PPO; Continuous → SAC/PPO | | P (Transition) | Dynamics of the environment | Usually unknown, learned through interaction | | R (Reward) | Signal encoding the objective | Most critical design decision; see Reward Shaping below | | γ (Discount) | Future reward discount factor | 0.99 for long-horizon tasks; 0.95 for shorter episodes |

Algorithm Selection Guide

| Scenario | Algorithm | Key Advantage | |---|---|---| | Discrete actions, off-policy, sample efficient | DQN (+ PER + Dueling) | Replay buffer enables high sample efficiency | | Continuous control, off-policy | SAC (Soft Actor-Critic) | Maximum entropy RL; stable training, automatic α tuning | | Discrete or continuous, on-policy, stable | PPO | Clipped objective prevents destructive policy updates | | Partial observability, memory required | PPO + LSTM backbone | Recurrent policy handles non-Markovian observations | | Human preference alignment | RLHF (PPO + reward model) | Reward from human comparisons, not engineering |

Reward Shaping

The reward function is the most consequential RL design decision:

Potential-based shaping (safe — guarantees policy invariance):

• F(s, a, s') = γ·Φ(s') − Φ(s) where Φ is a potential function
• Does not change the optimal policy of the original MDP (Ng et al., 1999)

Common shaping components:

• Progress reward: proportional to movement toward the goal state
• Constraint penalty: large negative reward for safety violations
• Smoothness reward: penalize jerky or oscillatory actions (robotics)
• Curiosity bonus: intrinsic motivation for exploration in sparse-reward tasks

Red flags (reward hacking indicators):

• Reward that can be maximized by a degenerate policy (e.g., agent finds a bug in the simulator)
• Reward achievable without learning the intended behavior

See references/policy-gradient.md for PPO/SAC mathematical derivations and implementation details. See references/value-based-methods.md for DQN, Double DQN, Dueling DQN, and Rainbow implementations. See references/mdp-framework.md for environment design patterns and Gym interface specifications.

Training Stability Guidelines

Apply all of the following to any RL training run:

1. Gradient clipping (max_grad_norm = 0.5): prevents catastrophic policy updates
2. Observation normalization: running mean/variance normalization across episodes
3. Reward normalization: divide rewards by running standard deviation (not mean)
4. GAE (Generalized Advantage Estimation): λ = 0.95 for PPO — balances bias-variance in advantage estimates
5. Entropy regularization: encourages exploration; adjust coefficient during training
6. Multiple seeds: always report results across 5+ random seeds with mean ± std error

Evaluation Protocol

• Run 100 deterministic evaluation episodes (ε = 0 for DQN, deterministic policy for SAC/PPO).
• Report: mean reward, std, min/max, success rate (if applicable), episode length.
• Compare against baselines: random policy, hand-coded heuristic, previous best agent.
• Learning curve: episode reward vs. total environment steps (not wall-clock time).