Reinforcement Learning Control Lab
Train a DDPG or TD3 policy and compare it with PID control on the same process.
Controller comparison
Root mean square error (RMSE)
Total input variation (TV) (pp)
Process variable
Controller output
Output limits · 0–100%Training history
Episode return
10-episode moving averageValidation IAE
Lab exercises
- Select Start simulation to watch both controllers respond. At 1×, a 48 s run takes 48 seconds. Full preview shows the entire scenario immediately.
- Try Disturbance rejection to see the scheduled disturbance at 19 s. During a live run, enter a setpoint and disturbance and select Apply inputs to apply them to both controllers at the next sample. Follow scenario returns to the schedule at the current time. Reset run clears manual inputs.
- Compare IAE and total input variation. Then train DDPG and TD3 with the same seed and number of episodes to see how the algorithms and the control move penalty λ affect the trade-off. The same seed gives both algorithms the same training scenarios.
State, reward, and evaluation
Reinforcement learning
The policy takes the normalized error, setpoint, process variable, accumulated error, change in the process variable, and previous control action. It outputs a continuous control action u ∈ [−1, 1].
Reward = −[(next process value − current setpoint)² + λ × (u − previous u)²]. Training varies the initial state, setpoints, and disturbances across episodes and adds exploration noise.
DDPG uses an actor and one critic. TD3 uses the smaller of two target critic estimates, updates the actor less frequently, and adds clipped noise to the target action.
PID comparison
Each controller starts from the same process state and receives the same setpoint and input disturbance. Both use a 0.25 s sampling interval and 0–100% output limits over 48 simulated seconds. PID uses normalized deviations, conditional integration for anti-windup, and a filtered derivative of the measurement.
Validation IAE is the integral of normalized absolute error, averaged over four fixed scenarios. The checkpoint with the lowest validation IAE is used for comparison. Evaluation runs use zero exploration noise.
Metrics cover the elapsed portion of the run. IAE integrates absolute tracking error, RMSE measures root mean square tracking error, and TV sums absolute output changes from the initial 50% command. Input disturbances and TV are expressed in percentage points (pp).