← ECML Hub

Reinforcement Learning Control Lab

Train a DDPG or TD3 policy and compare it with PID control on the same process.

Controller comparison

Ready 0.00 / 48 s

Integral absolute error (IAE)

DDPGPID

Root mean square error (RMSE)

DDPGPID

Total input variation (TV) (pp)

DDPGPID

Process variable

Setpoint DDPG PID
Setpoint and process responses for both controllers.

Controller output

Output limits · 0–100%
Control output over time for both controllers.
Hover over a chart or use the arrow keys to inspect samples.
Lab exercises
  1. Select Start simulation to watch both controllers respond. At 1×, a 48 s run takes 48 seconds. Full preview shows the entire scenario immediately.
  2. Try Disturbance rejection to see the scheduled disturbance at 19 s. During a live run, enter a setpoint and disturbance and select Apply inputs to apply them to both controllers at the next sample. Follow scenario returns to the schedule at the current time. Reset run clears manual inputs.
  3. Compare IAE and total input variation. Then train DDPG and TD3 with the same seed and number of episodes to see how the algorithms and the control move penalty λ affect the trade-off. The same seed gives both algorithms the same training scenarios.

State, reward, and evaluation

Reinforcement learning

The policy takes the normalized error, setpoint, process variable, accumulated error, change in the process variable, and previous control action. It outputs a continuous control action u ∈ [−1, 1].

Reward = −[(next process value − current setpoint)² + λ × (u − previous u)²]. Training varies the initial state, setpoints, and disturbances across episodes and adds exploration noise.

DDPG uses an actor and one critic. TD3 uses the smaller of two target critic estimates, updates the actor less frequently, and adds clipped noise to the target action.

Original papers DDPG · TD3

PID comparison

Each controller starts from the same process state and receives the same setpoint and input disturbance. Both use a 0.25 s sampling interval and 0–100% output limits over 48 simulated seconds. PID uses normalized deviations, conditional integration for anti-windup, and a filtered derivative of the measurement.

Validation IAE is the integral of normalized absolute error, averaged over four fixed scenarios. The checkpoint with the lowest validation IAE is used for comparison. Evaluation runs use zero exploration noise.

Metrics cover the elapsed portion of the run. IAE integrates absolute tracking error, RMSE measures root mean square tracking error, and TV sums absolute output changes from the initial 50% command. Input disturbances and TV are expressed in percentage points (pp).