Offline RL for Business: Train Safe AI Agents from Logs with CQL and d3rlpy

Offline RL for Business: How CQL and d3rlpy Let You Train Safe AI Agents from Logs

Train reliable, low-risk reinforcement learning policies from historical logs using Conservative Q-Learning (CQL) and d3rlpy — no dangerous online exploration required.

Executive summary

TL;DR: When live exploration risks physical harm, financial loss, or regulatory trouble, collect conservative behavior logs and train offline using Conservative Q-Learning (CQL). Using d3rlpy and a reproducible GridWorld example, CQL reduced risky, out-of-distribution behavior compared with simple imitation (Behavior Cloning). For leaders: invest in logging, diagnostics, and staged rollouts rather than risky exploratory pilots.

“We build a safety-critical RL pipeline that learns entirely from fixed offline data rather than live exploration.”

Why offline RL for safety-critical AI agents?

Some domains cannot tolerate probing unknown actions. Imagine a warehouse robot that could crash expensive equipment, a clinical assistant making untested treatment suggestions, or an automated trading strategy that accidentally shorts a key asset. Those are not places for trial-and-error learning.

Offline reinforcement learning (offline RL) trains from fixed historical datasets — logs of conservative, human, or safe-controller behavior — so you can learn policy improvements without letting the agent experiment on live systems. The big challenge is distributional shift: naive algorithms can be overly optimistic about actions not present in the logs and propose dangerous policies. Conservative Q-Learning (CQL) is an algorithm designed to counteract that optimism.

Vignette: a 15×15 hazardous GridWorld

To test ideas quickly and reproducibly, a safety-focused GridWorld was used: a 15×15 grid with hidden interior hazards (stepping on one gives -100), a goal (+50), and a step penalty (-1) to encourage efficient paths. Transitions are noisy (5% slip probability), so the environment is stochastic. Random exploration can hit hazards easily — a helpful toy analogy for real-world risk.

Dataset: how the logs are generated

Generate logs with a risk-averse behavior policy: an epsilon-style strategy that mostly avoids hazards while injecting limited randomness. Example epsilons in experiments were ~0.15–0.22 — enough randomness for coverage but not reckless exploration. Episodes are made reproducible with fixed random seeds (e.g., SEED=42). Typical dataset sizes used for the example were 400–500 episodes; larger, high-dimensional domains will need more coverage.

Why those choices matter:

  • Lower epsilon → safer but narrower data (poor coverage).
  • Higher epsilon → more coverage but higher chance of dangerous actions appearing in logs.
  • Reproducible seeds and consistent environment configs are essential for fair comparisons and debugging.

Algorithms compared: Behavior Cloning (BC) vs Conservative Q-Learning (CQL)

Two learners illustrate common choices for offline RL:

  • Behavior Cloning (BC) — supervised imitation of logged actions. Fast and simple, but it will mimic the behavior policy and lacks a mechanism to penalize optimism in unseen states.
  • Conservative Q-Learning (CQL) — an offline RL algorithm that adds a penalty term to discourage high Q-values for actions not supported by the dataset. This reduces the policy’s tendency to select unsupported (and potentially dangerous) actions.

Practical hyperparameters used in experiments:

  • BC: learning_rate ≈ 3e-4, batch_size = 256, training ~25k steps.
  • CQL: conservative_weight = 6.0, n_action_samples = 10, batch_size = 256, training ~80k steps.

Device detection (CUDA if available) and fixed seeds were used to make runs stable and comparable.

Diagnostics and evaluation you should run

Before trusting a policy trained offline, run diagnostics to detect blind spots and risky behavior.

Basic dataset checks

  • State-visitation heatmap — where the dataset has coverage and where it doesn’t.
  • Reward distribution histogram — check for bias or rare high/low outcomes dominating learning.
  • Policy vs dataset action-mismatch (OOD diagnostic) — sample several thousand observations and compare actions chosen by your policy to actions in the log.

Controlled rollout evaluation

Run controlled online rollouts with safety metrics measured:

  • Mean return and standard deviation
  • Mean episode length
  • Hazard rate — fraction of episodes that hit a hazard
  • Goal rate — fraction that reach the goal

Recommended protocol: run ~30 episodes across 3 seeds (10 episodes per seed) for preliminary assessment; for publication-grade claims, use more seeds and bootstrapped confidence intervals. Use paired comparisons or bootstrapping to compare BC vs CQL reliably.

Action-mismatch / OOD diagnostic

Compute the fraction of sampled observations where the policy action differs from the logged action. Heuristics:

  • >20% mismatch → red flag: policy is frequently out-of-distribution and likely to propose unsupported actions.
  • 10–20% mismatch → caution: acceptable in some domains if rollouts show low hazard rates.
  • <10% mismatch → better confidence, but still run rollouts — low mismatch does not guarantee safety.

What the experiment showed

Empirically in the GridWorld example:

  • Behavior Cloning performed decently in-sample but displayed higher action-mismatch and a higher hazard rate during controlled rollouts compared with CQL.
  • CQL reduced selection of unsupported actions by penalizing over-optimistic Q-values, which translated into fewer hazard hits in rollout evaluation.
  • CQL can be conservative: when conservative_weight is too high, the agent avoids useful actions and performance (mean return, goal_rate) can drop. Tuning is required to balance safety and effectiveness.

“We demonstrated that Conservative Q-Learning yields a more reliable policy than simple imitation when learning from historical data in safety-sensitive environments.”

Exact numbers and plots (heatmaps, reward histograms, hazard/goal bar charts, learning curves) are available in the project notebook linked in the References; the qualitative pattern — CQL trades some optimism for measurable safety gains — is robust across seeds and dataset sizes in this toy domain.

Practical guidance: hyperparameters, dataset size, and tuning

  • conservative_weight: start around 3–6; lower values may under-penalize optimism, higher values can make the policy overly cautious.
  • n_action_samples: 5–20 is typical for discrete action spaces; more samples increase stability at the cost of compute.
  • Dataset size: toy GridWorlds work with hundreds of episodes; real-world, high-dimensional domains often require thousands to millions of logged transitions.
  • Coverage matters: add targeted logging policies or data-collection drives to cover risky states safely (human-in-the-loop or safer controllers).

Deployment guardrails

Offline training reduces risk but does not eliminate it. Use staged rollout patterns:

  • Shadow mode: run the policy in parallel to the production controller and log divergences.
  • Canary release: small, monitored deployments with human oversight.
  • Human-in-the-loop gating: require human approval for actions in uncertain states.
  • Live monitoring and automatic rollback thresholds (e.g., hazard events per hour).

Quickstart

High-level steps to reproduce this pipeline (example commands):

  1. Install dependencies: pip install d3rlpy gymnasium torch numpy matplotlib scikit-learn
  2. Generate conservative logs: python generate_dataset.py –env SafetyCriticalGridWorld –episodes 500 –seed 42
  3. Train CQL: python train_cql.py –data data.npz –algo CQL –steps 80000 –conservative_weight 6.0
  4. Run controlled rollouts: python evaluate_policy.py –policy grid_cql_policy.pt –episodes 30

Complete code, notebooks, and exact commands are linked in the References. Use a requirements.txt or Dockerfile to lock versions; d3rlpy has breaking API changes between releases, so pin the version used in experiments.

Checklist for engineering and product teams

  • Collect conservative logs (human, safe controller) with consistent seeds and metadata.
  • Run quick dataset diagnostics: heatmaps, reward histograms, action-mismatch.
  • Train and compare BC and CQL baselines; keep hyperparameters and seeds fixed for fair comparison.
  • Evaluate with controlled rollouts and quantify hazard_rate, goal_rate, and return distribution.
  • Stage deploy: shadow → canary → gated production with rollback rules.
  • Store experiment config (seeds, hyperparameters, env) in a single JSON/YAML for reproducibility.

For the C-suite

  • Risk mitigation: Prioritize logging safe behavior — it’s often cheaper than building a safe trial environment.
  • Investment priorities: Data engineering (structured logs), monitoring, and tooling around offline RL (d3rlpy, evaluation pipelines).
  • Time-to-value: A proof-of-concept in a constrained domain can take 2–6 weeks; enterprise-scale pilots will require more data and governance steps.
  • ROI framing: The costs of thorough logging and conservative offline training are typically far lower than the potential cost of unsafe exploration.

Counterpoints and trade-offs

CQL improves safety by being conservative, but that conservatism can sacrifice performance if applied blindly. In some cases, a hybrid approach — constrained online fine-tuning with strong monitoring or conservative ensembles — can produce better long-term returns. Also, offline RL cannot correct systematic biases in the logs: if the dataset consistently omits a safe strategy, no algorithm will invent it without additional data or controlled exploration.

Glossary

  • Offline RL: training reinforcement learning policies from fixed historical data (logs) instead of live interaction.
  • CQL (Conservative Q-Learning): an offline RL algorithm that penalizes overestimated Q-values for actions not supported by the data.
  • BC (Behavior Cloning): supervised learning to imitate logged actions.
  • OOD: out-of-distribution — states or actions not well covered by the dataset.
  • d3rlpy: Python library implementing offline RL algorithms (useful for prototyping CQL, BC, and others).

References and reproducibility

  • Conservative Q-Learning (CQL) — Aviral Kumar et al., arXiv 2020.
  • d3rlpy — offline RL library used for experiments.
  • Gymnasium — environment API used for the GridWorld.
  • Code and reproducible notebooks with plots and exact metrics: see the project repository linked from the tutorial’s source (notebook contains heatmaps, reward histograms, learning curves and full evaluation results).

“By structuring the workflow around offline datasets, careful evaluation, and conservative learning objectives, robust decision-making policies can be trained where unsafe exploration is not an option.”

Next steps

Start by instrumenting safe logging in a limited pilot area, run diagnostics to check coverage, and experiment with CQL on that dataset. Combine offline training with shadow-mode testing and human-in-the-loop gating before any live control handover. Offline RL is not a magic bullet, but when paired with conservative algorithms and disciplined deployment, it’s a practical path to safer AI agents.

0