I needed a reliable way to modify RL algorithms and iterate on ideas fast. Existing options are either single-file scripts that don't scale to many experiments (CleanRL) or big frameworks too rigid to change (SB3).

A middle ground: every agent is defined by a config and four factory functions (state, explore, process experience, update). I reimplemented PPO, SAC and others from scratch on this pattern and validated them against reference benchmarks, then grew the library around my own needs — GPU-vectorized environments and self-play — and used it for all the experiments in my master's thesis.