Learning-Rate Scheduling for Neural Network Training: A Scoping Review of ReduceLROnPlateau, Cosine Annealing with Warm Restarts, Cyclical Learning Rates, and Linear Warmup with Linear Decay

Abstract

This scoping review presents a structured synthesis of learning-rate scheduling for neural network training. The learning rate is a consequential hyperparameter in gradient based neural network training because it mediates the trade-off between early exploration and late-stage convergence precision. The review focuses on four widely used learning-rate scheduling paradigms: (1) ReduceLROnPlateau (RLROP), a feedback-driven reactive scheduler; (2) Stochastic Gradient Descent with Warm Restarts (SGDR), a periodic cosine-shaped schedule; (3) Cyclical Learning Rates (CLR), an oscillatory triangular or exponential schedule; and (4) Linear Warmup with Linear Decay (LW+LD), a two-phase schedule widely used with transformer models. We use a common notation as a pedagogical organizing device, distinguish formal results from empirical tendencies and heuristic analogies, and summarize how each method relates to classical step-size conditions and adaptive-optimizer stability. The quantitative tables are explicitly treated as heterogeneous literature summaries rather than controlled head-to-head benchmarks; they are retained only to document source-specific evidence and practical signals. The article includes a scoping-search log, a quality-assessment rubric, caveats on benchmark comparability, hyperparameter-sensitivity diagnostics, failure-mode checks, and an AI tool usage and human contribution statement. The resulting scheduler-selection framework is intended as practical guidance, not as a statistically ranked performance claim.