StepLR

If you have ever worked on a Computer Vision project, you might know that using a learning rate scheduler might significantly increase your model training performance. On this page, we will:

  • Сover the Step Learning Rate (StepLR) scheduler;
  • Check out its parameters;
  • See a potential effect from StepLR on a learning curve;
  • And check out how to work with StepLR using Python and the PyTorch framework.

Let’s jump in.

StepLR is a scheduling technique that decays the learning rate by gamma every N epochs (or every N evaluation periods, if iteration training is used).

Compared to ExponentialLR, which divides the learning rate every epoch, the StepLR keeps the learning rate the same over N epochs before reducing it.
  • Step Size - the decay of the learning rate happens every N epochs. This "N" is the step size.
  • Gamma - a multiplicative factor by which the learning rate is decayed. For instance, if the learning rate is 1000 and gamma is 0.5, the new learning rate will be 1000 x 0.5 = 500.
The gamma value should be less than 1 to reduce the learning rate.
Learning rate over epochs, StepLR.
Source
The comparison of StepLR with other schedulers.
Source
python 
      import torch
model = [Parameter(torch.randn(2, 2, requires_grad=True))]
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=0.01, amsgrad=False)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1, last_epoch=-1, verbose=False)
for epoch in range(20):
    for input, target in dataset:
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()
    scheduler.step()

Boost model performance quickly with AI-powered labeling and 100% QA.

Learn more
Last modified