Adjusting Learning Rate of a Neural Network in PyTorch

Adjusting Learning Rate of a Neural Network in PyTorch

In PyTorch, adjusting the learning rate of a neural network is commonly done using learning rate schedulers provided in the torch.optim.lr_scheduler module. A learning rate scheduler adjusts the learning rate based on the number of epochs or the current performance of the model.

Here's a brief overview of how to use a learning rate scheduler in PyTorch:

Step 1: Define your Model, Optimizer, and Initial Learning Rate

First, you'll want to define your model and optimizer as usual. Let's assume you're using the Adam optimizer:

import torch.nn as nn import torch.optim as optim model = nn.Sequential( nn.Linear(10, 50), nn.ReLU(), nn.Linear(50, 1) ) initial_lr = 0.001 optimizer = optim.Adam(model.parameters(), lr=initial_lr)

Step 2: Choose a Learning Rate Scheduler

PyTorch provides several learning rate schedulers. Some of the most common ones include:

  • StepLR: Decays the learning rate by gamma every step size epochs.
  • ExponentialLR: Decays the learning rate by gamma every epoch.
  • ReduceLROnPlateau: Reduces the learning rate when a metric has stopped improving.

Step 3: Implement the Learning Rate Scheduler

Here's how you can implement a few of these:

StepLR:

from torch.optim.lr_scheduler import StepLR # Decrease learning rate by 0.1 every 10 epochs scheduler = StepLR(optimizer, step_size=10, gamma=0.1)

ExponentialLR:

from torch.optim.lr_scheduler import ExponentialLR # Decrease learning rate by 0.1 every epoch scheduler = ExponentialLR(optimizer, gamma=0.1)

ReduceLROnPlateau:

This scheduler requires you to monitor a metric, and if the metric stops improving, the learning rate will be reduced.

from torch.optim.lr_scheduler import ReduceLROnPlateau # Reduce learning rate when 'val_loss' has stopped improving scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=5, verbose=True)

Step 4: Update the Learning Rate during Training

For schedulers like StepLR and ExponentialLR, call scheduler.step() at the end of each epoch:

for epoch in range(epochs): # Training loop... scheduler.step()

For ReduceLROnPlateau, you'll want to pass in the metric you're monitoring to scheduler.step(). Typically, this will be your validation loss:

for epoch in range(epochs): # Training loop... val_loss = compute_val_loss() scheduler.step(val_loss)

By integrating a learning rate scheduler into your training loop, you can efficiently adjust the learning rate during training to potentially achieve better model performance.

More Tags

jupyter-console serve paging decision-tree php-openssl uitextfielddelegate translate put ckeditor core-data

More Programming Guides

Other Guides

More Programming Examples