Skip to content

Optimizers¤

SGD¤

SGD ¤

SGD(params, lr=0.001, momentum=0.0, weight_decay=0.0)

Bases: Optimizer

Stochastic gradient descent with optional momentum and weight decay.

Source code in minitorch/optim.py 
21
22
23
24
def __init__(self, params, lr=1e-3, momentum=0.0, weight_decay=0.0):
    super().__init__(params, lr, weight_decay)
    self.momentum = momentum
    self.velocities = [np.zeros_like(p.data) for p in self.params]

Adam¤

Adam ¤

Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0)

Bases: Optimizer

Adam optimizer with bias-corrected first and second moment estimates.

Source code in minitorch/optim.py 
39
40
41
42
43
44
45
def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0.0):
    super().__init__(params, lr, weight_decay)
    self.beta1, self.beta2 = betas
    self.eps = eps
    self.t = 0
    self.m = [np.zeros_like(p.data) for p in self.params]
    self.v = [np.zeros_like(p.data) for p in self.params]

Gradient clipping¤

clip_grad_norm ¤

clip_grad_norm(params, max_norm)

Clip gradient norm across all parameters (like torch.nn.utils.clip_grad_norm_).

Source code in minitorch/optim.py 
62
63
64
65
66
67
68
69
70
71
72
73
74
def clip_grad_norm(params, max_norm):
    """Clip gradient norm across all parameters (like torch.nn.utils.clip_grad_norm_)."""
    total_norm_sq = 0.0
    for p in params:
        if p.grad is not None:
            total_norm_sq += float((p.grad ** 2).sum())
    total_norm = total_norm_sq ** 0.5
    if total_norm > max_norm:
        scale = max_norm / (total_norm + 1e-12)
        for p in params:
            if p.grad is not None:
                p.grad *= scale
    return total_norm

clip_grad_value ¤

clip_grad_value(params, clip_value)

Clip all gradients element-wise to [-clip_value, clip_value].

Source code in minitorch/optim.py 
77
78
79
80
81
def clip_grad_value(params, clip_value):
    """Clip all gradients element-wise to [-clip_value, clip_value]."""
    for p in params:
        if p.grad is not None:
            np.clip(p.grad, -clip_value, clip_value, out=p.grad)

Learning rate schedulers¤

StepLR ¤

StepLR(optimizer, step_size, gamma=0.1)

Multiply lr by gamma every step_size epochs.

Source code in minitorch/optim.py 
89
90
91
92
93
def __init__(self, optimizer, step_size, gamma=0.1):
    self.optimizer = optimizer
    self.step_size = step_size
    self.gamma = gamma
    self.epoch = 0

CosineAnnealingLR ¤

CosineAnnealingLR(optimizer, T_max, eta_min=0)

Cosine annealing from base lr down to eta_min over T_max epochs.

Source code in minitorch/optim.py 
103
104
105
106
107
108
def __init__(self, optimizer, T_max, eta_min=0):
    self.optimizer = optimizer
    self.T_max = T_max
    self.eta_min = eta_min
    self.base_lr = optimizer.lr
    self.epoch = 0