學習率迭代策略

2021-10-16 16:56:54 字數 2831 閱讀 4608

base_lr =

0.01

lr_policy =

"fixed"

每迭代step_size次後減少gamma倍。lr=

lr×g

amma

lr=lr×gamma

lr=lr×

gamm

a

base_lr =

0.01

lr_policy =

"step"

gamma =

0.1stepsize=

10000

lr(

t)=b

ase_

lr×(

tt)p

ower

lr(t)=base\_lr\times(\frac)^

lr(t)=

base

_lr×

(tt​

)pow

er

base_lr =

0.01

lr_policy =

"step"

gamma =

0.1stepsize=

10000

learning rate隨迭代次數增加而下降。lr(

t)=b

ase_

lr×(

1+ga

mma×

iter

)pow

er

lr(t)=base\_lr\times(1+gamma\times iter)^

lr(t)=

base

_lr×

(1+g

amma

×ite

r)po

wer

nit_lr =5e-

4max_iter =

15000

optimizer = sgd(params=net.parameters(

), lr=init_lr, momentum=

0.9, weight_decay=

0.0005

)lambda_func =

lambda step:(1

-step/max_iter)

**0.9

scheduler = torch.optim.lr_scheduler.lambdalr(optimizer, lr_lambda=lambda_func)

for epoch in

range(50

):for i in

range

(1000):

... optimizer.step(

) scheduler.step(

)

注意這裡使用lambda函式時只返回init_lr需要乘的係數即可。

需要繼承torch.optim.lr_scheduler._lrscheduler類:

from torch.optim.lr_scheduler import _lrscheduler

class

polylrdecay

(_lrscheduler)

:def

__init__

(self, optimizer, max_decay_steps, end_learning_rate=

0.0001

, power=

1)

需要實現兩個方法:

def

get_lr

(self)

:def

step

(self, step=

none

):

例子見: poly_lr_decay

l r=

lr×(

1−it

erma

x_it

er)p

ower

lr=lr\times (1-\frac)^

lr=lr×

(1−m

ax_i

teri

ter​

)pow

er

init_lr =5e-

4max_iter =

15000

optimizer = sgd(params=net.parameters(

), lr=init_lr, momentum=

0.9, weight_decay=

0.0005

)lambda_func =

lambda step:(1

-step/max_iter)

**0.9

scheduler = torch.optim.lr_scheduler.lambdalr(optimizer, lr_lambda=lambda_func)

lrs =

for epoch in

range(15

):for i in

range

(1000):

... optimizer.step()0

]["lr"])

scheduler.step(

)plt.plot(

range

(15000

), lrs)

plt.show(

)

torch.optim.lr_scheduler模組中都有相關定義。常見的學習率變化函式及其曲線見: kaggle_pytorch_lr_scheduler

pytorch框架學習(12) 學習率調整策略

主要方法 1.steplr 調整方式 lr lr gamma 2.multisteplr 調整方式 lr lr gamma milestones 50 125 160 scheduler lr optim.lr scheduler.multisteplr optimizer,milestones,g...

pytorch使用 (十二)學習率調整策略

梯度下降 lr 學習率 learning rate 控制更新的步伐 class lrscheduler object def init self,optimizer,last epoch 1 def get lr self raise notimplementederror 調整方式 lr lr g...

tensorflow中學習率的調參策略

learning rate decay type 調參策略 1 exponential decay learning rate,global step,decay steps,decay rate,staircase false,name none learning rate 初始值 global ...