Lion 8 bit - #188

Merged

TimDettmers merged 21 commits into

bitsandbytes-foundation:mainfrom

lucidrains:main

Apr 11, 2023

Commits on Mar 9, 2023

initial commit, slowly work from interface into the kernel
lucidrains
committed
7247cb4
make sure interface is correct
lucidrains
committed
d43ea97
do a bunch of typical bookkeeping before getting to main lion logic
lucidrains
committed
cb4c3c8
forget about tests for now, will test live on local enwik8 training
lucidrains
committed
8de29fc
add a sign function, for lion
lucidrains
committed
64bb1ae
use epsilon as beta2 for lion, complete most of the logic in kernel.cu for all functions
lucidrains
committed
c83888a
remove something rmsprop specific
lucidrains
committed
ead570a
fix weight decay for lion to be decoupled, using a switch
lucidrains
committed
af03430
missed adagrad
lucidrains
committed
c558272

Commits on Mar 10, 2023

Commits on Mar 22, 2023

follow advice of Tim to fix update of momentum vs parameters in blockwise 8 bit
lucidrains
committed
9b656f4
add some code in test_optim.py, although it seems to be failing
lucidrains
committed
a43cd20
add some comments, and fix use of g_val
lucidrains
committed
aa9b939
fix consistent tabs / spaces
lucidrains
committed
916000c
another tab/spaces fix
lucidrains
committed
978ba2d
fix comment
lucidrains
committed
2a6828e