Size from network weights

[AI-Casanova]

This PR add the --size_from_weights flag to override --dim and blockdims and directly loads the previously trained LoRA no matter the shape.

This allows finetuning a LoRA that has been resized with resize_lora.py (see #243) or with an unknown blockdims shape.

My Current Workflow:

• Train LoRA with dim=128
• Use sd-webui-lora-block-weight and sd-webui-supermerger to tweak block weights
• Use resize_lora.py with --dynamic_method="sv_ratio" --dynamic_param=(between 10 and 20) to greatly reduce size and untrained noise.
• Use this PR with lr cut in half to retrain the smaller LoRA, this seems to do a great job in cleaning up small details like eyes.

There may be a better way to handle the interpretation of block lrs. since there isn't a separate function in lora.py to call.

[kohya-ss]

Thank you for this. I think this feature is quite useful!

It seems to me that the parsing net_kwargs would be better to be common with lora.py. I will check and merge as soon as I get time.

[AI-Casanova]

Agreed on the parsing function.

Cheers!

[kohya-ss]

I'd like to rename the option to --dim_from_weights. I think it will be more intuitive. I appreciate your understanding.

[AI-Casanova]

You are always welcome to make any and all changes to my PRs. I find things that are useful to me and share them, but I am in no means a great programmer.

[xdfeng07]

Hi, can you give some examples or detailed description on how to use sd-webui-lora-block-weight and sd-webui-supermerger to tweak block weights? Thanks very much!