Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add DoRA (weight-decompose) support for LoRA/LoHa/LoKr #15160

Merged
merged 1 commit into from Mar 8, 2024

Conversation

KohakuBlueleaf
Copy link
Collaborator

Description

Implementation for inference with DoRA
The key name is based on the implementation in LyCORIS.

And since weight-decompose is a general idea on top of all low-rank method. I implement it in the NetworkModule instead of each algo's module.

And here is a quick sanity check:
image

Checklist:

@AUTOMATIC1111 AUTOMATIC1111 merged commit 9409419 into dev Mar 8, 2024
6 checks passed
@pessimal
Copy link

pessimal commented Mar 8, 2024

@KohakuBlueleaf I don't know if it's for every Lora but with this PR I've constantly been getting warnings/errors like this.

loading network /srv/linuxdata/stable-diffusion-webui_dev/models/Lora/SDXL_Lora/add-detail-xl.safetensors: AttributeError
Traceback (most recent call last):
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/networks.py", line 280, in load_networks
net = load_network(name, network_on_disk)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/networks.py", line 219, in load_network
net_module = nettype.create_module(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network_lora.py", line 11, in create_module
return NetworkModuleLora(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network_lora.py", line 18, in init
super().init(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network.py", line 150, in init
self.dora_mean_dim = tuple(i for i in range(len(self.shape)) if i != 1)
AttributeError: 'NetworkModuleLora' object has no attribute 'shape'

@KohakuBlueleaf
Copy link
Collaborator Author

@KohakuBlueleaf I don't know if it's for every Lora but with this PR I've constantly been getting warnings/errors like this. Other than that I feel Loras now have much more effect on the result of the generation

loading network /srv/linuxdata/stable-diffusion-webui_dev/models/Lora/SDXL_Lora/add-detail-xl.safetensors: AttributeError
Traceback (most recent call last):
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/networks.py", line 280, in load_networks
net = load_network(name, network_on_disk)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/networks.py", line 219, in load_network
net_module = nettype.create_module(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network_lora.py", line 11, in create_module
return NetworkModuleLora(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network_lora.py", line 18, in init
super().init(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network.py", line 150, in init
self.dora_mean_dim = tuple(i for i in range(len(self.shape)) if i != 1)
AttributeError: 'NetworkModuleLora' object has no attribute 'shape'

Thx for this info
I will fix it later

@clayne
Copy link
Contributor

clayne commented Mar 8, 2024

Also seeing it:

Using already loaded model sdxl\sd_xl_base_1.0_0.9vae.safetensors [e6bb9ea85b]: done in 2.8s (send model to cpu: 1.8s, send model to device: 1.
0s)
loading network F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\models\Lora\sdxl\Syd Mead Style.safetensors: AttributeError
Traceback (most recent call last):
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\networks.py", line 280, in load_networks
    net = load_network(name, network_on_disk)
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\networks.py", line 219, in load_network
    net_module = nettype.create_module(net, weights)
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\network_lora.py", line 11, in create_module
    return NetworkModuleLora(net, weights)
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\network_lora.py", line 18, in __init__
    super().__init__(net, weights)
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\network.py", line 150, in __init__
    self.dora_mean_dim = tuple(i for i in range(len(self.shape)) if i != 1)
AttributeError: 'NetworkModuleLora' object has no attribute 'shape'. Did you mean: 'scale'?

@pessimal
Copy link

pessimal commented Mar 8, 2024

Until now I assumed this was more a warning but any Lora that shows this error is just discarded and not used during generation. (doesn't get listed under "Lora hashes:" in the generation output)

@SlZeroth
Copy link

someone tested dora in this code?

I got this error

merged_scale1 / merged_scale1(dim=self.dora_mean_dim, keepdim=True) * self.dora_scale

TypeError: 'Tensor' object is not callable

@SlZeroth
Copy link

merged_scale1 = updown + orig_weight

merged_scale1 type is a tensor that is not callable

@KohakuBlueleaf
Copy link
Collaborator Author

merged_scale1 = updown + orig_weight

merged_scale1 type is a tensor that is not callable

I said
I will fix it later

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants