Fix the OFT/BOFT bugs when using new LyCORIS implementation #14973
Conversation
|
|
||
| def calc_updown(self, orig_weight): | ||
| oft_blocks = self.oft_blocks.to(orig_weight.device) | ||
| eye = torch.eye(self.block_size, device=oft_blocks.device) | ||
|
|
||
| if self.is_kohya: | ||
| if not self.is_R: | ||
| block_Q = oft_blocks - oft_blocks.transpose(1, 2) # ensure skew-symmetric orthogonal matrix |
There was a problem hiding this comment.
@KohakuBlueleaf I used 2.1.0.post3 to train a new boft and diag-oft
boft got an error during inference
There was a problem hiding this comment.
@KohakuBlueleaf I used 2.1.0.post3 to train a new boft and diag-oft boft got an error during inference
Got it
There was a problem hiding this comment.
@KohakuBlueleaf I used 2.1.0.post3 to train a new boft and diag-oft boft got an error during inference
Try new Commit
There was a problem hiding this comment.
Thanks , boft looks like work,
And I found that diag-oft needs to specify "constrain" in training, otherwise it doesn’t take effect.
Is it a bug?
There was a problem hiding this comment.
Thanks , boft looks like work, And I found that diag-oft needs to specify "constrain" in training, otherwise it doesn’t take effect. Is it a bug?
It is bug
There was a problem hiding this comment.
@Exist-c fixed
There was a problem hiding this comment.
Noticed the same thing with my diag-oft so went to find if anything has been updated, but after pulling the changes there is an error about 'MultiHeadAttention' object has no attribute 'weight'
send help pls 
edit just don't train TE, works fine
Fix the OFT/BOFT bugs when using new LyCORIS implementation
Checklist: