Description

• This makes the default behavior of refiner switchover aligned with model timesteps instead of sampling timesteps. This is easier to use, since setting refiner switchover to 0.8 (for a refiner trained on the last 200 timesteps like SDXL's) will now always switch to the refiner as soon as we're sampling from timesteps the refiner was trained on, where previously using img2img or different noise schedules could lead to unexpected model behavior (examples below).
• I changed this to work off of the existing refiner switchover slider. The new default behavior will be to derive timesteps from sigma (or in DDIM's case take them directly) and use that to determine whether it is time to switch over. Old behavior is supported by a compatibility option.
• Implements [Feature Request]: Refiner switchover should be controlled by (fraction of) training timesteps and not by fraction of sampling #14970
• edit: converted to draft while I troubleshoot an off-by-one error There is a bug where model alphas_cumprod changes (compatibility casting option or zero terminal snr) are reverted during the timestep the refiner is applied. This will have to be fixed separately.

Screenshots/videos:

Examples of old behavior in txt2img:

The refiner model used here was trained for the last 200 timesteps. The Karras schedule type, especially on zero snr, drastically changes the model timesteps called during this 50 step sampling process, which results in the refiner being switched to too early on what is actually the correct setting on the default noise schedule. The effectively correct setting for Karras samplers is 0.88 for this refiner under the old configuration.

Now for the fixed version:

With this fix, the behavior of the refiner is consistent with the same settings across different schedules, and it no longer triggers too early. 0.8 is reliably a correct setting.

Examples of old behavior in img2img/inpainting (inpainting mask is over the head, adding a hat, 0.75 denoising strength):

This one is more complicated, and the differences are subtle. The effectively correct settings for the normal schedule is to switch over at 0.75, and for Karras it is correct to switch over at 0.85. Using the expected setting of 0.8 therefore is too late for normal schedules and too early for Karras ones. As denoising strength gets lower, the problem becomes more severe.

This grid shows the behavior after the fix. Switch at 0.8 is now correct for both.

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests