NTC Slider Factory Documentation

DoRA is not the new LoRA!

Despite significant effort put into training and debugging DoRAs we have had to remove DoRA support from NTC Sliders.

What are LoRA and DoRA?

LoRAs are weight augmentations onto existing networks called parameter efficient fine tunes. Instead of changing all the weights of the network you can leave the original weights intact and just modify them using this technique. DoRA claims to improve parameter efficient fine tunes by separating magnitude and direction.

What happened?

The whitepaper was vague on details and every major tool and library implemented it differently. After months of work trying to fix this we had to give up.

In April, after much investigation the NTC team found issues in the ComfyUI implementation of DoRA. The team submitted a patch, but it was not merged. Instead the author rewrote it claiming to have fixed this but the bug remained.

In July we attempted to fix the problem (again) in ComfyUI but they pointed to an incorrect reference implementation. Note that the DoRA whitepaper does not include information on this.

We reached out to the implementer of the reference implementation but this was met with closed-minded insistence on a wrong formulation. The reference implementer proceeded to break the implementation of A1111. We offered to pay him money and even donated to him for his attention, yet the reference implementer refused to fix the issue.

As such, despite DoRAs in theory having better performance, in practice they are significantly broken by poor implementation standard.

Details

Essentially DoRA changes LoRA from:

w' = w + ba

Into:

w' = m * (w + ba) / norm(w + ba)

Future

In the future we may bring back DoRAs but as of right now the major open source implementations are too broken to consider. We will standardize on LoRAs going forward and existing DoRAs will be automatically converted then deleted. We are also increasing the rank of LoRA sliders which should make them comparable in quality.

References