LoRA :: definition and meaning

Imagine you're playing with a friend who loves drawing pictures. He's good at drawing all sorts of things, but they might not know everything.
A LoRA is like a special notebook you can give your friend to help them draw specific things better.

• The notebook: The LoRA is filled with examples of what you want your friend to draw well. For example, a LoRA trained on cats would have lots of pictures of cats in different poses and with different features.
• Helping your friend: When you give your friend the notebook, they can use it to learn how to draw cats better. They can look at the pictures for inspiration and see how different parts of a cat fit together.
  This helps them draw cats that look more realistic and accurate.
• Still using their skills: Even though your friend has the notebook, they still use their own drawing skills. They won't just copy the pictures directly, but they'll use the information in the notebook to improve their overall drawing of a cat.

So, a LoRA in Stable Diffusion doesn't replace the model's knowledge, it just helps it focus on specific details and improve the final image in that way.

A quick reminder before: a checkpoint is a file containing the weights and biases (parameters) of a trained neural network within the broader Stable Diffusion architecture.
These weights embody the model's learned ability to translate text prompts into images.
Checkpoints are often large files (several gigabytes) due to the complexity of the image generation task and the model's structure.

LoRA is a training technique that efficiently modifies a small subset of the weights in a pre-trained Stable Diffusion checkpoint.
These changes don't replace the entire checkpoint but instead adapt it towards a new style or subject.
LoRA primarily focuses on modifying the weights within linear layers of the neural network. This allows for focused adjustments without requiring massive retraining.
You can apply multiple LoRAs to a single checkpoint to introduce a combination of different styles or specializations.

A checkpoint utilizes a U-Net architecture with convolutional and attention layers, responsible for processing text prompts and generating images.
Within these layers, the model's knowledge resides as weights and biases which are numerical values that determine the connections and activation strengths between neurons, essentially defining the model's behavior.

LoRA utilizes a technique called spectral filtering to identify a low-rank subspace within the original checkpoint's weight space.
This subspace captures the essential information needed to represent the target concept (e.g., an art style, specific character).
LoRA primarily concentrates on modifying the weights within the linear layers of the network.
These layers, at the end of the network, directly map the internal representation to the final image output.
By adjusting these weights through the identified low-rank subspace, LoRA efficiently steers the model towards generating images consistent with the desired concept.

LoRA employs eigenvalue decomposition to analyze the weight matrix of the chosen linear layer from the checkpoint.
This decomposition identifies the eigenvectors and eigenvalues that capture the most significant variations in the data, forming the aforementioned low-rank subspace.
LoRA constructs a projection matrix based on the identified subspace. This matrix essentially represents the "direction" of change needed to achieve the desired adaptation.
The projection matrix is then applied to a small, pre-defined "weight matrix of adaptation" to obtain the final weight update.
This update is then added to the original weight matrix of the chosen linear layer, effectively adapting it towards the target concept.

Targeted modifications result in significantly smaller files compared to full checkpoint retraining, making them easier to store and share.
Multiple LoRAs can be combined with a single checkpoint, allowing for the creation of unique artistic blends and specializations.
LoRA offers a faster and more efficient alternative to full model retraining for specific adaptations, but the level of control and potential for introducing new information is limited compared to comprehensive retraining.