mattertune.configs.recipes

class mattertune.configs.recipes.EMARecipeConfig(*, name='ema', decay, validate_original_weights=False, every_n_steps=1, cpu_offload=False)[source]
Parameters:

  • name (Literal['ema'])

  • decay (Annotated[float, Gt(gt=0)])

  • validate_original_weights (bool)

  • every_n_steps (int)

  • cpu_offload (bool)

name: Literal['ema']
decay: C.PositiveFloat

The exponential decay used when calculating the moving average. Has to be between 0-1.

validate_original_weights: bool

Validate the original weights, as apposed to the EMA weights.

every_n_steps: int

Apply EMA every N steps.

cpu_offload: bool

Offload weights to CPU.

create_lightning_callback()[source]

Creates the PyTorch Lightning callback for this recipe, or returns None if no callback is needed.

class mattertune.configs.recipes.LoRARecipeConfig(*, name='lora', lora)[source]

Recipe for applying Low-Rank Adaptation (LoRA) to a model. LoRA is a method for fine-tuning pre-trained models via the injection of low-rank “adapter” weights into the model’s linear layers. This allows for efficient fine-tuning of large models on small datasets, while preserving the pre-trained weights in the backbone.

Reference: https://arxiv.org/abs/2106.09685

Parameters:
name: Literal['lora']

Discriminator for the LoRA recipe.

lora: LoraConfig

LoRA configuration.

classmethod ensure_dependencies()[source]

Ensure that all dependencies are installed.

This method should raise an exception if any dependencies are missing, with a message indicating which dependencies are missing and how to install them.

create_lightning_callback()[source]

Creates the PyTorch Lightning callback for this recipe, or returns None if no callback is needed.

class mattertune.configs.recipes.LoraConfig(*, peft_type=None, task_type=None, inference_mode=False, r=8, target_modules=None, lora_alpha=8, lora_dropout=0.0, fan_in_fan_out=False, bias='none', use_rslora=False, modules_to_save=None, init_lora_weights=True, layers_to_transform=None, layers_pattern=None, rank_pattern={}, alpha_pattern={})[source]
Parameters:

  • peft_type (str | None)

  • task_type (str | None)

  • inference_mode (bool)

  • r (int)

  • target_modules (list[str] | str | None)

  • lora_alpha (int)

  • lora_dropout (float)

  • fan_in_fan_out (bool)

  • bias (Literal['none', 'all', 'lora_only'])

  • use_rslora (bool)

  • modules_to_save (list[str] | None)

  • init_lora_weights (bool | Literal['gaussian'])

  • layers_to_transform (list[int] | int | None)

  • layers_pattern (list[str] | str | None)

  • rank_pattern (dict[str, Any])

  • alpha_pattern (dict[str, Any])

r: int

LoRA attention dimension (rank).

target_modules: list[str] | str | None

Names of modules to apply LoRA to. Can be a list of module names, a regex pattern, or ‘all-linear’.

lora_alpha: int

Alpha parameter for LoRA scaling.

lora_dropout: float

Dropout probability for LoRA layers.

fan_in_fan_out: bool

Set True if target layer stores weights as (fan_in, fan_out).

bias: Literal['none', 'all', 'lora_only']

Bias type for LoRA. Controls which biases are updated during training.

use_rslora: bool

Whether to use Rank-Stabilized LoRA which sets adapter scaling to lora_alpha/sqrt(r).

modules_to_save: list[str] | None

Additional modules to be trained and saved besides LoRA layers.

init_lora_weights: bool | Literal['gaussian']

Initialization method for LoRA weights.

layers_to_transform: list[int] | int | None

Specific layer indices to apply LoRA transformation to.

layers_pattern: list[str] | str | None

Layer pattern name used with layers_to_transform.

peft_type: str | None

Type of PEFT method being used.

task_type: str | None

Type of task being performed.

inference_mode: bool

Whether to use inference mode.

rank_pattern: dict[str, Any]

Mapping of layer names/patterns to custom ranks different from default r.

alpha_pattern: dict[str, Any]

Mapping of layer names/patterns to custom alphas different from default lora_alpha.

class mattertune.configs.recipes.NoOpRecipeConfig(*, name='no-op')[source]

Example recipe that does nothing.

Parameters:

name (Literal['no-op'])

name: Literal['no-op']

Discriminator for the no-op recipe.

create_lightning_callback()[source]

Creates the PyTorch Lightning callback for this recipe, or returns None if no callback is needed.

Return type:

None

class mattertune.configs.recipes.PeftConfig(*, peft_type=None, task_type=None, inference_mode=False)[source]
Parameters:

  • peft_type (str | None)

  • task_type (str | None)

  • inference_mode (bool)

peft_type: str | None

Type of PEFT method being used.

task_type: str | None

Type of task being performed.

inference_mode: bool

Whether to use inference mode.

class mattertune.configs.recipes.RecipeConfigBase[source]

Base configuration for recipes.

abstract create_lightning_callback()[source]

Creates the PyTorch Lightning callback for this recipe, or returns None if no callback is needed.

Return type:

Callback | None

classmethod ensure_dependencies()[source]

Ensure that all dependencies are installed.

This method should raise an exception if any dependencies are missing, with a message indicating which dependencies are missing and how to install them.

Modules

base

ema

lora

noop