Fine-tune a language model with LoRA

LoRA trains a small adapter instead of the whole model. With TRL it is one script. This one teaches Qwen2.5 0.5B from a public chat dataset:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("trl-lib/Capybara", split="train[:1000]")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=dataset,
    args=SFTConfig(
        output_dir="output/qwen-lora",
        num_train_epochs=1,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
        report_to="none",
    ),
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", task_type="CAUSAL_LM"),
)
trainer.train()
trainer.save_model()  # the LoRA adapter, about 35 MB, in output/qwen-lora
nido create tune --gpu l4 --network huggingface.co --network "**.hf.co"
nido cp ./lora tune:/sandbox/lora
nido exec tune -- bash -c "cd /sandbox/lora && python train.py"
nido cp tune:/sandbox/lora/output ./output

Use your own data with load_dataset("json", data_files="data.jsonl"), one conversation per line in the form {"messages": [{"role": "user", "content": "…"}, {"role": "assistant", "content": "…"}]}. Larger models fit with 4-bit quantization (bitsandbytes, QLoRA). The same libraries also do preference and reinforcement training, for example DPOTrainer and GRPOTrainer in TRL.