Fine-tune a language model with LoRA
LoRA trains a small adapter instead of the whole model. With TRL it is one script. This one teaches Qwen2.5 0.5B from a public chat dataset:
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
dataset = load_dataset("trl-lib/Capybara", split="train[:1000]")
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
train_dataset=dataset,
args=SFTConfig(
output_dir="output/qwen-lora",
num_train_epochs=1,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4,
bf16=True,
logging_steps=10,
report_to="none",
),
peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", task_type="CAUSAL_LM"),
)
trainer.train()
trainer.save_model() # the LoRA adapter, about 35 MB, in output/qwen-lora
nido create tune --gpu l4 --network huggingface.co --network "**.hf.co"
nido cp ./lora tune:/sandbox/lora
nido exec tune -- bash -c "cd /sandbox/lora && python train.py"
nido cp tune:/sandbox/lora/output ./output
Use your own data with load_dataset("json", data_files="data.jsonl"), one conversation per line
in the form {"messages": [{"role": "user", "content": "…"}, {"role": "assistant", "content": "…"}]}. Larger models fit with 4-bit quantization (bitsandbytes, QLoRA). The same
libraries also do preference and reinforcement training, for example DPOTrainer and
GRPOTrainer in TRL.