LlamaFactory Agent工作流

Requirement

Hardware Requirement

\* estimated

Install dependencies

pip install -r requirements-dev.txt

Install the dependencies for the compilation tools. Note that the commands for this step may vary depending on the operating system. The following are provided for reference

apt-get install -y build-essential cmake

Getting Started

Installation

[!IMPORTANT] Installation is mandatory.

Install from Source

git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

Optional dependencies available: metrics, deepspeed. Install with: pip install -e . && pip install -r requirements/metrics.txt -r requirements/deepspeed.txt

Additional dependencies for specific features are available in examples/requirements/.

Install from Docker Image

docker run -it --rm --gpus=all --ipc=host hiyouga/llamafactory:latest

This image is built on Ubuntu 22.04 (x86\_64), CUDA 12.4, Python 3.11, PyTorch 2.6.0, and Flash-attn 2.7.4.

Find the pre-built images: https://hub.docker.com/r/hiyouga/llamafactory/tags

Please refer to build docker to build the image yourself.

<details><summary>Setting up a virtual environment with <b>uv</b></summary>

Create an isolated Python environment with uv:

uv run llamafactory-cli webui

</details>

<details><summary>For Windows users</summary>

Install PyTorch

You need to manually install the GPU version of PyTorch on the Windows platform. Please refer to the official website and the following command to install PyTorch with CUDA support:

pip uninstall torch torchvision torchaudio
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
python -c "import torch; print(torch.cuda.is_available())"

If you see True then you have successfully installed PyTorch with CUDA support.

Try dataloader_num_workers: 0 if you encounter Can't pickle local object error.

Install BitsAndBytes

If you want to enable the quantized LoRA (QLoRA) on the Windows platform, you need to install a pre-built version of bitsandbytes library, which supports CUDA 11.1 to 12.2, please select the appropriate release version based on your CUDA version.

pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.2.post2-py3-none-win_amd64.whl

Install Flash Attention-2

To enable FlashAttention-2 on the Windows platform, please use the script from flash-attention-windows-wheel to compile and install it by yourself.

</details>

<details><summary>For Ascend NPU users</summary>

To install LLaMA Factory on Ascend NPU devices, please upgrade Python to version 3.10 or higher: pip install -r requirements/npu.txt. Additionally, you need to install the Ascend CANN Toolkit and Kernels. Please follow the installation tutorial.

You can also download the pre-built Docker images:

```bash

Docker Hub

docker pull hiyouga/llamafactory:latest-npu-a2 docker pull hiyouga/llamafactory:latest-npu-a3

Install bitsandbytes from source

Compile & install

cmake -DCOMPUTE_BACKEND=npu -S . make pip install .

2. Install transformers from the main branch.

1. Set double_quantization: false in the configuration. You can refer to the example.

</details>

Build Docker

For CUDA users:

cd docker/docker-cuda/
docker compose up -d
docker compose exec llamafactory bash

For Ascend NPU users:

cd docker/docker-npu/
docker compose up -d
docker compose exec llamafactory bash

For AMD ROCm users:

cd docker/docker-rocm/
docker compose up -d
docker compose exec llamafactory bash

<details><summary>Build without Docker Compose</summary>

For CUDA users:

docker build -f ./docker/docker-cuda/Dockerfile \
    --build-arg PIP_INDEX=https://pypi.org/simple \
    -t llamafactory:latest .

docker run -dit --ipc=host --gpus=all \
    -p 7860:7860 \
    -p 8000:8000 \
    --name llamafactory \
    llamafactory:latest

docker exec -it llamafactory bash

For Ascend NPU users:

docker build -f ./docker/docker-npu/Dockerfile \
    --build-arg PIP_INDEX=https://pypi.org/simple \
    -t llamafactory:latest .

docker run -dit --ipc=host \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -p 7860:7860 \
    -p 8000:8000 \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    --name llamafactory \
    llamafactory:latest

docker exec -it llamafactory bash

For AMD ROCm users:

docker build -f ./docker/docker-rocm/Dockerfile \
    --build-arg PIP_INDEX=https://pypi.org/simple \
    -t llamafactory:latest .

docker run -dit --ipc=host \
    -p 7860:7860 \
    -p 8000:8000 \
    --device /dev/kfd \
    --device /dev/dri \
    --name llamafactory \
    llamafactory:latest

docker exec -it llamafactory bash

</details>

<details><summary>Use Docker volumes</summary>

You can uncomment VOLUME [ "/root/.cache/huggingface", "/app/shared_data", "/app/output" ] in the Dockerfile to use data volumes.

When building the Docker image, use -v ./hf_cache:/root/.cache/huggingface argument to mount the local directory to the container. The following data volumes are available.

• hf_cache: Utilize Hugging Face cache on the host machine.
• shared_data: The directionary to store datasets on the host machine.
• output: Set export dir to this location so that the merged result can be accessed directly on the host machine.

</details>

Deploy with OpenAI-style API and vLLM

API_PORT=8000 llamafactory-cli api examples/inference/qwen3.yaml infer_backend=vllm vllm_enforce_eager=true

[!TIP] Visit this page for API document. Examples: Image understanding | Function calling

Used by [Amazon](https://aws.amazon.com/cn/blogs/machine-learning/how-apoidea-group-enhances-visual-information-extraction-from-banking-documents-with-multimodal-models-using-llama-factory-on-amazon-sagemaker-hyperpod/), [NVIDIA](https://developer.nvidia.com/rtx/ai-toolkit), [Aliyun](https://help.aliyun.com/zh/pai/use-cases/fine-tune-a-llama-3-model-with-llama-factory), etc.

Easily fine-tune 100+ large language models with zero-code [CLI](#quickstart) and [Web UI](#fine-tuning-with-llama-board-gui-powered-by-gradio)

</div>

👋 Join our WeChat, NPU, Lab4AI, LLaMA Factory Online user group.

\ English | [中文 \]

Fine-tuning a large language model can be easy as...

https://github.com/user-attachments/assets/3991a3a8-4276-4d30-9cab-4cb0c4b9b99e

Start local training: - Please refer to usage

Start cloud training: - Colab (free): https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing - PAI-DSW (free trial): https://gallery.pai-ml.com/#/preview/deepLearning/nlp/llama_factory - LLaMA Factory Online: https://www.llamafactory.com.cn/?utm_source=LLaMA-Factory - Alaya NeW (cloud GPU deal): https://docs.alayanew.com/docs/documents/useGuide/LLaMAFactory/mutiple/?utm_source=LLaMA-Factory

Read technical notes: - Documentation (WIP): https://llamafactory.readthedocs.io/en/latest/ - Documentation (AMD GPU): https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/fine_tune/llama_factory_llama3.html - Official Blog: https://blog.llamafactory.net/en/ - Official Course: https://www.lab4ai.cn/course/detail?id=7c13e60f6137474eb40f6fd3983c0f46&utm_source=LLaMA-Factory

[!NOTE] Except for the above links, all other websites are unauthorized third-party websites. Please carefully use them.

Quickstart

Use the following 3 commands to run LoRA fine-tuning, inference and merging of the Qwen3-4B-Instruct model, respectively.

llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml
llamafactory-cli export examples/merge_lora/qwen3_lora_sft.yaml

See examples/README.md for advanced usage (including distributed training).

[!TIP] Use llamafactory-cli help to show help information. Read FAQs first if you encounter any problems.

Supported Models

[!NOTE] For the "base" models, the template argument can be chosen from default, alpaca, vicuna etc. But make sure to use the corresponding template for the "instruct/chat" models. If the model has both reasoning and non-reasoning versions, please use the _nothink suffix to distinguish between them. For example, qwen3 and qwen3_nothink. Remember to use the SAME template in training and inference. \: You should install the transformers from main branch and use DISABLE_VERSION_CHECK=1 to skip version check. \\*: You need to install a specific version of transformers to use the corresponding model.

Please refer to constants.py for a full list of models we supported.

You also can add a custom chat template to template.py.