学习方法

学习时，请使用 projects 目录下的 yaml 配置文件.<br> 例如， projects/opt/exp001.yml 的内容如下.

training_config:
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
num_train_epochs: 1
dataloader_num_workers: 16
fp16: true
optim: "adamw_torch"
learning_rate: 5.0e-5
logging_steps: 100
evaluation_strategy: "steps"
save_strategy: "steps"
eval_steps: 4000
save_steps: 4000
save_total_limit: 1
deepspeed: ./configs/deepspeed/ds_config_zero1.json
output_dir: ./output/
report_to: "wandb"

model_config:
fp16: true
pretrained_path: # None or path to model weight
model_type: git_llm
language_model_name: facebook/opt-350m
vision_model_name: openai/clip-vit-base-patch16
num_image_with_embedding: 1 # if 1, no img_temporal_embedding
max_length: 512
keys_to_finetune:
   - visual_projection
   - num_image_with_embedding
keys_to_freeze: []

use_lora: true
lora:
   r: 8
   lora_alpha: 32
   target_modules:
      - q_proj
      - k_proj
      - v_proj
   lora_dropout: 0.01
   bias: none
   task_type: CAUSAL_LM

dataset_config_path:
- ./configs/datasets/m3it.yaml

training_config "为训练设置, "model_config "为模型设置，"dataset_config_path "为数据集设置.<br> 目前支持以下可为 model_type 指定的 LLM 模块. 将来会添加更多支持的模块.

• LLama-2

• MPT

• OPT

• GPT-NeoX

• Japanese StableLM

• ELYZA-japanese-Llama-2

要开始学习, 请执行以下命令.

./scripts/run.sh

学习需要 GPU; 我们在 Ubuntu 20.04 和 CUDA 11.7 上对系统进行了测试.