Domain customization requires the completion of the following four stages of operation:
- Data preparation phase::
- Collect professional data and organize it into
lora_medical.jsonlspecification - It is recommended to maintain a 512 token length to fit the model architecture
- Collect professional data and organize it into
- Parameter configuration phase::
- modifications
./model/LMConfig.pyhit the nail on the headn_layerswaiting parameter - align
batch_sizeAvoid video memory overflow (3090 recommended ≤ 8)
- modifications
- Model training phase::
- fulfillment
python train_lora.pyInitiate field habilitation - increase
--use_wandbParameter Monitoring Loss Curve
- fulfillment
- Deployment application phase::
- utilization
serve_openai_api.pyStarting services - pass (a bill or inspection etc)
curlCommand Test Medical Q&A Interface
- utilization
Note: Specialized field training suggests basic pre-training (2-3 rounds) before fine-tuning for LoRA.
This answer comes from the articleMiniMind: 2 hours from scratch training 26M parameters GPT open source toolsThe
May not be reproduced without permission:AI productivity tools " What is the exact process of using MiniMind for domain customized training (e.g. medical quizzing)?
