Small Data Modeling Solutions
When the training data is less than 2000, the following methods can be used to enhance the effect:
- data enhancement: Use GPT-4 to generate semantically similar extended dialogs, taking care to maintain the original language style. Recommended prompt format: "Please rewrite the following dialog in the style of [username]: ..."
- transfer learning: Load public conversation data such as Zhihu and Weibo for pre-training before fine-tuning personal data. Modify the settings.json in the
pretrained_pathparameters - regularization technique: set dropout_rate=0.3 and enable early stopping (patience value set to 5 epochs)
Additional suggestions: 1) Use K-fold cross-validation 2) Limit LoRA rank ≤ 16 3) Add label smoothing (smoothing=0.1). After training is done be sure to useweb_demo.py --test_mode=truePerform overfitting detection
This answer comes from the articleWeClone: training digital doppelgangers with WeChat chats and voicesThe
May not be reproduced without permission:AI productivity tools " How to overcome the model overfitting problem caused by small chat logs?
English 
简体中文 
日本語 
Deutsch 
Português do Brasil