4-Step Program to Eliminate Text Repetition
The following comprehensive measures can be taken to address the possible problem of duplicate generation of Tifa-Deepsex-14b-CoT:
- Version Selection: Preferred version of Tifa-Deepsex-14b-CoT-Chat as it contains 30K DPO data dedicated training for anti-duplication
- Parameter combinations::
parameters recommended value corresponds English -ity, -ism, -ization repeat_penalty 1.2-1.5 Penalize duplicate tokens top_k 40 Increased sampling diversity top_p 0.95 Balancing quality and diversity - Cue word engineering: Include explicit instructions in system prompts, for example:
"请避免重复使用相同的句式结构和形容词，保持词汇多样性" - Reprocessing methods: Develop a cosine similarity-based filtering script that automatically triggers re-generation when the similarity of consecutive paragraphs is > 0.7, which can be realized using the sentence-transformers library
This answer comes from the articleTifa-Deepsex-14b-CoT: a large model that specializes in roleplaying and ultra-long fiction generationThe
May not be reproduced without permission:AI productivity tools " How to avoid duplicate content when generating novels for Tifa-Deepsex-14b-CoT?
