Complete solution for long document processing
It needs to be realized by hardware configuration and parameter tuning:
- Switch to 128k version: Jan-nano-128k natively supports 128k tokens context windows, with key arguments added at startup:
--rope-scaling '{"rope_type":"yarn","factor":3.2,"original_max_position_embeddings":40960}' --max-model-len 131072 - Improved input formats: Use XML/JSON markup segmentation (e.g.
<section>...</section>) to help the model recognize document structure - Memory Optimization Tips: Shut down extraneous processes and reserve swap space at least 1.5 times the size of the model
For very long documents (e.g., books), the recommendation is: first pass theLlamaIndexand other tools to construct vector indexes, and then process them in sections
This answer comes from the articleJan-nano: a lightweight and efficient model for text generationThe
May not be reproduced without permission:AI productivity tools " How to solve the problem of context loss when Jan-nano processes long documents?
