The Need for Cue Engineering in Large Language Modeling

下面主要介绍了提示工程的基本理念，及其如何提升大型语言模型(LLM)的性能…

LLM's interface:One of the key reasons why large-scale language models are so hot is that their text-to-text interfaces enable a minimalist operational experience. In the past, solving tasks with deep learning usually required us to fine-tune at least some of the data to teach the model how to solve the corresponding task. However, most models only had specialized depth on specific tasks. But now, thanks to LLM's excellent contextual learning capabilities, we can solve a variety of problems with a single textual hint. The original complex problem solving process has been abstracted by the language!

“提示工程是一门相对较新的学科，主要聚焦于为各种应用和研究主题高效地利用语言模型进行开发和优化提示。” – 引自[1]

What is the Tip Project?LLM's simplicity makes it accessible to more people. You don't need to be a data scientist or a machine learning engineer to use LLM, just understand English (or any language of your choice) and you can use LLM to solve relatively complex problems! However, when LLM is allowed to solve problems, the results often depend heavily on the textual cues we give to the model. As a result, cue engineering (i.e., the method of using experience to test different cues to optimize the performance of LLM) has become very popular and has had a great deal of influence, which has led us to many techniques and best practices.

How to design a prompt:There are many ways to design LLM prompts. However, most hint strategies share a few common points:

– 输入数据(Input Data)：LLM需要处理的实际数据（比如，需要翻译或分类的句子，需要总结的文档，等等）。
– 示例(Exemplars)：在提示中包含的正确输入输出对的实例。
– 指令(Instruction)：模型预期产出的文字说明。
– 标签(Indicators)：在提示中创建结构的标签或格式元素。
– 上下文(Context)：在提示中给予LLM的任何额外信息。

In the following illustration, we give an example of how all the above mentioned cue elements can be merged in a single sentence categorization cue.

Context window:During pre-training, the LLM sees input sequences of a specific length. During pre-training, we choose this sequence length as the "context length" of the model, which is the maximum sequence length that the model can handle. If a piece of text is given that is significantly longer than this pre-determined context length, the model may react unpredictably and output incorrect results. In practice, however, there are methods-such as self-expansion [2] or positional interpolation [3]-that can be used to extend the model's context window.

Recent research on LLMs has emphasized the creation of long context windows, which allow the model to process more information per cue (e.g., more examples or more context). However, not all LLMs are able to process their contextual information perfectly! The ability of an LLM to process information in a long context window is usually evaluated by the "needle in a haystack test" [4], which.

1. Embed a random fact in the context.
2. The model is required to find this fact.
3. Repeat this test in a variety of context lengths and locations of facts in the context.

This test generates an image like the one below (source [4]), in which we can easily spot the problems with the context window.

My cue engineering strategy:The details of cue engineering will vary depending on the model used. However, there are some general principles that can also be used as a guide in the cue engineering process:

– 实证法：提示工程的第一步就是设定一个可靠的评估你的提示的方式（如，通过测试案例，人类的评估，或LLM作为评判）以便简单地衡量对提示的修改。
– 由简入繁：首次尝试的提示应应尽可能简单，而不是一开始就尝试复杂的链式思考提示或其他特殊的提示技术。最初始的提示应该简单，然后逐渐增加复杂程度，同时衡量性能的变化，来判断是否需要增加额外的复杂性。
– 尽可能具体和直接：尽力消除提示中的模糊性，在说明预期输出时尽量简洁、直接、具体。
– 加入示例：如果描述预期产出比较困难，试着在提示中加入示例。示例提供预期输出的具体案例，可以消除模糊性。
– 尽可能避免复杂性：复杂的提示策略有时是必要的（例如，为了解决多步推理问题），但是我们在采取这种方法前要三思。采取实证的观点，并使用已经建立的评价策略真正判断是否必要加入这种复杂性。

To summarize all of the above, my personal cue engineering strategy is to i) commit to a good evaluation framework, ii) start with a simple cue, and iii) slowly add complexity as needed to achieve the desired level of performance.

Annotation:

[1] https://promptingguide.ai
[2] https://arxiv.org/abs/2401.01325
[3] https://arxiv.org/abs/2306.15595
[4] https://github.com/gkamradt/LLMTest_NeedleInAHaystack