Prompt Engineering

Prompt engineering is the practice of designing the input given to a generative model, including its instructions, context, examples, and requested output format, to steer what the model produces without changing its model weights. It took off with GPT-3: in Language Models are Few-Shot Learners (2020), OpenAI showed that a 175-billion-parameter model could perform new tasks without any gradient updates when the prompt described the task and gave a few examples, an ability the paper calls in-context learning. Today the prompt is the main way to control large language models (LLMs), vision-language models (VLMs), text-to-image generators, and promptable vision models.

Core Techniques#

• Instructions and roles: A system prompt sets the model's role, rules, and audience, and delimiters such as headings or XML-style tags separate instructions from the data being processed.
• Few-shot examples: Worked input-output pairs show the task and the answer format more precisely than a description. A prompt with no examples is zero-shot.
• Output format: The prompt names the format, such as a list or JSON with given keys. When code must parse the result, structured outputs enforce a schema during decoding instead of relying on the request.
• Chain-of-thought prompting: Asking for intermediate reasoning steps before the answer, which raises accuracy on multi-step problems.
• Prompt chaining: Splitting a complex job into several prompts, each with one narrow task, connected by code.
• Retrieved context: Inserting documents or data found by search so the model answers from supplied facts instead of memory.

Supplied facts reduce hallucinations, and where they sit matters: the Lost in the Middle study found that models use information best when it appears at the beginning or end of a long context window and worst when it is buried in the middle.

The effect of examples is easiest to see on a base model that was never instruction-tuned. This snippet gives GPT-2, loaded with Hugging Face Transformers, the same instruction with and without three examples:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

# The same instruction, without and with three worked examples that show the answer format
instruction = "Convert each country to its capital city.\n"
prompts = {
    "zero-shot": instruction + "Italy:",
    "few-shot": instruction + "France: Paris\nJapan: Tokyo\nCanada: Ottawa\nItaly:",
}

for name, prompt in prompts.items():
    inputs = tokenizer(prompt, return_tensors="pt")
    output = model.generate(**inputs, max_new_tokens=4, do_sample=False, pad_token_id=tokenizer.eos_token_id)
    completion = tokenizer.decode(output[0, inputs.input_ids.shape[1] :])
    print(f"{name}: {completion!r}")

It prints zero-shot: '\n\nItaly is' and few-shot: ' Rome\nSpain:'. The instruction alone left GPT-2 writing prose about Italy, while the three examples made it answer "Rome" in the expected format and continue the pattern with the next country.

Prompt Design Is Empirical#

Small changes in wording or layout can move results a long way. Melanie Sclar and colleagues measured accuracy differences of up to 76 points on LLaMA-2-13B from formatting changes alone, such as separators, casing, and spacing, and the sensitivity remained with larger models, more examples, and instruction tuning. Prompts are therefore developed like code: tested against a fixed set of inputs with known good outputs, revised on the failures, and versioned, a routine part of LLMOps.

The loop can be automated. Automatic Prompt Engineer (APE), from Yongchao Zhou and colleagues, has an LLM propose candidate instructions and keeps the ones that score best on examples, and its instructions matched or beat those written by people on 19 of 24 tasks in Large Language Models Are Human-Level Prompt Engineers.

Prompts for Vision Models#

In computer vision, prompts take several forms. Text-to-image generators such as Stable Diffusion respond to descriptions of subject, style, lighting, and composition, plus negative prompts listing what to avoid. Promptable segmentation models accept visual prompts such as points and boxes, and Meta's SAM 3 also accepts short noun phrases and segments every instance of the named concept, which Ultralytics Platform uses for text-prompted auto-annotation.

Open-vocabulary detectors turn class names into prompts. Ultralytics YOLOE detects and segments whatever classes set_classes() names, such as "forklift" or "double-decker bus", by comparing region features against the text embeddings of those names, so wording directly changes what it finds. Because the match is on appearance, prompts that depend on state, position, or comparison, such as "damaged", "left-most", or "the one being carried", have no reliable visual handle, and names close to everyday categories work best.

Prompt Engineering vs. Related Terms#

• Prompt tuning: Learns continuous prompt vectors by gradient descent on labeled data. Prompt engineering writes the prompt by hand in natural language.
• Fine-tuning: Changes the model's weights by training on a labeled dataset. Prompt engineering leaves the model unchanged and works at inference time.
• Context engineering: Covers everything that enters the model's context in an application, including retrieved documents, tool results, and memory. Prompt engineering is the part concerned with writing the instructions and examples.
• Prompt injection: An attack that hides instructions in text or images the model reads, so they override the developer's prompt.