Few-Shot Stopped Paying on GPT-4o. Qwen Still Wants It.

Your shared prompt library has a shelf life, and nobody on your team knows what it is. A study of 19,620 code generations, published to arXiv on 25 August 2026 as Aging of Prompt Engineering Techniques Across LLM Versions and accepted at ICSME 2026, held the prompting technique constant and varied only the model generation. Few-shot examples that bought GPT-3.5-Turbo an 8.6% to 11.0% lift over a bare zero-shot prompt bought GPT-4o somewhere between −2.4% and +2.4% — noise. The same few-shot arm, moved to the Qwen family, went the other way and gained.

That is the finding that should change your process: prompt technique effectiveness does not decay uniformly as models improve. It decays per model family, in different directions, on the same task. So the model-upgrade ticket your platform team writes every quarter — the one that currently contains a latency diff, a price diff and a smoke test — is missing the arm that actually determines whether the upgrade helps.

What the Study Actually Held Constant

The design is the point. Anastasiia Rudyk, Julian Oertel and Regina Hebig at the University of Rostock took five prompt engineering techniques — Zero-Shot, Few-Shot, Chain-of-Thought (CoT), Contrastive Chain-of-Thought (CCoT) and an adapted Program-of-Thought (PoT) — and ran all of them against six instruction-tuned models arranged into three matched version pairs: GPT-3.5-Turbo → GPT-4o, Qwen2 7B Instruct → Qwen2.5 7B Instruct, and Mistral-7B-Instruct → Mistral-Large.

The task set is a cleaned 218-function subset of CodePromptEval, the 7,072-prompt dataset Khojah et al. built on top of CoderEval for function-level Python generation — context-rich functions that call project APIs, not self-contained puzzles. Every prompt ran three times at temperature 0.2 with a 1,024-token output cap, through OpenRouter for five of the six models and Replicate for Qwen2. After discarding 23 prompts that failed to produce extractable code, 195 tasks were scored on pass@k functional correctness against CoderEval's own Docker test environments. The replication package is public.

A prompt engineering technique is a reusable scaffold you wrap around a task — worked examples, an instruction to reason step by step, a persona — that changes the output without changing the model. The question this study asks is not "does the scaffold work." It is "does the scaffold that worked last year still work on this year's model," and the answer turns out to depend on whose model.

Few-Shot Paid Double Digits on GPT-3.5 and Went Flat on GPT-4o

Few-shot prompting is the technique most likely to be sitting in your codebase, and it is the one that aged worst inside the GPT family. Against a zero-shot baseline of 82, 82 and 81 passing functions out of 195, GPT-3.5-Turbo's few-shot arm scored 91, 91 and 88 — an 8.6% to 11.0% relative lift, reproducible across all three runs. GPT-4o's zero-shot baseline was 84, 83 and 81, and its few-shot arm scored 82, 85 and 82. The sign flips run to run.

Aggregated as the successor's change relative to its predecessor, GPT-4o came in 7.4 percentage points below GPT-3.5-Turbo on the few-shot delta. Qwen2.5 came in 7.9 above Qwen2. Mistral-Large was flat at +0.8. The authors' explanation is that few-shot examples do two jobs — they disambiguate the task and they pin the output format — and a newer instruction-tuned model has already internalised the second job, so the examples add tokens and remove nothing.

The effect does not wash out with more sampling. The authors report "GPT-4o's consistent degradation across all k levels (−4.9% to −6.3%)" at pass@1, pass@2 and pass@3, and read it as evidence that the examples now perturb "an already well-internalized task representation." If you are already sampling multiple candidates — and if you run any kind of self-consistency check, you probably are — the examples stop supplying useful diversity and start perturbing a representation the model already had right.

Chain-of-Thought Reversed Sign Inside One Vendor's Lineup

The sharpest result in the paper is not GPT. It is Mistral. Mistral-7B-Instruct gained 4.3%, 15.4% and 10.3% from a plain "think carefully and logically, explaining your answer step by step" instruction. Mistral-Large lost 2.5%, 5.1% and 3.8% from the identical string. Averaged, the CoT delta across that version pair is −13.8 points, and pass@k analysis shows it worsening as k rises, which the authors read as a structural effect rather than a sampling artefact. Read that pair with one caveat the authors raise themselves: Mistral-Large is roughly 123B parameters against Mistral-7B's 7B, so version and sheer scale move together here in a way the other two pairs avoid.

This is not an isolated academic curiosity. Wang et al., in Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering? (ACM TOSEM, October 2025), found that reasoning models' built-in decomposition "reduces the impact of complex prompts, sometimes making simple zero-shot prompting more effective." Wharton's Generative AI Labs measured the same thing outside code: their second Prompting Science report, also on arXiv, found CoT gave non-reasoning models real gains — Gemini Flash 2.0 +13.5%, Sonnet 3.5 +11.7% — and gave reasoning models +2.9% on o3-mini, +3.1% on o4-mini and −3.3% on Gemini Flash 2.5, while adding 20% to 80% to response time.

Three independent teams, three benchmarks, one shape: the generic reasoning scaffold gets absorbed into the model, and then it becomes overhead. What none of them can tell you is which side of that line your next model sits on, because the line moves per family.

---

The Scaffold That Survived Added Information, Not Instructions

Here is the part that stops this from being "delete your prompts." One technique improved in all three version pairs: Contrastive Chain-of-Thought — CoT plus two correct worked examples and two deliberately wrong ones with their broken rationales. Successor-over-predecessor deltas came in at +11.5 for Qwen, +5.6 for Mistral and +1.1 for GPT. On GPT-4o specifically, where few-shot and CoT both went flat, CCoT delivered +6.0%, +9.6% and +12.4% over zero-shot across the three runs — and the gain grew run over run.

CCoT is also the most expensive scaffold in the set. That is the useful inversion: the cheap scaffolds aged out and the expensive one held. The techniques that only tell a model how to think — step-by-step instructions, output-format demonstrations — are exactly what instruction tuning has been absorbing. The technique that tells it what wrong looks like supplies, in the authors' words, "discriminative signal that mitigates some of the noise introduced by free-form CoT." That is their reading of the result, not a demonstrated mechanism — nothing in the design separates the contribution of the negative examples from the extra tokens they cost.

The adapted Program-of-Thought arm is the counterexample worth naming: it posted a positive successor delta for GPT (+4.2) while remaining negative in absolute terms at every pass@k level. A technique can improve across a version bump and still be worse than doing nothing. If your evaluation reports deltas without absolute numbers, you will ship that.

The same principle holds where the scaffold carries domain content rather than instruction. Bruni et al.'s benchmark for secure code generation found a security-focused prompt prefix cut vulnerability occurrence by up to 56% on GPT models. A prefix that names threat classes is information. "Think step by step" is not.

The Vendors Have Been Telling You This in the Documentation

Both frontier labs now ship prompting guidance that only makes sense if prompts age. OpenAI's reasoning best practices instructs developers to avoid chain-of-thought prompts — "prompting them to 'think step by step' or 'explain your reasoning' is unnecessary" — and to "try to write prompts without examples first," adding few-shot only if output requirements demand it. It states plainly that techniques carried over from GPT models "may not enhance performance (and can sometimes hinder it)."

Anthropic goes further and ships a separate prompting page per model — Fable 5, Sonnet 5, Opus 5, Opus 4.8 each get their own — with migration considerations at the end of the shared guide. Item six is the whole thesis in one instruction: "If your prompts previously encouraged the model to be more thorough or use tools more aggressively, dial back that guidance. Claude 4.6 models are more proactive and may overtrigger on instructions that were needed for previous models."

Overtriggering is not an accuracy footnote. It is an agent making extra tool calls, running longer, and billing you for it. This is the same failure pattern as a repair loop that keeps iterating past the point of improvement, and the same class of finding as memory scaffolding that costs accuracy when you never test with it off.

Meanwhile the clock is not optional. OpenAI's deprecations page commits to a minimum of six months' notice for generally available models, and 2026 retires legacy GPT snapshots on 23 October and GPT-5 and o3 snapshots on 11 December. Six months is your re-evaluation cycle whether you budget for one or not.

What Carrying Dead Scaffolding Costs

Run the arithmetic on your own traffic. A few-shot block with two worked input/output pairs plus a CoT instruction is on the order of 500 extra input tokens per call. At one million calls a month that is 500 million input tokens: $2,000 a month at gpt-5.6-sol's $4.00 per million uncached input rate, per OpenAI's pricing page as of August 2026, or $200 if the scaffold sits in a stable cached prefix at $0.40. That $4.00 is promotional pricing OpenAI commits to only through 21 November 2026, so treat it as a floor. A CCoT block with four examples is roughly double that. Neither number will show up in a budget review, and both are pure waste if the arm is not earning.

The optimistic version of the same math: PET-Select, from Wang, DaghighFarsoodeh and Pham, routed each query to a technique based on predicted code complexity and reported "up to a 1.9% improvement in pass@1 accuracy, along with a 74.8% reduction in token usage" on MBPP and HumanEval (arXiv:2409.16416). Most of that saving comes from not applying heavy scaffolds where they do nothing — which is only knowable if you measured. Teams that already track per-run token cost by prompt variant have the instrumentation for this; teams billing a single aggregate do not.

Where This Study Is Weak — Read It Before You Cite It

The strongest objection is the model roster, and the authors state it themselves. GPT-3.5-Turbo → GPT-4o, Qwen2 → Qwen2.5 and Mistral-7B → Mistral-Large-2407 are 2023–2024 vintage. Nobody has replicated this on GPT-5.x, Claude 5 or Qwen3, so the direction of aging on the models you actually run in production is an open question — which is precisely the argument for measuring it yourself rather than importing a conclusion.

Three more limits, all disclosed in the paper's threats-to-validity section:

• The Mistral pair is not a clean version bump. Mistral-Large is roughly 123B parameters against Mistral-7B-Instruct's 7B. The authors acknowledge they cannot disentangle architectural evolution from sheer parameter count, because no intermediate open-access Mistral of comparable size existed.
• Three runs per prompt-model combination, and no significance testing. The authors say ten would be needed for robust confidence intervals under LLM non-determinism, and the paper reports descriptive statistics only. Read every delta above as a direction, not a measured effect size.
• The replication does not reproduce its own source study's levels. On the same 195 tasks, pass@1 came in an average 6.5 percentage points below Khojah et al.'s original — zero-shot 42.3% against 47.7%, few-shot 43.6% against 51.6% — with few-shot showing the largest 8.0-point gap. Relative ordering held; absolute levels did not.

Scope is narrow too: Python, function-level, correctness via unit tests only. No security scanning, no readability, no full-program generation. And a fair reading of the counter-case is that a platform team has better things to do than run a prompt tournament every quarter. The answer is that you are not running a tournament. You are adding one control arm to an eval suite you should already have.

Put These in the Next Model-Upgrade Ticket

This Week:

1. Inventory the scaffolds. Grep the shared prompt library and the agent system prompts for "step by step," "for example," persona preambles and worked input/output blocks. Count the tokens each one adds per call. Most teams have never had that number written down.
2. Add a stripped zero-shot arm to your existing eval set — the same task, same model, none of the scaffolding. It is a config flag, not a project. Without it you cannot tell a scaffold that helps from a scaffold you are paying for.
3. Pull the per-model prompting page for whatever you run today and diff it against the one for the version you are about to move to. Anthropic publishes them per model; OpenAI's reasoning guidance is separate from its GPT guidance for the same reason.

Before the Next Version Bump:

4. Make "re-run the prompt-technique comparison" a required checklist item on the upgrade ticket, alongside the latency and price diff. Score every arm on absolute pass rate, not just delta versus the previous model — the Program-of-Thought result is what happens when you only look at deltas.
5. Score at the sampling depth you actually ship at. Several techniques hold at pass@1 and decay at pass@3; if you sample three candidates in production, a pass@1 result is the wrong measurement. Grade on the acceptance criterion you will really use.
6. Split your scaffolds into "adds information" and "adds instruction," and retire the second category first on any model whose vendor documents internal reasoning. Contrastive negative examples, threat-class prefixes and domain vocabulary are information. "Think carefully" is not.

Before Renewal:

7. Put a prompt-regression clause in the eval budget, not the engineering backlog. At six months' deprecation notice you will do this at least twice a year per provider, and it needs an owner. Pin the winning technique per model version in your prompt registry with the eval run that justified it, so the next upgrade starts from evidence rather than folklore.
8. Re-baseline your token forecast after the comparison, not before. If a scaffold comes out, your input-token profile changes and any usage commitment priced against the old profile is now mis-sized.

The Bottom Line

The industry spent three years treating prompt engineering as accumulated craft — a library you build once, share across teams, and carry forward. This study holds the technique fixed and moves only the model generation, across three matched within-family version pairs, and what it found is that the library is not an asset. It is a calibration against one model version, and calibration expires.

We already know how this ends, because we have watched it before. Compiler flags tuned for one CPU generation, JVM garbage-collection settings tuned for one heap profile, database hints pinned against one query planner — each was a hard-won optimisation that quietly became a performance bug the moment the thing underneath it improved, and each stayed in production for years because nobody re-ran the measurement that justified it. Prompt scaffolds are the same object, on a six-month cycle instead of a five-year one. The upgrade decision your team already runs on capability and price has one more column now, and it is the cheapest one to add.

Your prompts are not knowledge. They are a fit to a model that is about to be retired.

Continue Reading

• Haiku Burned More Tokens Than Sonnet. Spec It in Code.
• Agent Memory Cost 14 Points at Best. Test With It Off.
• ChatGPT Made Spec Reviewers Worse. Save It for Round 2.
• Your Terraform Repair Loop Broke Passing Checks. Stop at 3.
• Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.
• Claude Opus 4.6 vs 4.7: Should Your Enterprise Upgrade?
• Best AI Agent Monitoring: Langfuse, Then a Real Kill Switch