Five Traps of Agent Harness
- Self-evaluation is a trap. Use an adversarial evaluator. Self-evaluation is a trap. Use an adversarial evaluator. Many teams will do this: Agent generates the answer ↓ Agent scores it himself ↓ Agent says: 95 points The result is overturned after going online. The reason is simple: there is a strong agreement bias between the model and itself.
- Compaction doesn't cure coherence drift. Structured handoffs do. Context compression cannot solve consistency drift, only structured handover can solve it. This is the biggest misunderstanding of many Agent frameworks. Long tasks: 10k token → 50k token → 200k token The context is getting longer and longer. Many people think that the problem will be solved by summarizing it, compressing it, and continuing to run. In fact: Compression ≠ Fidelity The compression process itself is losing information. Finally appeared: target drift, constraint loss, demand deformation. For example, Codex: At the beginning: Reconstruct the payment module. After 20 rounds: Start modifying the login logic and it has gone awry. What is really effective is: Structured Handoff. Fixed format for each handover: Goal: Current State: Completed: Pending: Constraints: Known Risks: Next Action: Anthropic’s Claude Code, OpenAI Codex Cloud, and Cursor Background Agent are all evolving in this direction.
- Make subjective quality gradable with rubrics the model can apply. Make subjective quality gradable with rubrics the model can apply. Many requirements are vague in nature: Is the article well written? Is the UI beautiful? Is the code elegant? The model has no idea what it means: good, beautiful, elegant, so it must be turned into an accurate score sheet.
- Read the traces. They're your primary debugging loop. Trace is the Agent's debugging entry point. Many people adjust Agent input ↓ output and only look at the results. But Agent’s biggest problems all occur in the middle. For example: Planner ↓ Search ↓ RAG ↓ Tool ↓ Reasoning ↓ Action A certain step has gone wrong. Therefore the Agent's logs are more important than the results. For example, OpenAI Agent SDK: Trace/Span/Event LangSmith: Execution Trace Langfuse: Agent Trace Arize Phoenix: LLM Observability The essence is observability. The golden rule of Agent development: 80% of the time watching Trace and 20% of the time changing Prompt. Many teams do the opposite.
- Delete scaffolding when the model catches up. The frontier moves. When the model's capabilities improve, dare to delete scaffolding. This is the death trap for many Agent products. Standard approach in 2024: Prompt + Planner + Reflection + Self Critic + Chain of Thought + Tree Search 2026: GPT-5, Claude Sonnet 5 may already have these capabilities built-in. As a result, the team still retains: 10 layers of Agent and 20 layers of Workflow. The system is becoming more and more complex. Latency is getting higher and higher. The cost is getting more and more expensive. The effect has not improved. Typical cases: Many RAGs in the early days: Query Rewrite ↓ Multi Query ↓ Rerank ↓ Summary ↓ Answer Now many models directly: Search ↓ Answer are good enough. So Harness is not as complicated as possible. Excellent Agent engineers have a habit: every time the model is upgraded: add one layer of capabilities ↓ delete two layers of scaffolding and keep doing subtractions. Therefore, Harness is a dynamic update process as the model evolves.