Many people discuss Agent, and their focus still remains on: - Is the answer correct this time - Is the tool adjusted accurately this time - Is the task completed this time. But from a system perspective, the more dangerous problem is actually: systemic drift at the Agent level. It is a system with memory, tools, and feedback loops. After multiple rounds of operation, it deviates more and more from the original goal, and often "seems to still work." And it’s basically difficult for you to solve problems all at once, just like plugging a hole, one problem pops up all at once. This is the most easily overlooked type of risk when many Agent systems move from demo to production.
- What is systemic drift at the agent level? In a word: Systematic drift = Agent's goals, context, tool behavior, and evaluation signals gradually deviate in multiple rounds of closed loops, eventually producing a result that "seems reasonable locally but is wrong globally." The focus is not on "making a mistake once", because errors are not isolated, deviations will be amplified by memory and feedback loops, and the system will continue to make mistakes more and more self-consistently, so it is more troublesome than ordinary hallucination (hallucination). Ordinary hallucination is more like answering a certain point incorrectly, while systematic drift is more like the entire runtime leading itself astray.
- What is the essential difference between it and ordinary model errors? Ordinary LLM errors are usually single-round errors: - One fact is wrong - One inference is wrong - One tool parameter is filled in incorrectly But the Agent is a closed-loop system, which has several more layers: - The planner will split the task - The memory will retain the history - The context assembler will assemble the context - The tool layer will call the external world - The evaluator will score "success" - Policy / retry / fallback will affect the next round of actions. So the problem changes: "Is this closed loop still optimizing towards the original goal?" Once the loop is closed the direction If you make a mistake, you may be "seriously doing something wrong" in every subsequent round.
- Where does systematic drift usually begin? There are 4 most common sources: Category 1: Target drift. The initial target is A, but gradually becomes A', A'', or even B during execution. Typical performance: - The planner replaces "complete the task" with "produce as soon as possible" - replaces "high quality and correct" with "looks completed" - replaces "solve the user's real problem" with "let the evaluator pass" This type of problem is essentially: the optimization goal is rewritten during execution. Category 2: Memory pollution. More memory is not better. If old information, incorrect intermediate states, and invalid assumptions are brought into subsequent rounds, the system will become increasingly hijacked by historical noise. Typical manifestations: - Expired state is regarded as the current truth - Wrong summaries are entered into long-term memory - Early wrong assumptions are repeatedly cited in the future - Context is contaminated by "wrong but high-confidence" summaries. The result is: the system is not reasoning based on facts, but reasoning based on its own accumulated illusions. Category 3: Tool misuse. Once Agent can adjust the tool, the error will no longer stay at the text layer. Typical manifestations: - Selecting the wrong tool - Passing the wrong parameters - Wrong calling sequence - Treating intermediate results as final results - Abusing a high-success rate tool in order to "complete the task" The essence of this type of risk is that the tool brings bias from the cognitive layer to the execution layer. Category 4: Evaluation Distortion Many systems think that adding an evaluator is safe. But the evaluator itself can also be a source of drift. Typical manifestations: - The indicator is too rough and only looks at "whether it looks like it is completed" - The reward is exploited - The automatic evaluation is inconsistent with the real business goals - The system learns to "pass the evaluation" and does not "solve the real problem" This is actually the Agent version of reward hacking.
- Why is it getting worse? What is the principle? Because Agent is not a one-time forward reasoning, but a dynamic system with feedback. Systematic drift usually conforms to this chain: small deviation → enters the memory/state → affects the next round of decision-making → produces a larger deviation → is absorbed by the system → forms a self-reinforcing loop. In other words: - A little deviation in the first round - Continue in the "deviated state" in the second round - The system feels that it is right starting from the third round - What you see after the fourth round is "organized error" This is very similar to drift in control systems, recommendation systems, and reinforcement learning. There are three core mechanisms:
- State accumulation Agent is not stateless. It writes past decisions, summaries, tool results, and memories back to the system. So the deviation is not cleared, but saved.
- Feedback amplification: subsequent actions depend on the state of the previous step. As long as the previous step is slightly wrong, the next step will continue to optimize based on the error.
- Local correctness covers up global errors. The most dangerous thing is that many nodes look "OK" just by looking at them. - The tool call was successful - The output format is correct - The evaluator gave a high score - The log looks like the process is complete but the overall goal has deviated. Being locally correct does not mean that it is globally stable.
- What are the consequences of systematic drift? If it's just a demo, the result may just be "looking weird". But once you enter production, reality will slap you in the face:
- The results are increasingly untrustworthy. The entire link steadily produces biased results, and repairing it is not a simple bug fix.
- The system will become increasingly difficult to debug. Traditional systems can locate and repair single-point bugs, while Agent systems require linkage issues across planner/memory/tool/eval.
- Errors will be solidified in memory, because after the bad intermediate state enters memory, it is simply not enough to repair the prompt later.
- The system will generate false confidence, making the system appear to be right when it is wrong, spinning in an error spiral.
- Business accidents will occur in high-value scenarios, such as: - Error in executing automated processes - Error in summarizing knowledge - Error in calling enterprise systems - Decline in stability in customer service, finance, and operation and maintenance scenarios
- How to prevent it? Treat Agent as a system to manage, rather than as a prompt to adjust. First: the goals should be explicit, hierarchical, and checkable, and should be broken down into: - Original business goals - Current wheel goals - Boundaries that prohibit optimization - Success judgment conditions. Let the system always know: - What to pursue - What not to pursue - To what extent is it considered completed? Otherwise, the planner will easily change the topic on its own. Second: Memory needs to be managed, not piled up. The focus of memory is not to simply "save more", but to become like this: - What can be entered into long-term memory - What is just an intermediate state - What must expire - What needs to be trusted - What must be traceable. Without a complete memory strategy, memory will quickly turn from an asset into a source of pollution. Third: The tool layer must have constraints and do not run naked. Tool calls must at least have: - Clear tool selection rules - Parameter verification - Status checks before and after calling - Human confirmation or double judgment of high-risk tools - Semantic verification of tool results. Do not treat tool use only as a function calling success rate issue. It essentially controls the entire execution process. Fourth: The evaluation must cover the closed loop. It is not enough to only evaluate the output and only evaluate the final answer. We need to look at the following layers: - Is the planner reasonable? - Is the tool route reasonable? - Is the memory retrieval reasonable? - Is the context assembly reasonable? - Is the evaluator exploited? - Is the fallback triggered properly? Otherwise, you will only know that "it was wrong in the end", but you will not know which layer the error started to grow from. Fifth: It must have continuous observability. The Agent is not suitable for "looking at it after going online". It must have: - trace - intermediate state - memory snapshot - tool call timeline - evaluator decision log - drift signal dashboard. Because the most feared thing about systematic drift is "you can't see when it starts to drift".
- How to correct it? There are 3 levels of correction. The first layer: Stop bleeding. First block the drift and continue to zoom in. Common actions: - Clear contaminated memory - Pause long-term writeback - Downgrade tool permissions - Shorten the context window - Change to a more conservative fallback - Manually take over high-risk links. First, let the system not continue to lead itself astray. Level 2: Locate the source of drift. Answer clearly: - Is it target drift? - Is it memory pollution? - Is tool route wrong? - is the evaluator distorted? - Is the release gate too weak? Don't call all problems hallucinations. The third layer: precipitate bad cases into guardrail and eval. A truly mature system will not be satisfied with "it is fixed this time". This drift should be settled into: - new eval case - new memory rule - new tool governance - new release gate - new observability signal, otherwise it will come back.
- A Practical Judgment Criteria If you look at an Agent system and can only answer whether its output this time is correct but cannot answer: - Why is it optimized in this direction - Which of its states are written back - How does it ensure that it will not be more biased in the next round - Will its evaluator be exploited - Will its memory solidify errors? Then there is a high probability that this system has not truly entered a "governable" engineering state.
- The last sentence The most difficult thing in the Agent era is that the system is not just “able to do things”. What’s more, you need to have memory, tools, feedback, and the ability to automatically execute, so that you don’t drift for a long time. This is the watershed from demo to production. This is also a problem that has plagued many vibe coders recently. One of the reasons why the harness project is widely mentioned is that it truly values every detail or dark knowledge that is easily ignored by the model/agent from the engineering level, so that it can truly control the agent system and prevent systematic drift. The evolution of the model will not continuously internalize the best practices of harness, but this is like an exhaustive question. It is impossible for the model to internalize all harness scenarios. This is also an opportunity for the application layer, and there are endless opportunities. There is no user who doesn’t like better and better product experience, which means that the optimization of the harness layer will never stop.
