This direction is interesting: Skill +RAG The core pain point of the traditional search-enhanced generation (RAG) system is the persistent failure that cannot be solved after repeated searches. The paper found that most of these failures are not due to the lack of relevant information in the knowledge base, but the "alignment deviation" between the query and the evidence space (for example, the colloquial question does not match the knowledge base index, multi-hop questions are not disassembled, resulting in information that cannot be associated, etc.). Previous adaptive RAG solutions mostly used the binary judgment of "whether retrieval is needed", which was unable to diagnose the specific cause of the failure and could only blindly retry, resulting in low efficiency and limited effect. Core method: Skill-RAG framework This framework consists of two core modules, forming a closed-loop process of "detection-diagnosis-repair":
- Lightweight hidden state probe (Hidden-State Prober) ◦ Working principle: By detecting the hidden state of the large language model (LLM), it determines whether a retrieval failure occurs at two key nodes:
- After the initial retrieval, determine whether the existing evidence is sufficient to support the answer;
- Before generating the answer for the first time, determine whether the model is unable to generate a reliable answer due to information bias. ◦ Advantages: No additional training of large classifiers is required, it is lightweight and efficient, and can be directly embedded into the existing RAG process.
- Skill Router based on prompt words (Skill Router) When the probe detects a failure, the router will diagnose the cause of the failure and select the corresponding solution from 4 targeted "retrieval skills" instead of simply retrying: Skill type Applicable scenarios Function Query rewriting Mismatch in expressions such as colloquial questions, abbreviations, pronoun references, etc. Rewrite the query into a more standard form that matches the knowledge base Problem decomposition Multi-hop reasoning, complex logic problems Decompose complex problems into multiple sub-problems, and perform step-by-step retrieval and solution Evidence focus The search results are broad and the information is redundant. Key fragments that are strongly related to the problem are screened from massive evidence. Additional search. Extreme situations that cannot be solved by the first three skills. Trigger supplementary searches to repair deviations. Experimental results and core findings
- Significant performance improvement: In multiple benchmark tests such as open domain question answering and complex reasoning, Skill-RAG has significantly improved accuracy on "hard samples" compared to traditional RAG and adaptive RAG, especially on out-of-distribution (OOD) data sets, with stronger generalization capabilities.
- Structural characteristics of the failure state space: Analysis shows that the failure scenarios corresponding to these four skills do not exist in isolation, but are distributed in independent areas in the failure state space. This shows that "query-evidence alignment bias" is not a single problem, but a multi-type structured problem, and also verifies the necessity of skill-based routing. Value:
- The RAG paradigm of "failure state awareness" is proposed for the first time, breaking through the limitations of traditional binary judgments and achieving retrieval enhancements such as accurate diagnosis and on-demand repair.
- The proposed lightweight probe + skill routing solution does not require large-scale model training, can be directly adapted to existing LLM and RAG systems, and has low implementation costs.
- Reveals the structural nature of RAG failure and provides a new direction for subsequent fine-grained and interpretable retrieval enhancement research. Limitations and future directions: The paper points out that there is still room for optimization in the current solution: • The diagnostic accuracy of skill routing depends on the prompt word design, and misjudgments may occur in complex scenarios; • The scalability of skill needs to be improved. Currently it only covers 4 common failure modes, and more skills can be expanded in the future; • The adaptability of LLMs of different sizes needs further verification.