Three major challenges of llm long context (million level): 1. Attention mechanism defe...

Three major challenges of llm long context (million level):

Attention mechanism defects: The "Lost in the Middle" phenomenon of the Transformer architecture causes the model's attention to the middle paragraph to drop by 40%-60%. 2 Cost barrier: In the context of processing 128k tokens, the API cost of GPT-4 is
2 times that of the RAG solution. 3 Risk of hallucination: Noise interference increases the probability of error generation by 15%-30% (Google DeepMind, 2024). These problems are basically very obvious after 50 rounds of conversations. Whether it is a common chat scenario or an AI coding scenario, similar problems will cause the experience, cost, and performance to deteriorate exponentially.

Visual summary

Article argument map

Generated from the post's content graph

Mermaid outline

flowchart LR
  format-long_post["longform note"]
  topic-memory["memory"]
  topic-retrieval["retrieval"]
  capability-technical-writing["technical writing"]
  format-long_post -->|covers| topic-memory
  format-long_post -->|covers| topic-retrieval
  format-long_post -->|signals| capability-technical-writing

Visual structure

Essay structure map

Built from summary and key paragraph positions

Mermaid outline

flowchart LR
  thesis["Three major challenges of llm long context (million level): 1. Attention mechanism defects: The \"Lost in th..."]
  signal["Three major challenges of llm long context (million level):"]
  operator["1. Attention mechanism defects: The \"Lost in the Middle\" phenomenon of the Transformer architecture causes..."]
  implication["3.2 times that of the RAG solution. 3 Risk of hallucination: Noise interference increases the probability o..."]
  thesis -->|frames| signal
  signal -->|develops| operator
  operator -->|lands in| implication

Source: View the original post