Source Original video: What does the next training paradigm look like?
Channel: Dwarkesh Patel Link: https://www.youtube.com/watch?v=20p5-kQXF_Q Start In this video, Dwarkesh Patel puts the issue of "next generation training paradigm" on the table.
All labs are making the same bet: training AI to complete massive tasks in thousands of verifiable RL environments can approach AGI.
Optimists believe that "flaws" such as low data efficiency and lack of continuous learning can be overcome by relying on scale.
But Dwarkesh didn’t stop at slogans.
He asked an underestimated question: Why is computer use progressing so much slower than coding and math?
The answer is that it is not enough to be "verifiable", but also to be "able to write questions in batches" - to be able to repeatedly run a large number of parallel rollouts from a certain starting point.
This clue leads to the real bottleneck of the training paradigm: sample efficiency, and how to write back the weights learned during deployment.
For engineers and product managers, the value of this issue is in judging cadence.
RLVR is still reaping dividends, but the dividends will be diminishing at the margin; continuous learning is the next real wall.
Whoever figures out "how to write back weights on-the-job learning" first will get the next order of magnitude advantage.
Insight 01 | All laboratories are betting on the same bet.
The laboratories are betting on a very specific hypothesis: if AI is allowed to complete millions of verifiable tasks in thousands of different RL environments, AGI will basically become it.
this This kind of training will create a "problem-solving agent" that can advance open-ended tasks in the face of errors and ambiguities for weeks on end.
Optimists will go on to say that the training flaws we often talk about—data inefficiency, lack of continuous learning—can be flattened by scale.
Just like those "fundamental" research problems in NLP back then, they collectively collapsed after LLM had enough computing power.
In the last issue, Dwarkesh mentioned that the model sample efficiency is only one millionth that of humans.
Supporters will respond: That is just a matter of the training phase.
Training is a one-time cost that is spread over billions of sessions that the model goes through.
What really matters is in-session intelligence, generality, and sample efficiency, and this is clearly on the rise with RL training.
Anyone who has used these models to write code can feel it.
Insight 02｜Verifiability is not enough, you must also be able to "batch questions".
Before continuing to discuss this bet, Dwarkesh raised a seemingly off-topic, but actually very sharp question: Why is the progress of computer use so much slower than other fields?
It's obviously easy to verify - whether the Etsy goods have been delivered, whether the event venue has been reserved, whether taxes have been paid.
There are many reasons.
There is at least one piece of high-quality multi-modal data seen during model pre-training.
But he believes there is one that is severely underestimated, and it reveals the "canyon wall" that the river of AI progress is slowly chipping away at: a field that is not verifiable alone, but also needs to be highly grindable, that is, you can repeatedly run a large number of parallel rollouts from the same starting point in a deterministic, replayable simulator.
It's easy to practice coding: define a container, put a code warehouse with missing functions in it, and let a thousand agents each hold the same copy to solve it in parallel.
Computer use is not so straightforward - you can't have a thousand agents going through the Amazon checkout process over and over again, and Andy Jassy will find your bot and ban you.
It is possible to clone Slack and Gmail, but currently this is a labor-intensive and non-scalable way.
Insight 03｜The sluggishness of computer use exposes the true boundaries of the training paradigm.
Dwarkesh incidentally pointed out a "killing two birds with one stone": when AI's own coding is strong enough to reconstruct these applications with high fidelity, computer use will speed up, and reconstructing the entire application itself is an excellent coding RL goal.
So computer use may be solved soon, but its current sluggishness actually tells us one thing - unless a reproducible training target can be created for a field, it is difficult for the model to make substantial progress.
The reason is that the model Extremely inefficient sample utilization during training.
For computer use, we may be able to fill the sample gap by building a bunch of deterministic simulators that can be flashed in batches.
But many of the skills we need simply don’t do this.
How to train an AI to start a business from scratch, win a lawsuit, make a day's money in the market, and help candidates win elections?
The rollout here needs to interact with the real world and cannot be replicated in the data center.
Outer-loop verification may take months or even years of real actions to trigger, and there is no way to slightly perturb model behavior in thousands of parallel rollouts to isolate "which step really works." Dealing with this non-reset, non-stationary environment is a recognized open problem in RL.
Insight 04｜Whether RLVR can be generalized is an empirical question.
The lab is betting that RLVR will generalize: if trained on enough containerized and reproducible environments, a very general agent will be obtained that can formulate and execute plans in a single session, quickly learn from new information, and even master new skills.
Throw such an infinitely RLVR AI into Texas politics in 1948, and its advice would be more accurate than LBJ’s.
Dwarkesh cautions that whether this holds true is a matter of experience.
If labs went from spending billions of dollars doing RL environments to spending a trillion dollars, would you get a fully human-like general intelligence within the context window?
Dario's words in the podcast give clues: the performance of the model decreases under long context because "the training context length and the service context length" are inconsistent.
Dwarkesh admits that he may over-interpret, but this sentence reads like he is saying - short-field RL training may not be able to generalize to long-field performance.
If it cannot even cross short to long, how can an agent be trained from a bunch of white-collar tasks and generalized to the point where it can start from scratch like Sam Walton in the real world?
Even if the AI becomes Henry Ford or Einstein after enough in-context experience, if it cannot write these learnings back into weights, everything will dissipate when the session ends.
Insight 05｜Learning during deployment is being wasted.
There is a very eye-catching waste here: about 30% to 50% of the laboratory's computing power is spent on inference, and this computing power currently does not contribute any output to improving the model.
This is worse than it sounds, because only the deployment phase exposes the most important bits of information the model needs to learn.
"In which organizations am I used?
What do they do with me?
What kinds of mistakes do I make in the real world?
"We have a talented graduate student who has never been allowed to intern, but we keep feeding him classroom cases in RL training environments.
Models have been widely deployed in the economy, participated in massive tasks, and have access to a large amount of domain- and organization-specific tacit knowledge, but they cannot use it - this is ridiculous.
The root of the problem is that continuous learning must go back to weights.
AI cannot digest what it learns from more and more users by piling up the KV cache.
That is not scalable, and it is not a human approach.
There is no clear distinction between parameters and activation in the human brain, and the skull will not keep expanding as you learn more things.
There is obviously compression when people learn, and this compression helps generalization and grokking.
Those autistic savant humans who can repeat tables of random numbers and meaningless syllables verbatim a few years later have exactly the fidelity of the model in-context - and such a large amount of information weakens their ability to understand abstractions and metaphors.
Continuous human learning is to chisel right intuition and overall view back to weight, rather than just talking about all observations.
Insight 06｜OPSD: Steaming the experience in conversation back to weight.
Once you enter the weight, You have to give up in-context sample efficiency because gradient updates are extremely inefficient.
All online learning models that have been successfully launched have millions of users learning the same thing.
Cursor Tab relies on the goal of predicting "which edits are accepted by users" and learns online on 400 million requests every day; currently there is no model that can learn different things online for different users - the data produced in a single session is enough for anthropology, but not enough to train a stronger AI.
But the significance of continuous learning lies precisely in the fact that the world is very complex, and every job, every company, and every problem is different.
The agent has to learn specific information in a specific deployment scenario, and these cannot be squeezed into a shared training run.
The so-called "on-the-job learning" refers to how everything in your organization fits together, how to cooperate with infrastructure and others to promote large projects, and what are the common failure modes.
Dwarkesh himself uses Mercury to process bills as an example: the invoice is sent to a fixed mailbox, and Mercury automatically downloads, scans, and extracts it.
Contractor name/address/amount/invoice number/due date, a payment draft to be reviewed is generated, and he only needs to check it.
These capabilities of "doing all the basic things well and putting them in one place" are exactly what the on-the-job scenario is like.
Sample efficiency and continuous learning are actually two sides of the same problem.
There is little data available on the job, and learning from it depends on sample efficiency.
Model in-context can do this using fast weights built on the spot using attention, but its memory scalability is extremely poor.
Therefore, there is a need for architectural innovation that can carry some kind of intermediate representation - sparse attention, KV cache compression, there are new papers every week.
But Dwarkesh judged that the architecture does not seem to be the real bottleneck of continuous learning.
The bottleneck is more likely to be the loss function: how to update weights based on the information learned in a session.
A hotly discussed technology here is on-policy self-distillation (OPSD).
The idea is to encourage the base model to make the same predictions as the "teacher model that has accumulated long conversation context" when solving real problems, and evaporate what is learned in the conversation back into weight.
It is better than RLVR in two ways: it does not require an outer ring to verify rewards, as long as the model can If you learn the right things in the context window, you can train the base model to align with the experienced teacher; and the supervision signal is much denser than simple RL, and it can be trained on the token-by-token probability difference between teacher and student, rather than projecting a single reward to the entire trajectory.
OPSD is also better than SFT for continuous learning.
The simplest SFT is to let the base model predict all tokens observed in the session, but this does not make sense as a learning goal - the key to your success lies in consolidating those few truly relevant insights.
RL does not suffer from this problem.
It is good at concentrating updates on the parts that are "really relevant to getting the result".
The updates are extremely sparse, which is critical for continuous learning, because you don't want to overwrite and forget other things known by the base model.
OPSD retains this property of RL: only extracting the knowledge necessary to achieve the same results as the teacher, rather than catapulting towards the teacher distribution like SFT.
Insight 07｜Dreaming: the fourth scaling axis OPSD It is a way to attack sample efficiency: squeeze signals from scarce real experience into a small and precise Accurate updates.
There is a more speculative path, which Dwarkesh calls "dreaming." If an AI can create a good enough simulation of reality in which it can rehearse new skills, try out different strategies, and reinforce what actually works, then it can experience orders of magnitude more simulation samples in the same wall-clock time.
There are precedents in history.
A few years after AlphaZero was released, a group of researchers trained EfficientZero to focus on data efficiency.
Give it and a human two hours each to play an Atari game they've never seen before, and it's likely to beat a novice human.
Does this mean it is more sample efficient than humans?
It depends on how it is measured - because EfficientZero has played dozens of simulation games in his mind while making every move in the real game.
In the future, LLM may also be able to practice endlessly in its own environment while consuming very little real data.
The difference is that simulating the entire world is much more difficult than simulating Go, so this path is more speculative.
If it is established, it will become the fourth scaling axis besides pretraining, RL, and inference-time compute.
It can be called test-time training or dreaming: the model spends computing power to write the RL environment and then trains it to rehearse the skills it really needs to use in the production of the user.
So you no longer press /compact (spending a little computing power to write summaries in Codex, Cursor, and Claude, giving you a simulacrum of continuous learning), but press /dream - burning a lot of computing power to build and train a "video game version of what the model sees in the real world." What continuous learning might look like in 2027 So what will continuous learning look like in 2027 or 2028?
Dwarkesh gives a scenario.
The agent produced by RLVR training can find directions in front of unfamiliar problems, try different strategies, and iterate when encountering obstacles.
This is the key asset that RLVR gives you: an AI that at least has the ability to start accumulating real experience and learn from it.
With it, you send it to work in the real world, even if the project falls outside the training distribution.
Assume that the effective context length at this time has been extended so that AI can collaborate with you for a whole week; at the end of the week, you give it a thumbs up or a negative review, and do a work review.
After you give a thumbs up, the base model distills back everything learned in this session, maybe using OPSD, using dreaming, using some technology that we are not aware of yet, or a combination of both.
The AI can therefore become stronger in adjacent areas outside the area it was explicitly trained by RLVR; in the next round, it will become stronger in adjacent areas that it has previously learned online.
In this way, the AI's skill and knowledge landscape can extend far beyond the verifiable domain in which it was trained before deployment.
Just as pretraining creates basic intelligence that is smart enough and adds enough RLVR to become a qualified agent; RLVR also creates an agent that is qualified enough to be widely deployed.
Once the training formula for continuous learning is in place, it can learn on the job from widespread deployment.
At this point, the main way for AI to become stronger is no longer the training received before release, but the experience accumulated by it being widely deployed in the economy and participating in massive tasks.
Every time you interact with the AI, it gets smarter—both because it learns from your past conversations with it and Because it has learned all interactions with other users around the world.
It’s scary, it’s exciting, and it’s completely different from how AI gets stronger now.
Judgment for engineers and product managers: RLVR is still reaping dividends, but it is necessary to start setting upper limit expectations for it.
Whether short-field training can be generalized to long-field training and whether white-collar tasks can be generalized to real-world entrepreneurship are inconclusive empirical questions.
Don't take "if you invest ten times more computing power, you will definitely get AGI" as the default premise when doing product planning to evaluate whether a new field can be captured by RL, ask two things first: verifiable and batch replayable.
Without the second one, it will be difficult to advance the model.
This explains why computer use is slow, and also helps you predict which Vertical categories (such as real transactions, political and business games) will not rely on RL to make breakthroughs in the short term.
Deploying data in other ways is the biggest gold mine that is wasted.
30 to 50% of the computing power is used in inference without returning to the model, which is the most counter-intuitive input-output gap.
Whoever makes "on-the-job learning write-back weights" into an engineerable pipeline first will master the next order of magnitude.
OPSD is currently the most worthy of follow-up in specific technical directions.
The real bottleneck in continuous learning is more likely to be the loss function rather than the architecture.
There are new architecture papers every week, but how to filter out a small number of signals that should be updated from a single session without covering existing knowledge is an unsolved problem.
The sparse update of RL is an advantage rather than a disadvantage here.
For products, "learning within a session" and "being able to write back weights" are two different selling points.
The former relies on context and fast weights, which is already available but has poor memory scalability; the latter has not yet become a reality.
Cooked plan.
In the short term, it is more stable to focus on in-context learning + artificial distillation for product experience.
In the long term, we need to keep an eye on the product form around the fourth axis of 2027, such as OPSD and "Dream", which will shift from "the more the model is trained, the stronger it becomes" to "the more it is used." The network effect of making the model smarter with each interaction will reshape the distribution and moat logic - the data flywheel moves from the pre-training stage to the deployment stage, and whoever can compliantly aggregate and write back multi-user experience will win.
This is exciting, but you also need to think clearly about privacy and belonging early on.