IIya explained everyone's misunderstanding of some of his views: He believes that the current scaling route can make progress without leading to stagnation, but based on the current route, some things will be lost.
This makes people even more confused.
The following is my opinion: A.
Although Scaling Law predicts that model performance will improve as the amount of calculation and data increases, in actual application experience, this improvement is facing the challenge of diminishing marginal effects: Fitting rather than understanding: Existing LLM is essentially a "Next Token Predictor" based on probability.
Increasing the number of parameters and training data can indeed make its probability distribution more accurate, but this is still fitting the existing text statistical laws rather than truly understanding the logic behind it.
Data Wall: High-quality human text data has almost been exhausted.
If you continue to rely on stacked data (Linear Growth), the model will soon face the problem of "garbage in, garbage out" or model collapse caused by synthetic data.
Computing efficiency bottleneck: In order to obtain a 10% performance improvement, it is often necessary to invest 10 times the computing power.
This linear or even sublinear input-output ratio is commercially and energy unsustainable.
B.
Simply making the current model bigger cannot solve the following essential flaws: Lack of physical world (grounding/embodiment): LLM only learns the world through text.
It knows that the word "apple" and "red" often appear together, but it has no visual and tactile experience and cannot understand gravity, friction or object permanence.
The nature of the hallucination: The hallucination is not because the model is not large enough, but because the training goal of LLM is "reasonableness" rather than "authenticity".
This mechanism determines that it cannot verify facts on its own, which is unsolvable in the extension of Scaling Law.
Lack of causal inference: Statistical correlation does not equal causation.
Current AI is good at induction, but extremely lacking in deductive and counterfactual reasoning.
C.
Why might a “world model” be the answer?
As Yann LeCun (Turing Award winner and chief AI scientist at Meta) contends, the path to AGI is not just LLMs, but world models.
From "predicting the next word" to "predicting the state of the world": The world model does not focus on what the next word will be, but on what changes will occur to the environment as a result of actions.
It builds an internal simulator capable of understanding physical laws and cause-and-effect relationships.
Planning and reasoning: AI with a model of the world can preview multiple possible futures in an internal "sandbox" and evaluate the consequences before taking action.
This gives AI real planning capabilities (for example: to make coffee, you need to find water first and then beans), rather than just generating steps based on language sense.
A qualitative change in data efficiency: Humans don’t need to read trillions of words to learn not to drive into a wall because we build models of the physical world through observation.
World models based on architectures such as JEPA (Joint Embedding Predictive Architecture) aim to learn abstract concepts with extremely low data samples by observing videos and real environments (rather than just text).