How is the Notion AI team’s time allocated? ❌ 90% writing prompts (Prompting), 10% changing code. ✅ 10% writing prompts, 90% doing evaluations (Evals). Sarah, the head of Notion AI, said something very penetrating: "Our core IP is not prompts, but how we evaluate these prompts." If you are still obsessed with the "prompt word project", you may have gone astray. How does Notion make AI available to hundreds of millions of users? The breakdown is as follows: It is easy to make an AI demo because you can only show the time when it performs well. But the most difficult thing about making AI products (especially products like Notion with 100M+ users) is "certainty". Core philosophy: Demo is not equal to Prod. Running it locally does not mean it can go online. Observability is the foundation. You have to know what AI is talking about in a production environment. Evals as IP. Your moat is not that prompt, but the thousands of test cases (Golden Datasets) you have accumulated. It’s impossible to see it all with the human eye. Notion’s strategy is LLM as a Judge (let the model be the judge). But there is a big pitfall here: ❌ Even the latest models such as Gemin3, Claude Opus
- 5, if you ask it "Is this answer good?", its standards are erratic. ✅ Strategy: “Atomize” judgment. Don't ask "is it good?", ask "does it contain Japanese?", "does it reference document A?". Dismantle subjective "good or bad" into objective "rules". Every 👎 (Thumbs down) clicked by a user is a gold mine. The way Notion handles it is awesome: they don't look at thumbs up (because that's noise), but they stick to thumbs down. They will extract the "original needs" of the users at that time and add these failure cases to the test set (Dataset). Before the next release, these failed cases must be run through. This is called: feeding success with failure and building a data flywheel. Notion didn’t exactly spin all the wheels themselves, they used a platform like Braintrust to manage Evals. Tool stack suggestions for developers: Playground: Quickly verify ideas (perceptual cognition). SDK/CI: Write the evaluation into the code and run it like a unit test (rational guarantee). Online Scoring: Score online conversations in real time to monitor quality. In the second half of AI products, the battle is not about whose model is smarter, but whose evaluation system is more rigorous. If you want to make it big, you must first learn how to "exam". #AI Product Manager #Notion #LLM #Braintrust #Engineering