6-11:
- Evals: LLM-as-judge + human evals Official Document | OpenAI Evaluation Best Practices — Learn eval rubric, the applicable conditions for LLM-as-judge, the role of human annotation, and the alignment of judge with human annotation. Link: Official Cookbook | Getting Started with OpenAI Evals — Getting Started What eval is and how to measure LLM app output quality with a test set. Link: Official Cookbook | Eval Driven System Design — Learn the "define eval, then iterate system" method from prototype to production. Link: Article | Hamel Husain: Using LLM-as-a-Judge For Evaluation — Learn how to write judge prompt, rubric, golden dataset, error analysis. Link: PDF | Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Learning LLM judge's bias: position bias, verbosity bias, self-enhancement bias, and why human preference calibration is needed. Link: YouTube | What Are AI Evaluations? Quickstart videos of type Human, Programmatic, and LLM-as-Judge — eval. Link:
- Cost attribution per feature, rather than just looking at the per model documentation | LangSmith Cost Tracking — Learn how to record token usage, automatically calculate costs, and aggregate costs to the trace level. Link: Documentation | LangSmith Metadata & Tags — Learn how to tag feature_name, user_id, tenant_id, environment, experiment_id with metadata/tags. Link: Documentation | W&B Weave Track Costs — Learn automatic cost tracking and custom cost tracking, especially for self-hosted or fine-tuned models . Link: Documentation | Braintrust Instrumentation — Learn the trace/span structure: task span, LLM span, and how each LLM span records models, messages, parameters, token usage, and cost. Link: Article | Traceloop: How to Track LLM Token Usage and Cost Per User — Establish cost attribution awareness: token/cost attribution by user, feature, team. Link:
- Agent guardrails & loop budgets Official Document | OpenAI Agents SDK: Running Agents — Learn agent loop: model call → inspect output → tool call/handoff/final answer → continue. Link: Official Documentation | OpenAI Agents SDK: max_turns — Learn turn budget; Exceeding max turns triggers MaxTurnsExceeded. Link: Official Documentation | OpenAI Agents SDK: Guardrails and Human Review — Learn input/output/tool guardrails and when to pause, continue, stop, or hand over to manual review Batch. Link: Document | LangChain Agents/Iteration Limit — The learning agent must have a stop condition: final output or iteration limit. Link: Documentation | CrewAI Agents max_iter — Learn how another framework uses max_iter to control agent inference loops. Link: YouTube | Must Haves For Agents in Production — Supplements production agent boundaries, tools, safety, observation and evaluation awareness. Link:
- LLM observability as a first-class discipline Documentation | LangSmith Observability — LLM app observability for learning from individual traces to production metrics. Link: Documentation | Arize Phoenix Tracing Tutorial — Learn how to capture LLM call, tool execution, retrieval, generation, inputs, outputs, latency, token usage. Link: Documentation | W&B Weave: What is Weave? — Learn how to combine observability with evaluation to continually improve the LLM app. Link: Documentation | OpenTelemetry - Learn the basics of universal observability: traces, metrics, logs, vendor-neutral instrumentation. Link: YouTube | Practical AI-Enabled Observability for Agents and LLMs — agent/LLM observability. Link: YouTube | How to Find the Agent Failures Your Evals Miss — Learn how to find online failures with telemetry outside of evals. Link:
- Model routing & graceful fallback logic Documentation | LiteLLM Fallbacks — Learn provider/model fallback: fallback to other model groups in order after primary failure. Link: Document | LiteLLM Routing & Load Balancing — Learn routing, load balancing, budget routing, health-check driven routing, timeouts. Link: Documentation | OpenRouter Model Fallbacks - Learn the actual configuration of multi-model fallback and provider routing. Link: Documentation | Portkey Fallbacks — Learning AI Gateway Layers like Why implement provider/model fallback with a priority list. Link: Article | Portkey: Retries, fallbacks, and circuit breakers in LLM apps — Learn why retry/fallback is not enough and circuit breakers are needed. Link: Documentation | TensorZero Retries & Fallbacks — See more complete gateway, variant, routing fallback designs. Link: YouTube | LLM Gateway Tutorial: Routing, Guardrails, Caching & Agent Workflows — routing, guardrails, caching, agent budget in one place Understand in a gateway architecture. Link:
- Knowing when to fine-tune vs. in-context learning Official Documentation | OpenAI Model Optimization — Learn the applicable scenarios for fine-tuning and how to choose between prompt engineering, rag, and fine-tuning. Link: Official Documentation | OpenAI Optimizing LLM Accuracy — Establish Judgment Matrix: prompt engineering, rag, fine-tuning are not linear upgrades, but different levers to solve different problems. Link: Article | IBM: rag vs. Fine-tuning vs. Prompt Engineering — Understanding rag is mainly used for knowledge enhancement and relevance, fine-tuning is used to make the model more stable on specific tasks/areas. Link: PDF | QLoRA: Efficient Finetuning of Quantized LLMs — QLoRA is the engineering entry point that must be read if you are really ready to fine-tune. Link: YouTube | Choosing Between rag, In-Context Learning, and Fine-Tuning — Quickly establish a selection framework. Link: YouTube | rag vs. Fine Tuning — Understand the difference between rag and fine-tuning from a product/application perspective. Link: