LangSmith Documentation
by LangChain
The reference for tracing, evaluating and monitoring LLM apps — LangChain or not.
Overview
LangSmith is LangChain's observability and evaluation platform for LLM applications, and its documentation is the reference an engineer actually works from when moving an agent out of a notebook. The docs split into two halves that mirror the product. The observability half defines the data model: a run is one unit of work such as an LLM call, a prompt format or a retrieval step; a trace groups the runs from a single request, capped at 25,000 runs; threads chain traces into multi-turn conversations via a thread_id metadata key; trajectories flatten an agent's path into an ordered, de-duplicated message list; and projects contain everything for one service. Instrumentation is documented several ways — automatic integrations for LangChain, LangGraph, OpenAI, Anthropic and CrewAI, the @traceable decorator, a trace context manager, and the lower-level RunTree API — plus OpenTelemetry export to other backends. The evaluation half covers datasets of input and reference-output examples, four evaluator families (LLM-as-judge, deterministic code evaluators, human review, and pairwise comparison), experiments that record outputs, evaluator scores and traces for every example, and the offline-versus-online split between pre-deployment regression testing and monitoring live production traffic. Annotation queues with custom rubrics, feedback scores, tags, metadata, dashboards, alerts and rule-driven automations round it out. Cloud, hybrid and self-hosted deployment are all documented, and traces are retained 180 days unless promoted into a dataset.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Freemium
- Duration
- Reference docs; ~2-3 hours to work through the tracing and evaluation guides
- Provider
- LangChain
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Instrument an application with @traceable, context managers or the RunTree API
- ✓Model your telemetry correctly as runs, traces, threads and projects
- ✓Trace non-LangChain stacks including OpenAI, Anthropic, CrewAI and Pydantic AI
- ✓Build evaluation datasets from input and reference-output example pairs
- ✓Score outputs with LLM-as-judge, code, human and pairwise evaluators
- ✓Separate offline regression testing from online production quality monitoring
- ✓Collect structured human feedback through annotation queues with custom rubrics
- ✓Export traces via OpenTelemetry to an alternative observability backend
Highlights
- •Framework-agnostic in practice, not just in marketing — documented integrations span OpenAI, Anthropic, CrewAI and Pydantic AI
- •Dedicated conceptual pages for the data model, so you design your tracing rather than guess at it
- •Evaluation is a first-class half of the docs, not an afterthought bolted onto tracing
- •Documents cloud, hybrid and self-hosted deployment for teams that cannot ship traces off-premises
- •States the awkward operational details plainly, including the 180-day trace retention window
- •Free Developer tier covers 5,000 base traces a month, enough to evaluate it properly
Who It’s For
Best For
- ✓Engineers taking a first LLM agent from prototype to production
- ✓Teams building regression suites so prompt changes stop breaking silently
- ✓Platform engineers choosing an LLM observability and evaluation stack
- ✓LangChain and LangGraph users who want the native tracing path
Prerequisites
- •Working knowledge of Python or TypeScript
- •An LLM application already calling a model provider
- •A LangSmith account — tracing requires the hosted or self-hosted service
FAQ
What is LangSmith Documentation?
The official reference for LangChain's LLM observability and evaluation platform, covering tracing, datasets, evaluators, annotation queues, dashboards and alerts. It is framework-agnostic in practice: it instruments plain OpenAI or Anthropic SDK calls, Pydantic AI and CrewAI as readily as LangChain itself. Work through it and you can trace an agent end to end, build a regression dataset, score runs with LLM-as-judge and human review, and monitor production quality.
Is LangSmith Documentation free?
LangSmith Documentation offers free content, with paid options for certificates or premium features.
What level is LangSmith Documentation for?
LangSmith Documentation is aimed at a intermediate audience. Recommended background: Working knowledge of Python or TypeScript, An LLM application already calling a model provider, A LangSmith account — tracing requires the hosted or self-hosted service.
How long does LangSmith Documentation take?
Expect roughly Reference docs; ~2-3 hours to work through the tracing and evaluation guides. Most learners work through it at their own pace.
What will I learn from LangSmith Documentation?
You'll learn: Instrument an application with @traceable, context managers or the RunTree API; Model your telemetry correctly as runs, traces, threads and projects; Trace non-LangChain stacks including OpenAI, Anthropic, CrewAI and Pydantic AI; Build evaluation datasets from input and reference-output example pairs; Score outputs with LLM-as-judge, code, human and pairwise evaluators; Separate offline regression testing from online production quality monitoring; Collect structured human feedback through annotation queues with custom rubrics; Export traces via OpenTelemetry to an alternative observability backend.
Topics
Sources
This page was written from 4 sources, 2 on domains other than docs.langchain.com.