Braintrust Documentation — Evals, Tracing and Observability for AI Products
by Braintrust
The most complete vendor docs for putting LLM evals into CI, with SDKs in six languages.
Overview
The docs are structured around a six-stage workflow rather than an API surface. Instrument covers tracing LLM calls and surrounding application logic, advanced tracing patterns, capturing user feedback and logging attachments. Observe covers viewing, filtering and searching logs, examining traces, Topics (automatic clustering of production behaviour, with pages on enabling it, reviewing insights, acting on findings and custom facets), dashboards and alerts. Annotate covers human review with multiple reviewers, labels and corrections, custom views, and datasets — creating them, dataset pipelines, using them in evaluations and tracking their performance. Evaluate is the core: writing prompts, writing scorers and classifiers, the built-in Autoevals library, custom code scorers, LLM-as-a-judge scorers, playgrounds, running experiments in code, in the UI and in CI/CD, comparing experiments, interpreting results, evaluating reasoning models, remote evals in sandboxes and a best-practices page. Deploy covers deploying prompts and functions, the Braintrust Gateway, supported models, environments and streaming. Administration covers access control, audit logs, data retention, bring-your-own-cloud and full self-hosting including architecture, networking, scaling and upgrade paths. Reference adds a glossary, the bt CLI (bt eval, bt sql, bt sync, bt datasets, bt topics), a SQL dialect for querying logs and experiments, and a full REST API. SDK sections exist for TypeScript, Python, Go, Java, Ruby and C#, alongside integration pages for 20+ agent frameworks and OpenTelemetry, plus a cookbook of 40+ runnable recipes.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Freemium
- Duration
- ~30 min for the tracing and evaluation quickstarts; ~8 hours to work the full workflow
- Provider
- Braintrust
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Express an eval as a dataset, a task function and a list of scorers
- ✓Write LLM-as-a-judge scorers and custom code scorers, and when each misleads you
- ✓Trace LLM calls and surrounding application logic into inspectable, comparable spans
- ✓Run experiments from code, the UI or CI/CD and diff them for regressions
- ✓Curate production traces into datasets and pipelines using human review workflows
- ✓Query logs and experiments with the SQL dialect and the bt command-line tool
- ✓Score production traces online and wire up dashboards and alerting thresholds
- ✓Evaluate agents and reasoning models, including remote evals inside sandboxes
- ✓Self-host or run bring-your-own-cloud when trace data cannot leave your account
Highlights
- •First-class SDKs in six languages — TypeScript, Python, Go, Java, Ruby and C# — where most eval platforms ship two
- •Documented integrations for 20+ agent frameworks including LangGraph, CrewAI, AutoGen, Pydantic AI, Mastra, Google ADK and the OpenAI and Claude Agent SDKs
- •A dedicated 'Evaluating agents' best-practices page plus a cookbook of 40+ runnable recipes covering agent behaviour, production observability and video/audio QA
- •Self-hosting and BYOC are documented in full — architecture, networking, scaling, telemetry and versioned upgrade guides — not hidden behind a sales call
- •A SQL dialect and a bt CLI mean evals and log analysis fit into existing engineering workflow rather than a dashboard
- •Honest caveat: this is vendor documentation. The eval concepts transfer anywhere; the product surface does not, and the quickstarts need an account
Who It’s For
Best For
- ✓AI engineers who want eval gates in CI before a prompt or model change ships
- ✓Teams debugging why an agent behaves differently in production than in testing
- ✓PMs and domain experts running human review and labelling alongside engineers
- ✓Anyone comparing eval platforms against Langfuse, LangSmith, Opik or Phoenix
Prerequisites
- •Working Python or TypeScript and an LLM application you already call in code
- •A basic grasp of what an eval dataset and a scorer are and why regressions matter
- •A free Braintrust account, which the tracing and evaluation quickstarts assume
FAQ
What is Braintrust Documentation — Evals, Tracing and Observability for AI Products?
Braintrust's documentation for engineers who need evals, tracing and production observability on an LLM or agent application. It teaches the experiment model — dataset, task function, scorers — then extends it to online scoring, human review, dataset curation and CI gating, with SDK quickstarts in TypeScript, Python, Go, Java, Ruby and C# and integrations for most agent frameworks in use today.
Is Braintrust Documentation — Evals, Tracing and Observability for AI Products free?
Braintrust Documentation — Evals, Tracing and Observability for AI Products offers free content, with paid options for certificates or premium features.
What level is Braintrust Documentation — Evals, Tracing and Observability for AI Products for?
Braintrust Documentation — Evals, Tracing and Observability for AI Products is aimed at a intermediate audience. Recommended background: Working Python or TypeScript and an LLM application you already call in code, A basic grasp of what an eval dataset and a scorer are and why regressions matter, A free Braintrust account, which the tracing and evaluation quickstarts assume.
How long does Braintrust Documentation — Evals, Tracing and Observability for AI Products take?
Expect roughly ~30 min for the tracing and evaluation quickstarts; ~8 hours to work the full workflow. Most learners work through it at their own pace.
What will I learn from Braintrust Documentation — Evals, Tracing and Observability for AI Products?
You'll learn: Express an eval as a dataset, a task function and a list of scorers; Write LLM-as-a-judge scorers and custom code scorers, and when each misleads you; Trace LLM calls and surrounding application logic into inspectable, comparable spans; Run experiments from code, the UI or CI/CD and diff them for regressions; Curate production traces into datasets and pipelines using human review workflows; Query logs and experiments with the SQL dialect and the bt command-line tool; Score production traces online and wire up dashboards and alerting thresholds; Evaluate agents and reasoning models, including remote evals inside sandboxes; Self-host or run bring-your-own-cloud when trace data cannot leave your account.
Topics
Sources
This page was written from 3 sources, 1 on domains other than braintrust.dev.