FrameworksAgenticRAG

MLflow GenAI Documentation — Tracing, Evaluation and Prompt Optimization

by MLflow

IntermediateDocumentationFree~4-6 hours to work through the tracing and evaluation quickstarts; ongoing reference after that

Self-hosted, OpenTelemetry-native tracing and LLM-judge evaluation for agents — no eval SaaS required.

Start LearningAdded Aug 6, 2026 · Updated Aug 6, 2026

Overview

MLflow's GenAI documentation covers the LLM- and agent-facing half of the MLflow platform, which the project describes as the largest open source AI engineering platform for agents and LLMs. The docs are organised into a Getting Started track (Set Up MLflow Server, Start Tracing, Evaluate LLMs and Agents, Automatic Issue Detection) and four core components — Tracing (Observability), Evaluation & Monitoring, Prompt Management & Optimization, and AI Gateway — followed by Version Tracking, Packaging & Deployment, an MCP Registry and Agent Serving. The tracing section documents one-line automatic instrumentation for OpenAI, Anthropic, Google Gemini, LangChain, LangGraph, LlamaIndex, DSPy, PydanticAI, CrewAI, AutoGen, Amazon Bedrock, Mistral, Ollama and the Vercel AI SDK — 60+ frameworks across Python, TypeScript/JavaScript and Java — and explains that traces are fully OpenTelemetry-compatible and natively emit the GenAI Semantic Conventions, so instrumentation is not locked to a vendor backend; a slimmer mlflow-tracing package with a 95% smaller footprint is documented for production. The evaluation section walks through the three pieces of an MLflow evaluation — a dataset of inputs and expectations, one or more scorers, and a predict function — then covers built-in LLM judges including Correctness and the custom-criteria Guidelines scorer, writing your own scorers, evaluation datasets, human annotation, regression testing in CI/CD, automatic issue detection and production trace monitoring. MLflow is Apache-2.0 and self-hostable (27.4k GitHub stars, ~12.9k commits), with Databricks' Managed MLflow documented as the hosted alternative.

At a Glance

Topic
Frameworks
Level
Intermediate
Format
Documentation
Cost
Free
Duration
~4-6 hours to work through the tracing and evaluation quickstarts; ongoing reference after that
Provider
MLflow
Hands-on
Yes — code/exercises
Certificate
None

What You’ll Learn

  • ✓Instrument an existing LLM app or agent with one-line automatic tracing
  • ✓Read the MLflow trace data model: spans, attributes, inputs and outputs
  • ✓Build an evaluation from a dataset, a set of scorers and a predict function
  • ✓Write custom LLM-as-a-judge scorers alongside the built-in Correctness and Guidelines judges
  • ✓Wire agent evaluations into CI/CD as regression tests that gate a deploy
  • ✓Version, register and automatically optimize prompts with the prompt registry
  • ✓Route and cost-control multiple model providers behind the MLflow AI Gateway
  • ✓Export traces over OpenTelemetry GenAI semantic conventions to your own backend

Highlights

  • •Self-hosted and Apache-2.0 — unlike most LLM eval and tracing SaaS, the whole stack runs on infrastructure you control
  • •Traces are OpenTelemetry-native and emit the GenAI Semantic Conventions, so instrumentation survives a change of vendor
  • •One tool spans tracing, evaluation, prompt versioning and a model gateway instead of stitching four services together
  • •60+ auto-instrumented integrations with first-class Python, TypeScript/JavaScript and Java SDKs
  • •Builds on the experiment tracking and model registry many teams already run for classical ML, so GenAI work lands in an existing system rather than a new one

Who It’s For

Best For

  • ✓AI engineers who need agent observability without shipping traces to a third-party SaaS
  • ✓Teams already running MLflow for classical ML who are now adding LLM features
  • ✓Platform engineers building an internal eval and gateway layer across several LLM providers
  • ✓Anyone setting up prompt and agent regression tests before a production deploy

Prerequisites

  • •A working Python or TypeScript LLM application you can instrument
  • •Comfort with pip/uv, running a local server, and reading API reference docs
  • •Basic grasp of LLM evaluation concepts such as golden datasets and LLM-as-a-judge

FAQ

What is MLflow GenAI Documentation — Tracing, Evaluation and Prompt Optimization?

MLflow's GenAI documentation is the reference for using MLflow as a self-hosted observability and evaluation layer for LLM applications and agents. It is written for engineers who already have an app in production or heading there, and covers tracing, LLM-as-a-judge evaluation, prompt versioning and a multi-provider gateway. Work through it and you can instrument an agent, build a golden dataset, score it with judges, and gate deploys on regression tests — all on infrastructure you control.

Is MLflow GenAI Documentation — Tracing, Evaluation and Prompt Optimization free?

MLflow GenAI Documentation — Tracing, Evaluation and Prompt Optimization is free to access.

What level is MLflow GenAI Documentation — Tracing, Evaluation and Prompt Optimization for?

MLflow GenAI Documentation — Tracing, Evaluation and Prompt Optimization is aimed at a intermediate audience. Recommended background: A working Python or TypeScript LLM application you can instrument, Comfort with pip/uv, running a local server, and reading API reference docs, Basic grasp of LLM evaluation concepts such as golden datasets and LLM-as-a-judge.

How long does MLflow GenAI Documentation — Tracing, Evaluation and Prompt Optimization take?

Expect roughly ~4-6 hours to work through the tracing and evaluation quickstarts; ongoing reference after that. Most learners work through it at their own pace.

What will I learn from MLflow GenAI Documentation — Tracing, Evaluation and Prompt Optimization?

You'll learn: Instrument an existing LLM app or agent with one-line automatic tracing; Read the MLflow trace data model: spans, attributes, inputs and outputs; Build an evaluation from a dataset, a set of scorers and a predict function; Write custom LLM-as-a-judge scorers alongside the built-in Correctness and Guidelines judges; Wire agent evaluations into CI/CD as regression tests that gate a deploy; Version, register and automatically optimize prompts with the prompt registry; Route and cost-control multiple model providers behind the MLflow AI Gateway; Export traces over OpenTelemetry GenAI semantic conventions to your own backend.

Topics

mlflowllm observabilityagent evaluationtracingopentelemetryllm-as-a-judge

Sources

This page was written from 4 sources, 1 on domains other than mlflow.org.

  1. 1.mlflow.org — genaivendor
  2. 2.mlflow.org — tracingvendor
  3. 3.mlflow.org — eval monitorvendor
  4. 4.github.com — mlflow