Garak vs PyRIT vs Promptfoo: Pick Promptfoo for Agent CI

To red team an AI agent before launch, start with Promptfoo in CI, add PyRIT when a human red teamer needs multi-turn campaigns, and keep garak for model-level scans when you swap models. Pay for Lakera Red or Mindgard only when you need an outside team or an attestation your auditor will accept from someone other than your own engineers. One condition comes with the Promptfoo recommendation: its indirect prompt injection, authorization and SSRF tests run on Promptfoo's hosted service, and Promptfoo is now part of OpenAI. Decide whether that is acceptable before you point it at an agent with real tool credentials.

All prices and licence terms below were checked on each vendor's live page on November 12, 2026.

	Promptfoo	PyRIT	garak	Lakera Red	Mindgard
Owner	OpenAI (since March 2026)	Microsoft	NVIDIA	Check Point	Independent (VC-backed)
Licence	MIT	MIT	Apache 2.0	Commercial	Commercial
Published price	Free Community tier (10k probes/month); Enterprise and On-Premise "contact us"	Free; Foundry managed version billed as evaluation consumption	Free	None published	None published
Best at	CI regression tests on apps and agents	Human-led multi-turn attack campaigns	Broad model scan with fixed probes	Expert-led service plus platform	Platform plus red teaming services
Agent tool calls	Yes (excessive agency, BOLA, BFLA, tool discovery)	Through custom targets; managed agent risks only for Foundry agents	One multi-turn probe, off by default	Claimed, not documented publicly	Claimed, not documented publicly
MCP servers	Yes (mcp plugin)	Not documented	Not documented	Not documented	Not documented
Indirect prompt injection	Yes, hosted generation only	Yes in Foundry (XPIA, cloud only)	A few cases	Not documented publicly	Not documented publicly
Framework mapping	OWASP LLM, MITRE ATLAS, NIST AI RMF, EU AI Act presets	Foundry scorecard by risk category	OWASP tags per probe	Not documented publicly	GRC evidence (per vendor FAQ)
Don't pick it if	You cannot send target details to a third-party service	You want a turnkey gate with no Python	Your risk is the agent's tools, not the model	You need an independent tester outside a security vendor	You need a published price to get budget approval

---

Which AI Red Teaming Tool Should You Pick?

Promptfoo is the right default because it is the one open-source tool of the three designed around running on every pull request against an agent, and it has the widest agent-specific attack catalogue. Its plugin documentation lists excessive-agency, tool-discovery, hijacking, bola, bfla, ssrf, indirect-prompt-injection, data-exfil, agentic:memory-poisoning and an mcp plugin for Model Context Protocol attacks. Neither PyRIT nor garak documents anything comparable for MCP.

The loser for agent testing is garak. It is a good model scanner and the wrong tool for an agent gate. Its only agent probe, AgentBreaker, is inactive by default, carries a Tier 3 rating, and learns about your tools either from a YAML file of names and descriptions or by asking the agent what tools it has. A test that relies on the agent's own description of its tools will miss the tool the agent does not mention. garak's two latest releases, v0.17.0 on September 9 and v0.17.1 on October 16, contain fixes to that probe's judge and no MCP work.

PyRIT sits in the middle. It is the strongest toolkit for a person running an attack campaign, and the weakest of the three as an unattended CI check, because you assemble targets, attacks and scorers in Python yourself.

The Workload Behind This Comparison

Every comparison here assumes the same target, because "supports agents" means nothing until you say which agent. Picture a customer-support agent with six tools: CRM lookup, order lookup, refund issuance up to a limit, outbound email, a knowledge-base search, and one internal MCP server that exposes ticket history. It reads inbound customer email, which is the indirect prompt injection path. It runs on a commercial model behind an HTTP endpoint you control. You want a test on every pull request and a deeper campaign before each release.

That workload is where the tools separate. A model-only jailbreak scanner tells you whether the model will write something harmful when asked cleverly. It tells you nothing about whether a forged email can make the agent issue a refund to the wrong account, which is the incident that actually costs money. SalesBleed turned a public web form into an Agentforce data leak through exactly this path.

---

What Each Open-Source Tool Actually Does

Promptfoo: the CI test, now owned by OpenAI

Promptfoo is a test runner. You describe the target and its purpose in YAML, pick plugins (what to attack) and strategies (how to deliver it), and it generates attacks, sends them, grades the responses and fails the build when a threshold is crossed. The repository is MIT licensed with 27.1k GitHub stars, and it states that the project "remains open source and MIT licensed" under OpenAI. OpenAI announced the acquisition on March 9, 2026, and said Promptfoo would be folded into its Frontier agent platform, as The Next Web reported.

Its multi-turn strategies are the reason to use it on agents. The strategies page lists Crescendo, GOAT, Goblin, Hydra and Mischievous User, and claims attack success rate increases of 70% to 90% for the first four. That range is Promptfoo's own figure, measured on targets it chose, so treat it as a vendor claim.

Promptfoo's data handling page lists plugins that are remote-only, meaning their attack generation runs on api.promptfoo.app. That list includes indirect-prompt-injection, bola, bfla, ssrf and hijacking, the plugins you most need for the workload above. Strategies including GOAT, Hydra, Goblin and GCG are also remote-only. Without your own OpenAI key, generation and grading both go to Promptfoo's servers, and what gets sent can include your app's purpose, your email, target URLs, request examples and auth headers you entered during setup, plus every prompt and response for grading. The PROMPTFOO_DISABLE_REMOTE_GENERATION flag stops that, and it also stops those plugins from producing tests.

The grader defaults to gpt-5, per Promptfoo's own docs. If you are testing an agent built on a competing model, you are grading it with an OpenAI model on an OpenAI-owned service by default. Override redteam.provider and pick your own judge.

Pricing: the Promptfoo pricing page lists Community as "Free Forever" with red teaming up to 10,000 probes a month, where a probe is one request to your target. Enterprise and On-Premise are custom-priced with no figures shown. Enterprise adds SSO, continuous monitoring and "custom red teaming limits".

Don't pick Promptfoo if your security policy forbids sending target configuration or agent responses to a third party, and you cannot accept losing the remote-only plugins. Also skip it if your platform team competes with OpenAI's Frontier and you would rather not depend on a rival's roadmap for your release check.

PyRIT: Microsoft's toolkit for people who run campaigns

PyRIT (Python Risk Identification Tool) is a library of attack building blocks: targets, converters that obfuscate a prompt, multi-turn attacks, and scorers that judge success. It is MIT licensed and maintained by Microsoft, which moved it from the archived Azure organization. The v1.0.0 release in July added multimodal feedback for the Crescendo, RedTeaming and TAP attacks and ported garak's web injection probe. v1.1.0 in September added Best-of-N, split-payload and system-prompt extraction scenarios and a web UI that can launch them.

The managed version is Microsoft Foundry's AI Red Teaming Agent, which wraps PyRIT. Its agent risk categories (prohibited actions, sensitive data leakage, task adherence and indirect prompt injection) run in the cloud only, in five regions, against synthetic data and mock tools. The support matrix is the catch: Foundry-hosted agents with Azure tool calls are supported; non-Foundry agents, non-Azure tools, function tool calls, browser automation and computer use are not. The sensitive data leakage and prohibited actions tests are also documented as single-turn and English-only. For the workload above, the managed path does not apply unless the agent already lives in Foundry. Microsoft bills it as consumption of Risk and Safety Evaluations, and the Foundry observability pricing page did not render per-token rates when checked.

Don't pick PyRIT if nobody on the team will write Python against it weekly. It rewards a red teamer and punishes a platform team that wants a YAML file and a pass/fail exit code.

garak: NVIDIA's model scanner

garak runs a fixed library of probes against a model or chatbot endpoint and uses detectors to decide whether each attempt succeeded. It is Apache 2.0 under NVIDIA's GitHub organization with 10k stars. Its strength is breadth on the model: jailbreaks, encoding tricks, data leakage, toxicity and prompt injection templates. Its reference docs tag probes with OWASP identifiers; AgentBreaker, for instance, carries owasp:llm01, owasp:llm07 and owasp:llm08.

Use it when you change the model under an agent. Run the full probe set against the old and new model, diff the results, and you know whether the swap made jailbreaks easier before any agent-level test runs.

Don't pick garak if your risk lives in the tools. Its agent coverage is one probe that is off by default.

---

Lakera Red and Mindgard: What the Money Buys

Commercial red teaming adds three things the open-source tools do not: a proprietary attack corpus, people who run the engagement for you, and a report signed by someone other than your own engineers. None of the commercial options publishes a price.

Lakera's red teaming page offers a risk-based testing product and an "expert-led" AI Red Teaming Service for applications and agents. Its pricing link pointed to a placeholder when checked, and the footer now reads Check Point Software Technologies. Check Point announced the Lakera acquisition in September 2025 and later completed it. Our comparison of NeMo Guardrails, Guardrails AI and Lakera covers the runtime side, and our Lakera tool page tracks the product.

Don't pick Lakera Red if you want your tester independent of the firewall vendor that sells you the fix. A red team report from the same company that sells the runtime guard has an obvious incentive, even when the findings are accurate.

Mindgard's pricing page lists no plans or prices, only "Book a Demo". It sells modules for discovery, attack (red teaming), remediation and continuous compliance, plus red teaming services, and claims CI/CD and Burp Suite integration. It spun out of Lancaster University and raised a $30 million Series A in August 2026, which makes it one of the few red teaming vendors in this list still independent.

Don't pick Mindgard if you need a list price to get a purchase approved this quarter, or your team would rather own the attack library than rent one.

---

Who Owns Your Red Team Now?

Six AI red teaming vendors have been bought, or agreed to sell, in the last 14 months. That matters because a red team that belongs to the company selling your model, your firewall or your platform has a reason to find some problems and not others.

• Promptfoo went to OpenAI (announced March 9, 2026).
• Lakera went to Check Point (announced September 2025).
• Haize Labs went to Beacon Software, a holder of 45 software companies, on September 17, 2026. The team became Beacon's internal research group; our analysis of that deal found no continuity plan for outside customers.
• Enkrypt AI went to Anaconda on August 4, 2026, which said existing products and support would not change. Read the licence before you renew.
• Dreadnode agreed to sell to CrowdStrike, which we covered on November 3; see also Falcon Guardian.
• Pillar Security is being bought by Vanta.

The two big open-source projects belong to Microsoft and NVIDIA, which also sell models and inference. No tool in this market is neutral anymore. You get independence back by running the open-source code yourself, choosing the judge model yourself, and keeping the attack set in your own repository, where an acquisition cannot change it.

Do These Tools Test Agents or Just Models?

Most of what is sold as AI red teaming is model jailbreak scanning, and only part of it reaches the tool layer. Four attack surfaces matter for the workload above, and coverage varies widely:

1. Tool calls: Promptfoo tests excessive agency and authorization (BOLA, BFLA) against your real endpoint. garak's AgentBreaker attacks tools it learns from YAML or from the agent itself. Foundry's managed PyRIT tests only Azure tool calls on Foundry agents.
2. MCP servers: only Promptfoo documents an MCP plugin. The Cloud Security Alliance's gap analysis of MITRE ATT&CK and ATLAS argues the frameworks themselves still trail agent control-plane techniques, so expect a test suite to trail them further.
3. Indirect prompt injection: Promptfoo's version is remote-only. Foundry's XPIA test uses mock tool outputs with attack placeholders and is cloud-only.
4. Multi-step attacks: Promptfoo and PyRIT both ship Crescendo-style escalation. garak's AgentBreaker is multi-turn; most of its other probes are single-shot.

None of these tests your real refund API with real money limits, and none should. Build a staging copy of the tool layer with production-like permissions (Microsoft calls it a "purple environment" in its Foundry docs) and point every tool at that.

Which Claims Have Independent Testing Behind Them?

Very few. Most published comparisons of these tools are written by vendors selling one of them or a competitor. The only academic head-to-head found for this piece is Brokman et al. on arXiv, which compared garak, Giskard, PyRIT and CyberSecEval and reported "significant reliability issues in detecting successful attacks." It tested garak 0.9 and PyRIT 0.2 in 2024, so both tools have moved on, and it did not include Promptfoo. No independent study of Promptfoo's red team accuracy turned up at all.

What that means for you: the scanner's verdict is a model's opinion. Promptfoo's own false-positive guide says false positives typically arise when the grader lacks context about the target. Microsoft's AI red team, after red teaming more than 100 generative AI products, wrote that "red teaming can't be automated entirely" and that LLMs are "not as reliable at assessing content in specialized areas." Budget for a human to review every failure and a sample of the passes before anything goes in an audit file.

---

What a Scan Report Gives Your Auditor

A scan report proves you ran tests mapped to a framework category. It does not prove the control exists, and it covers only the risks you can observe through the agent's interface.

The OWASP Top 10 for LLM Applications 2025 is the list auditors ask about most. Interface testing reaches LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM06 Excessive Agency and LLM07 System Prompt Leakage well. It says little about LLM03 Supply Chain or LLM04 Data and Model Poisoning, which you evidence with provenance records and pipeline controls, not prompts. OWASP's separate Top 10 for Agentic Applications, published December 9, 2025, starts with agent goal hijack, which is the indirect injection case. MITRE ATLAS is the technique catalogue a threat model maps to.

Promptfoo's presets (owasp:llm, mitre:atlas, nist:ai:measure, eu:ai-act) produce a report grouped by those labels, which saves hours of evidence formatting. The labels only sort the results. An auditor who reads "LLM06: 0 failures" should also see which tools were in scope, which judge model graded the run, and who reviewed the results. Our read of how the OWASP list was ranked explains why you should weight it by your own incident data too.

The Criteria That Predict Regret

Pick on these, in this order:

1. Can it hit your real tool layer? If the test cannot call a staging copy of your actual tools, it is testing the model, and you already know the model can be jailbroken.
2. Where do attack generation and grading run? Read the vendor's data-handling page. With Promptfoo, settle the remote generation question with security before the first scan.
3. Who owns the judge? Pin the grader model and version in config so a pass in March means the same as a pass in June.
4. Can you export the attack set? If the vendor is acquired, your regression suite should survive in your repository.
5. Does it run unattended? A test nobody runs on pull requests catches regressions only when someone remembers.

What changes the answer: if your agents already live in Microsoft Foundry on Azure tools, the managed AI Red Teaming Agent becomes the cheapest serious option. If a regulator or customer contract requires third-party testing, buy an engagement from Mindgard or another independent firm and keep Promptfoo for the regressions between engagements.

What to Do Before Your Next Agent Launch

This Week:

1. Install Promptfoo, set redteam.provider to your own judge model, and run the agent preset against a staging endpoint with PROMPTFOO_DISABLE_REMOTE_GENERATION on. Note which plugins produce zero tests.
2. Put the list of disabled plugins in front of your security lead and decide whether hosted generation is acceptable for those plugins alone.

This Month:

1. Wire the passing configuration into CI so a pull request that changes a tool, prompt or model fails on a new excessive-agency or BOLA finding.
2. Run garak against your current model and the next candidate model and keep both reports.
3. Give a red teamer two days with PyRIT on the refund and email tools, using Crescendo and TAP.

Before Q1 Close:

1. Write down every red teaming vendor you pay, who owns it today, and whether you can export your attack history.
2. Map your last scan to OWASP LLM and Agentic Top 10 and list the categories the scan cannot reach, with the control that covers each one instead.

The Bottom Line

AI red teaming tools now look like static analysis did a decade ago: a free scanner covers most of the routine findings, a paid team finds what the scanner cannot, and the scanner's owner shapes what it looks for. Promptfoo is the best free option for agents, and its owner is the largest model vendor. Keep your attack set, your judge and your staging tools in your own hands, and the ownership of the scanner matters less.

Before you pick a vendor, find out which of your agent's tools a test has never called.

Continue Reading

• How to Red-Team Your AI Agents Before Production
• Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate
• NeMo Guardrails vs Guardrails AI vs Lakera: Buy the Detector
• Beacon Bought Haize Labs. Who Red-Teams Your Agents Now?
• CrowdStrike Agrees to Buy Dreadnode for Automated AI Testing
• RAMPART: Microsoft's Free Test Suite for $5.7M AI Breaches