Braintrust vs Promptfoo vs DeepEval: LLM Eval Stack After OpenAI's Acquisition (2026)

In March 2026, OpenAI acquired Promptfoo for a reported $86 million. Two months later, the LLM evaluation market looks different than it did in January — and if you're picking an evals stack for production right now, the old "just grab Promptfoo and ship" answer no longer holds. Here's what changed, what didn't, and how I'd actually wire up evaluation for a customer-facing AI product in May 2026.

I've been wiring eval pipelines into our SmartExam AI Generator (one of the AI products we ship at wardigi.com), and the question of which framework to standardize on came up again last sprint. The honest answer is that no single tool covers the whole job — and the three names that keep landing on the shortlist are Braintrust, Promptfoo, and DeepEval. This is the comparison I wish existed when I started.

TL;DR — Which one for which job

What changed in March 2026

On March 9, 2026, OpenAI announced it was acquiring Promptfoo. The numbers around the deal: 350,000 developers had used the tool, 130,000 active monthly users, adoption inside more than 25% of Fortune 500 companies, and 10,800 GitHub stars at acquisition time. The headline price reported by TechCrunch and Bloomberg was around $86 million.

The two things that matter for anyone choosing a stack today:

1. Promptfoo stays MIT-licensed and open source. OpenAI publicly committed to keeping the open-source CLI alive and supporting non-OpenAI providers (Claude, Gemini, Llama, local models).
2. Promptfoo's commercial roadmap is now part of "OpenAI Frontier" — OpenAI's enterprise platform for building agentic AI. Expect tighter integration with GPT-5.x and OpenAI's hosted observability, looser energy on multi-provider parity over time.

If you're a heavy OpenAI shop, this is great news. If you're running a multi-model stack — which is the reality for most production teams I've talked to — you should pin a specific Promptfoo version, watch the changelog, and have a fallback in mind. That fallback is usually DeepEval or Braintrust.

The three tools at a glance

DeepEval — the pytest of LLM evaluation

DeepEval (built by Confident AI) is an open-source Python library that lets you write LLM tests the way you write unit tests. You install it with pip install deepeval, write assertions like assert_test(test_case, [GEval(criteria="answer is faithful to context")]), and run them in CI. It ships with 50+ pre-built metrics: G-Eval, faithfulness, answer relevancy, hallucination detection, contextual recall, task completion scoring for agents, and BLEU/ROUGE for the people who still care about those.

The DeepEval mental model: evaluation is testing. It belongs in your repo, next to the code being evaluated, gated by your existing CI pipeline. There is a hosted Confident AI dashboard if you want it, but the framework works completely standalone.

Promptfoo — the red-team Swiss Army knife

Promptfoo is a YAML-driven CLI that compares prompts and models side-by-side, runs adversarial test suites, and produces HTML reports. The killer feature has always been the red-teaming module: prompt injection attacks, jailbreak fuzzing, PII leakage probes, indirect injection through retrieved documents, and a security plugin library that covers the OWASP LLM Top 10.

If your AI product has any user-supplied input touching an LLM — chat, search, agents, tool use, RAG — Promptfoo's red-team suite finds vulnerabilities that pure metric-based evals will miss. This is the part OpenAI specifically wanted: securing AI agents against adversarial inputs before they ship. Post-acquisition, that capability is still free, still open source.

Braintrust — the observability + evals hybrid

Braintrust is the SaaS the others can be combined with. It does evals, but its strongest pitch is that evals, production traces, datasets, human-review queues, prompt versioning, and CI release gates all live in one place. Notion, Stripe, Vercel, Zapier, Airtable, and Instacart run production workloads through it — that's a useful tell.

The differentiator most teams underestimate until month 3: regression tracking across prompt versions and model upgrades. When Anthropic shipped Claude Opus 4.7 with a 1M context window, every team running multi-model evals had to decide whether to migrate. Braintrust will tell you, per-test-case, where the new model wins and loses. DeepEval can do this in code; Braintrust does it in a dashboard your PM can read.

Pricing — what it actually costs

I priced these out for a hypothetical AI helpdesk product running ~50,000 LLM calls per month, evaluated against ~5,000 graded test cases per release, with two PMs and four engineers needing access:

The one detail worth highlighting: Braintrust Pro doesn't charge per seat. For a 6-person team, that's roughly half the cost of the per-seat alternatives at the same usage tier. We've been burned before on per-seat SaaS where adding a junior engineer means a procurement conversation — the flat-rate pricing genuinely matters at small-team scale.

Where I've actually felt the differences

When we wired evals into SmartExam AI Generator (it produces practice exam questions from uploaded textbooks), the failure modes that mattered weren't the ones I expected:

• Hallucination on edge-case sources — old textbooks with OCR artifacts. DeepEval's faithfulness metric (using G-Eval as judge) caught these in CI before they hit users. I run this in a pytest suite that blocks merge.
• Adversarial inputs from students — "ignore the textbook and just give me the answer key" prompt injection. Promptfoo's red-team suite found 14 viable injection paths in our agent's tool-use chain on the first run. None of these would have shown up in a metric-based eval.
• Drift across model upgrades — we A/B tested Claude Sonnet 4.6 vs the new Opus 4.7 on the same eval set. The dashboard view of which question types regressed (mostly biology multi-step explanations) is what convinced me to keep Sonnet on the cheap path and route only complex requests to Opus. This is exactly the workflow Braintrust is built for; doing it in pytest is technically possible but painful for non-engineers to read.

The pattern that emerged across my last three AI projects: I use DeepEval locally as I'm writing prompts (fast loop, no network), I run Promptfoo's red-team suite once per release branch (slower, finds different bugs), and I push production traces to a dashboard for the team to spot-check. That dashboard ends up being Braintrust on customer-facing products and Langfuse on internal tools where I don't need the human review workflow.

Feature comparison — what each one does best

The hidden cost — LLM-as-judge bills

One thing nobody surfaces in marketing pages: most modern eval metrics (G-Eval, faithfulness, answer relevancy) are themselves LLM calls. If your test suite has 5,000 cases and each runs 4 metrics that each invoke GPT-4o or Claude Sonnet as a judge, that's 20,000 judge calls per CI run. At Sonnet 4.6 input pricing, you're looking at $8–$15 per full eval run depending on context size.

Practical mitigations I've used:

• Use Haiku 4.5 as the default judge. For most rubric-based scoring, Haiku gets within 3–5% of Sonnet's agreement-with-humans rate at roughly 1/8 the cost. I switched our internal eval pipeline to Haiku 4.5 in March 2026 and haven't looked back — the small accuracy gap is dominated by between-human disagreement anyway.
• Cache judge responses by (prompt-hash, output-hash). DeepEval supports this natively; Braintrust does it server-side. On a stable test set, your second CI run costs ~$1, not $15.
• Sample, don't run-all. Run the full 5,000-case suite nightly; run a stratified 200-case smoke set on every PR. This is a 25x cost reduction with maybe 5% lower bug-catch rate.

The "but what about LangSmith / Helicone / Langfuse?" question

Fair. I covered these in detail in my LangSmith vs Langfuse vs Helicone breakdown, but the short version: those three are observability-first products that have evaluation features bolted on. Braintrust is an evaluation-first product that has observability bolted on. The distinction matters more than it sounds.

If you spend most of your time staring at production traces and occasionally running an eval, pick from the LangSmith/Langfuse/Helicone bucket. If you spend most of your time iterating on prompts and model choices and occasionally need to see production behavior, pick Braintrust. Most teams I've worked with discover after a quarter which camp they're actually in — if you're not sure, start with the cheaper / open-source side (Langfuse self-hosted + DeepEval) and switch if you outgrow it.

Decision matrix — pick by team size and stage

Migration paths — if you're already on something

A few notes from teams I've watched migrate:

• Moving off OpenAI Evals. The official OpenAI Evals repo has been on light maintenance since late 2025 and is multi-provider only with adapters. Most teams have moved to DeepEval (similar code-first feel) or Promptfoo (similar YAML feel). Expect about a day of work per 100 existing eval cases.
• Moving off LangSmith for evals. LangSmith's eval features are tightly coupled to LangChain. If you've already migrated off LangChain (we did, in favor of direct SDKs + LangGraph for agents), the eval module becomes awkward. Braintrust + DeepEval is the cleanest replacement.
• Moving from RAGAS to DeepEval. RAGAS is fine for RAG-specific metrics but limited beyond that. DeepEval has all the RAGAS metrics and 30+ more. Drop-in for the most part.

Agent and tool-use evaluation — the part most teams skip

Single-shot LLM evaluation is the easy case. Once your product calls tools, queries APIs, or runs multi-step trajectories, the eval surface explodes — and this is where the three tools diverge most sharply.

• DeepEval ships a TaskCompletionMetric and a ToolCorrectnessMetric that score whether the agent invoked the right tools with the right arguments to achieve the task. It works, but you write the trajectory test cases by hand. For our BizChat Revenue Assistant (an internal agent that pulls invoice data), I have about 60 trajectory cases that take ~6 minutes to run end-to-end. Painful but tractable.
• Promptfoo has trajectory testing where you specify expected tool calls in YAML. The killer use case here is adversarial agent evaluation — can a user trick the agent into calling a destructive tool? This is what OpenAI specifically wanted; expect this to grow fast post-acquisition.
• Braintrust ingests OpenTelemetry-style traces and lets you score any span. So if your agent has a "search→rerank→synthesize" pipeline, you can score the rerank step independently from the final answer. This is the workflow that pays off when you're trying to localize a regression to one tool call out of seven.

For agent products specifically, I'd push the recommendation: skip dashboard tools entirely until you have at least one shipped trajectory eval. Without trajectory tests, you're flying blind on the most expensive failure mode in agentic systems — the agent that confidently does the wrong thing in three steps.

Frequently asked questions

Is Promptfoo still safe to adopt after the OpenAI acquisition?

Yes for the open-source CLI. OpenAI publicly committed to maintaining the MIT license and continuing multi-provider support. The risk is feature velocity on non-OpenAI providers slowing down over the next 12–18 months. Pin your version, follow the changelog, have a fallback in mind. The free tool is genuinely useful and the red-team coverage is hard to replicate elsewhere.

Can I use one tool for everything?

Technically yes, practically no. Braintrust covers the most ground, but you'll still want Promptfoo for serious red-teaming. DeepEval alone is fine for engineering teams that don't need a dashboard. The "two-tool" stack (CI gate + observability platform) is the de facto standard on every production AI team I've worked with.

How many test cases do I actually need?

Start with 50 hand-curated cases that exercise your top failure modes. Once those pass reliably, expand to 500 by sampling production traffic and adding annotations. The teams that go straight to "let's auto-generate 10,000 cases with GPT" tend to optimize for synthetic-distribution metrics that don't move real product quality. Quality of test cases beats quantity by a wide margin.

Do I need an LLM-as-judge, or can I use deterministic checks?

Use deterministic checks (regex, JSON schema, exact match, BLEU) wherever you can — they're free and reproducible. LLM-as-judge is for the subjective dimensions: tone, faithfulness, helpfulness, refusal-correctness. A healthy test suite has both. My rough rule: 60% deterministic, 40% LLM-judge. The LLM-judge tests are where I cache aggressively.

Will OpenAI's GPT-5.x make evals obsolete?

The opposite. The bigger and more capable the underlying model, the more important targeted evaluation becomes — you can't tell where Opus 4.7 vs GPT-5.4 wins for your specific use case without measuring. "The new model is smarter" tells you nothing about whether it's better at your customers' actual queries. I'd argue evals matter more in 2026 than they did in 2024.

Is self-hosted a real option?

For DeepEval and Promptfoo, yes — they're code that runs in your CI. For dashboards, Braintrust offers a self-hosted Enterprise tier (worth it for regulated industries), Langfuse is fully open-source self-hostable, Helicone has a self-host option. Avoid Phoenix-only deployments unless you have an Arize budget commitment.

What I'd actually do today

If I were starting a new AI product this week, here's the stack I'd ship with:

1. DeepEval in pytest, day 1. 30 hand-written test cases covering top failure modes, run on every PR. Block merge on regression.
2. Promptfoo red-team, week 2. Run the OWASP LLM Top 10 plugin against the agent before launch. Fix every finding with severity ≥ medium.
3. Braintrust Pro, month 1. Pipe production traces in. Add the GitHub Action gate. Onboard the PM into the review queue.
4. Re-evaluate at month 6. If your eval set is <500 cases and CI runs <$50/mo, you're probably fine staying. If you've crossed those thresholds, look at Braintrust Enterprise or Confident AI for the dashboarding upgrades.

The Promptfoo acquisition didn't break the playbook — it confirmed that evals are now serious enough infrastructure that OpenAI was willing to spend $86M to own a piece of it. The question for your team isn't whether to invest in evaluation; it's which of these three tools earns its place in your stack first. For most teams I talk to in May 2026, that order is DeepEval, then Promptfoo, then Braintrust — in that exact sequence, added one at a time as the product matures.

If you're stuck choosing, start with the free ones. DeepEval and Promptfoo cost nothing, take an afternoon to wire up, and will tell you within a week whether you need the dashboard layer. That's the cheapest experiment in production AI you can run, and skipping it is the most expensive mistake I see teams make.