PromptfoovsDeepEval:whichLLMevaltoolshouldyouuse?

Promptfoo vs DeepEval is a choice between two testing styles, not two quality levels. Promptfoo treats an eval as configuration: a YAML file lists prompts, models and test cases, and the CLI fills a results matrix. DeepEval treats an eval as code: you write pytest functions, your code produces the output, and a metric scores it with a judge model. Versions checked for this page: promptfoo 0.125.1 on npm and deepeval 4.3.3 on PyPI, current on November 5, 2026.

Which should you use, Promptfoo or DeepEval?

Use Promptfoo to compare prompts or models and gate pull requests on cheap checks; use DeepEval when you need judged quality metrics inside a Python test suite.

Neither replaces a public benchmark. A score on SWE-bench tells you how a model does on someone else’s repos; these tools test the prompts and tasks you actually ship.

How does each tool work?

Promptfoo runs a declared matrix and grades each cell with assertions; DeepEval runs your test functions and grades each test case with metrics.

Promptfoo. The default file is promptfooconfig.yaml with three blocks: prompts (templates with {{vars}}), providers (the models) and tests (variables plus assert). The assertions page splits checks in two:

• Deterministic: equals, contains, icontains, regex, starts-with, is-json, contains-json, is-sql, is-xml, levenshtein, rouge-n, bleu, latency, cost, tool-call checks, and escape hatches javascript, python and webhook.
• Model-graded: llm-rubric, g-eval, factuality, similar (embeddings), answer-relevance, context-faithfulness, select-best and others.

Any assertion can be negated with a not- prefix, weighted with weight, and combined against a test-level threshold. Results open in a local web viewer with promptfoo view.

DeepEval. A test builds an LLMTestCase (input, actual output, optionally expected output and retrieval context) and calls assert_test(test_case, [metrics]). Each metric returns a score from 0 to 1 and a written reason, and passes if the score is at or above its threshold (0.5 by default for G-Eval, per the G-Eval docs). The README lists metric families for custom criteria (G-Eval, DAG), agents (task completion, tool correctness, plan adherence), RAG (faithfulness, contextual recall and precision), multi-turn chat, MCP, multimodal and voice. Note the current params enum is SingleTurnParams; older tutorials use LLMTestCaseParams.

The practical difference: in Promptfoo the tool calls your model; in DeepEval your code calls your model and hands the output over. That makes DeepEval better for testing a whole pipeline (retrieval, tools, post-processing) and Promptfoo faster for testing prompts and models in isolation.

How do you evaluate a local Ollama model with each?

Both tools can run the model under test and the judge on Ollama with no API key. Install commands first:

# Promptfoo (Node.js CLI; also on brew and pip)
npm install -g promptfoo
# DeepEval (Python 3.9+)
pip install -U deepeval openai
ollama pull qwen2.5-coder:7b && ollama pull qwen3:14b

The two configs below are minimal sketches built from each project’s docs: a 7B coding model writes a function, and a larger local model judges it. Set temperature to 0 for the model under test so reruns compare like with like (why temperature matters for coding).

Promptfoo. The Ollama provider uses IDs like ollama:chat:<model> and reads OLLAMA_BASE_URL (default http://localhost:11434). defaultTest.options.provider sets the grader for model-graded assertions.

# promptfooconfig.yaml
prompts:
  - 'Write a Python function {{name}} that {{task}}. Return only code.'
providers:
  - id: ollama:chat:qwen2.5-coder:7b
    config:
      temperature: 0
defaultTest:
  options:
    provider: ollama:chat:qwen3:14b   # local judge for llm-rubric
tests:
  - vars:
      name: slugify
      task: lowercases a string and joins words with single hyphens
    assert:
      - type: contains
        value: 'def slugify'
      - type: not-contains
        value: 'import requests'
      - type: llm-rubric
        value: Handles empty input and leading or trailing separators

Run promptfoo eval, then promptfoo view. The two deterministic checks cost nothing; only llm-rubric calls the judge.

DeepEval. The Ollama integration gives you OllamaModel for the judge, or deepeval set-ollama --model=<name> to make Ollama the default judge for every metric (deepeval unset-ollama reverts). Generation is your code; here it goes through Ollama’s OpenAI-compatible API.

# test_slugify.py — run with: deepeval test run test_slugify.py
from openai import OpenAI
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.models import OllamaModel
from deepeval.test_case import LLMTestCase, SingleTurnParams

llm = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
judge = OllamaModel(model="qwen3:14b", temperature=0)

def test_slugify():
    prompt = ("Write a Python function slugify that lowercases a string "
              "and joins words with single hyphens. Return only code.")
    out = llm.chat.completions.create(
        model="qwen2.5-coder:7b", temperature=0,
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content
    metric = GEval(
        name="Correct slugify",
        evaluation_steps=[
            "Check the code defines slugify and is valid Python.",
            "Check it handles empty input and leading or trailing separators.",
        ],
        evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT],
        threshold=0.7,
        model=judge,
    )
    assert_test(LLMTestCase(input=prompt, actual_output=out), [metric])

Every check here goes through the judge. For a hard “it compiles” gate you would add a plain Python assert next to assert_test, which is ordinary pytest.

How do they handle LLM-as-judge and its cost?

Promptfoo makes the judge optional per assertion; DeepEval makes it the default for nearly every metric, so judge choice drives both cost and reliability.

• Default judge. Promptfoo picks a default grader from whatever credentials are in your environment (OpenAI, Anthropic, Gemini, Mistral, Azure and others, per the model-graded docs). Override order: --grader CLI flag, then a per-assertion provider, then options.provider. Watch the fallback: if none is set, defaultTest.provider can silently become the judge. DeepEval’s quickstart expects OPENAI_API_KEY unless you configure another model.
• Calls per test. Promptfoo makes one target call per prompt x provider x test, plus grader calls only for model-graded assertions. In DeepEval each judged metric calls the judge per test case. G-Eval also generates its evaluation steps from criteria unless you pass evaluation_steps, which the docs say skips that step and makes scores more consistent.
• Caching. Promptfoo caches LLM requests in ~/.cache/promptfoo, and its GitHub Action example caches that folder between CI runs to cut time and cost.
• Local judge memory. Ollama keeps a model loaded for 5 minutes by default. Promptfoo’s docs suggest keep_alive: 0 and note that with --max-concurrency 1, grader calls are grouped by provider so one judge finishes before the next loads.
• Judge size. DeepEval parses the judge’s answer as structured output. Its custom-model guide says smaller models struggle to produce valid JSON from prompting alone. A local judge in the 14B+ range is a safer start than a 1.5B one.

With a fully local setup the marginal cost is electricity and wall-clock time. With a hosted judge, the cheapest Promptfoo suite is the one where most assertions are deterministic.

What about licence, ownership and alternatives?

Both are open source and free to self-run; the difference is who owns them and what the paid layer is.

On the deal: OpenAI announced on March 9, 2026 that it would acquire Promptfoo, and Promptfoo’s own post says it “will remain open source” and “will continue to support a diverse range of providers and models.” The GitHub README now reads “Promptfoo is now part of OpenAI” and still lists the MIT licence. Nothing in the docs restricts Ollama or other providers today. The MIT licence means a fork stays possible if that changes.

Telemetry is light on both. Promptfoo says it records commands and assertion types but not prompts, outputs or keys, and PROMPTFOO_DISABLE_UPDATE=1 stops the npm update check. DeepEval sends event and metric names plus an anonymous ID to PostHog, per its privacy page. Confident AI’s free cloud tier caps you at 5 test runs a week, but the local framework does not need the cloud at all.

Alternatives worth knowing: Ragas if you only need RAG metrics, Inspect AI if you write benchmark-style evals in Python, and tracing platforms with eval features if you want scores attached to production traces, covered in self-hosted Langfuse. Promptfoo also ships red-teaming scanners; that is a separate use and out of scope here.

What do developers report in practice?

Public reports split along the same line as the design: Promptfoo users like its low ceremony and dislike YAML; DeepEval users like the reasons attached to scores and hit trouble with small local judges.

• Non-Python stacks pick Promptfoo. One Hacker News commenter chose it for a Rails codebase because the Python alternatives, DeepEval and Ragas among them, did not fit and went “too into the weeds” when they wanted simple tests (HN, 2024).
• YAML is the main complaint. In the same thread another developer objected to learning a YAML spec and writing code as YAML array elements, preferring a programmatic approach (HN).
• Custom checks for local models. A developer testing local models for daily coding described a custom Python assertion in Promptfoo that detects repetition loops in output (HN, June 2026), a failure mode common in quantized local models.
• One judge at a time. A user whose hardware held a single Ollama judge saw models unload and reload for every row; the fix, grouping grader calls by provider in serial mode, merged in April 2026 (promptfoo #8391).
• Small judges fail in DeepEval. With OllamaModel on qwen2-math:1.5b, AnswerRelevancy returned an unfilled template as its reason; the maintainer replied that smaller Ollama models “don’t do so well for evaluation” (deepeval #2292). An older open issue reports some metrics still asking for an OpenAI key with a local model configured (deepeval #1243).
• Auto-updates. In an r/LLMDevs thread titled “Why I stopped using DeepEval,” a maintainer replied that release 2.7.6 made automatic updates opt-in (Reddit).
• Lock-in worry. After the acquisition, one HN commenter who rated Promptfoo highly asked what it means for teams testing Gemini or Claude through it (HN, March 2026).

The common lesson across both tools: decide what your judge is, pin it, and validate it on a few hand-graded cases before trusting its scores.

FAQ