Last reviewed: 2026-08-04

Direct answer

Use one promptfooconfig.yaml file with a shared prompt, two cometapi:chat:<model> providers, and deterministic assertions attached to fixed test cases. Promptfoo then evaluates each model against the same inputs and requirements. This keeps the prompt, variables, and pass criteria constant while the provider ID changes.

The model names below are placeholders, not claimed CometAPI catalog entries. Replace both with exact text-model IDs that are currently available. Before editing the provider list, check the current CometAPI model catalog .

description: Compare two CometAPI chat models on fixed support labels

prompts:
  - |
      Classify the request as billing, access, or reliability.
      Return exactly one lowercase label and nothing else.
      Request: {{request}}

providers:
  - id: cometapi:chat:model-a
    config:
      temperature: 0
      max_tokens: 12
  - id: cometapi:chat:model-b
    config:
      temperature: 0
      max_tokens: 12

tests:
  - description: Duplicate invoice charge
    vars:
      request: The same invoice appeared twice on my statement.
    assert:
      - type: equals
        value: billing

  - description: Sign-in failure
    vars:
      request: I cannot sign in after resetting access.
    assert:
      - type: equals
        value: access

  - description: Intermittent request failures
    vars:
      request: Requests return errors every few minutes.
    assert:
      - type: equals
        value: reliability

Populate COMETAPI_KEY in the process environment from your secret manager without placing its value in the YAML file. The marker [REDACTED] is documentation-only and must never be pasted as the value. This check fails clearly when the environment is empty, then runs the evaluation and opens the local viewer:

# Set COMETAPI_KEY from your secret manager before this check.
test -n "$COMETAPI_KEY" || { printf '%s\n' 'COMETAPI_KEY is required' >&2; exit 1; }
npx promptfoo@latest eval -c promptfooconfig.yaml
npx promptfoo@latest view

Do not print the environment value while troubleshooting. Keep it out of shell history, fixtures, screenshots, and committed files. The separate guide to keeping CometAPI keys out of tutorial repositories provides a useful repository boundary.

Happy-path operator workflow

  1. Confirm that both chosen IDs are text-capable models and use the cometapi:chat: prefix.
  2. Save the same provider settings for both models unless the test explicitly compares a setting.
  3. Populate COMETAPI_KEY from the approved secret manager in the same shell that will run Promptfoo.
  4. Run the complete matrix once. With one prompt, two providers, and three cases, every case should produce one result for each provider.
  5. Open the viewer and confirm that all six cells contain responses rather than setup or transport errors.
  6. Accept the baseline only when every equals assertion passes and the matrix is complete.
  7. Record a sanitized run summary, then change only one prompt, model, parameter, or assertion before the next run.

A passing run means both selected models met these three declared requirements. It does not prove broader accuracy, safety, latency, or production readiness.

Who this is for

This workflow is for developers who need a small, repeatable CometAPI Promptfoo suite before selecting a model or changing a production prompt. It is especially useful when manual side-by-side testing has become difficult to reproduce or when reviewers need an explicit pass condition.

It is not a load test, a security assessment, or a substitute for domain review. Start with closed-label tasks whose expected results are unambiguous. Expand into model-graded checks only when exact or structural assertions cannot express the requirement.

Key takeaways

  • Promptfoo has a dedicated cometapi provider and documents chat, completion, embedding, and image provider types.
  • A fair two-model comparison keeps prompts, cases, assertions, and shared parameters unchanged.
  • Deterministic assertions such as equals, contains, and JSON checks make the pass contract visible.
  • A provider error is not a failed quality assertion. Classify and resolve it separately.
  • Compare only complete matrices; missing cells invalidate a side-by-side result.
  • Store sanitized metadata and output hashes, not environment values, headers, or unreviewed raw responses.
  • Treat every current model ID and optional metric as something to verify before relying on it.

Sources checked

  • The CometAPI Promptfoo integration walkthrough shows the environment setup, provider format, YAML structure, and evaluation command used as the CometAPI-side baseline.
  • The Promptfoo CometAPI provider reference documents COMETAPI_KEY, the cometapi:<type>:<model> pattern, supported provider types, configuration parameters, and command-line examples. Its refetched page reports an update on August 4, 2026.
  • The Promptfoo getting-started guide explains prompts, providers, test variables, assertions, eval, and the result viewer. It also states that an evaluation tests every prompt, model, and test case.
  • The Promptfoo assertions and metrics reference defines deterministic checks including equals, contains, regular expressions, JSON validation, latency, and custom functions. It separately identifies model-assisted checks and notes that cost checks depend on cost information being available.

These four public pages returned successful responses in the supplied refetch. The tutorial does not rely on the promotional pricing, adoption, savings, privacy, uptime, or latency claims present in one source because they are unnecessary for the configuration contract.

Contract details to verify

Provider selection. Promptfoo documents cometapi:chat:<model> for chat models, with cometapi:<model> as a chat shorthand. Use the explicit form in a tutorial because the request family remains visible. Do not reuse this configuration unchanged for an embedding or image model; those have different type prefixes.

Environment boundary. The provider reference names COMETAPI_KEY as the required environment variable. The evaluation process must inherit it, but the config and logs must not contain its value. A failure to load it is a setup problem, not evidence that either model performed poorly.

Model identity. Record the exact provider strings used in every run. Friendly labels such as model A and model B are useful in prose, but the executable config needs current catalog IDs. If either ID changes, create a new baseline instead of silently comparing results produced by different model identities.

Shared settings. The provider reference shows parameters such as temperature and max_tokens. This example gives both providers the same values. A low temperature narrows the test setup, but the pass decision still comes from assertions rather than from assuming repeatability.

Assertion meaning. The equals assertion requires the complete output to match the expected value. That is appropriate for a one-label contract and deliberately treats extra explanation as a failure. For a different task, Promptfoo documents case-insensitive containment, regular expressions, JSON checks, custom functions, and model-assisted grading. Choose the narrowest assertion that represents the reader-visible requirement.

Completion rule. Define success before running the suite: all expected provider-and-case cells must return, and all deterministic assertions must pass. Do not average away a missing provider response. Do not turn a transport error into a zero quality score. Do not weaken an expected value merely to make an unexpected response pass.

Sanitized logging. Keep enough metadata to reproduce the decision without copying sensitive input or output. A compact record can use this shape:

run_id: eval-042
started_at: 2026-08-04T00:00:00Z
config_revision: local-commit
case_id: billing-duplicate
provider_id: cometapi:chat:model-a
assertion_type: equals
assertion_passed: true
latency_ms: null
cost_usd: null
error_class: null
http_status: null
output_sha256: '[REDACTED]'
raw_output_logged: false

Populate latency or cost only when the run actually reports those values. Leave unavailable fields as null; inventing zero would turn missing evidence into a measurement. Never log COMETAPI_KEY, request headers, full user-derived variables, or raw responses that have not passed a data-handling review. Hashing an output can help correlate repeated results without making the text part of routine logs.

Failure modes

Error-path operator workflow

  1. Stop the comparison when any expected cell is missing or marked as an error.
  2. Classify the event as environment, configuration, provider, transport, or assertion failure.
  3. Record only the sanitized fields available for that event.
  4. Fix setup and provider errors before evaluating model quality.
  5. For a valid response that fails an assertion, inspect the local output, identify whether the prompt, expected value, or model behavior is responsible, and add a regression case when appropriate.
  6. Rerun the full matrix after a fix so both providers are judged under the same config revision.
  7. Keep the failed run in the audit summary; do not replace it silently with a successful retry.

The environment value is absent. Promptfoo cannot use the documented provider setup if the process does not inherit COMETAPI_KEY. Set COMETAPI_KEY from your secret manager in the active shell and rerun. [REDACTED] is documentation-only and is not a usable value. Do not add the value to promptfooconfig.yaml as a workaround.

A model ID is stale or mistyped. One provider can fail before generating output. Verify the exact catalog ID and the chat type. Until both provider cells return, there is no valid quality comparison.

The provider type is wrong. The native provider distinguishes chat, completion, embedding, and image modes. A chat prompt paired with the wrong type is a contract error. Correct the provider identifier rather than changing assertions.

YAML or variables do not match. A malformed config, inconsistent indentation, or a missing request variable can prevent a case from reaching either model. Validate the file and confirm that every {{request}} placeholder has a corresponding test value.

A valid answer contains extra prose. An output such as billing - duplicate charge fails equals: billing. For this closed contract, that is a meaningful failure. Tighten the instruction or choose a model that follows it; do not switch to a broad contains assertion unless extra text is genuinely acceptable to the application.

An assertion is too permissive. A containment check can pass an answer that mentions the expected label while recommending a different one. Prefer exact or structural validation when the application consumes a closed value.

Only one model finishes. Do not report the completed model as the winner. Resolve the other provider’s error and rerun both sides under the same conditions.

Optional metrics are missing. The assertions reference says cost checks apply when cost information is available. Missing cost or latency evidence should remain unknown, not become a pass, failure, or zero.

Logs capture too much. Raw prompts and outputs can contain user material even when this small example does not. Keep routine records to case IDs, provider IDs, assertion outcomes, available metrics, error classes, and hashes. Use a separately controlled review path when raw content is necessary.

FAQ

Why compare exactly two models?

Two providers make the first matrix easy to inspect and keep the operator workflow concrete. Promptfoo can evaluate more providers, but every added model multiplies the number of result cells and the amount of failure triage. Establish the contract with two, then expand deliberately.

Can I use the shorthand provider ID?

Promptfoo documents cometapi:<model> as a chat shorthand. The explicit cometapi:chat:<model> form is clearer in a maintained tutorial because readers can see the intended provider type without relying on a default.

Should every evaluation use equals?

No. Use it when the application requires one exact value. Promptfoo also documents case-insensitive containment, regular expressions, JSON validation, custom code, similarity, and model-assisted rubrics. Match the assertion to the actual output contract, not to whichever check is easiest to pass.

When is a model-graded assertion appropriate?

Use one when the requirement is semantic and cannot be represented reliably by exact, structural, or programmatic rules. Keep the three closed-label cases deterministic. If you later add a free-form explanation task, define the grading rubric and its grading provider explicitly so reviewers know that the result depends on another model.

Does a passing suite prove that a model is production-ready?

No. It proves only that the selected model IDs passed the declared cases and assertions in that run. Production acceptance may also require broader domain cases, resilience checks, safety review, latency evidence, and a local CometAPI smoke test .

What should I do after a model or prompt changes?

Record the new exact model ID or config revision, rerun every case against both providers, and retain the old summary for comparison. Do not mix cells from separate configurations into one result matrix.

Reader next step

Create promptfooconfig.yaml, replace model-a and model-b with two verified chat-model IDs, and run the three-case matrix. Confirm that every cell returns, review any failed exact match, and add one real failure case from your application before treating the file as a baseline. Commit the config and test cases, but not environment values or raw user responses.

When you are ready to connect the provider and run the first controlled evaluation, Start with CometAPI .