Regression-test prompts, agents, and RAG outputs before shipping changes

Use promptfoo when an agent needs to evaluate prompt, agent, or RAG behavior against saved assertions before a change goes live. The value here is the repeatable evaluation workflow, not a generic AI tooling catalog entry.

Prerequisites

Node.js, CI pipeline, model provider credentials

Installation

Use the upstream install or setup path that matches your environment:

• npm install -g promptfoo
• Also available via brew install promptfoo and pip install promptfoo. You can also use npx promptfoo@latest to run any command without installing.

Requirements and caveats from upstream:

• Requires Node.js ^20.20.0 or >=22.22.0 for npm and npx usage.
• Most LLM providers require an API key. Set yours as an environment variable:
• Node.js Package

Basic usage or getting-started notes:

• <a href="https://www.promptfoo.dev/docs/getting-started/">Getting Started</a> ·

• sh

• promptfoo init --example getting-started

• Source: https://github.com/promptfoo/promptfoo

• Extracted from upstream docs: https://raw.githubusercontent.com/promptfoo/promptfoo/HEAD/README.md

Documentation

• https://www.promptfoo.dev/docs/intro/

Source

• Agent Skill Exchange