promptfoo

by Promptfoo (OpenAI)

Test, compare and red-team your prompts and LLM apps before shipping them.

25.9 k stars GitHub MIT FreeOpen sourceLocal

View on GitHub →

What it does

Automated evals and side-by-side model comparison
Red teaming and vulnerability scanning for apps and agents
Plugs into your CI/CD to catch regressions

Install it

Requirements: Node.js (or Homebrew / pip) and an API key for the LLM provider under test

Install

npm install -g promptfoo

Create an example project

promptfoo init --example getting-started

Run the eval (inside the new folder)

promptfoo eval

The prompt for your agent

Paste it into your coding agent: it checks the requirements and installs it for you.

Claude CodeCodexCursorGemini CLIopencode

Prompt

Install promptfoo (https://www.promptfoo.dev) on my machine.
1. Check the requirements: Node.js (or Homebrew / pip) and an API key for the LLM provider under test. If anything is missing, tell me before installing anything.
2. Follow the official method:
- Install: npm install -g promptfoo
- Create an example project: promptfoo init --example getting-started
- Run the eval (inside the new folder): promptfoo eval
3. Check that it works, then explain how to launch it and where to start.

Always review what your agent is about to run, especially install commands.