discourse/plugins/discourse-ai

Discourse AI Plugin

Plugin Summary

For more information, please see: https://meta.discourse.org/t/discourse-ai/259214?u=falco

Evals

The directory evals contains AI evals for the Discourse AI plugin. You may create a local config by copying config/eval-llms.yml to config/eval-llms.local.yml and modifying the values.

To run them use:

cd evals ./run --help

Usage: evals/run [options]
    -e, --eval NAME                  Name of the evaluation to run
    -m, --models NAME                Models to evaluate (comma separated, defaults to all)
    -l, --list                       List eval ids
        --list-models                List configured LLMs
        --list-features              List feature keys available to evals
        --list-agents              List agent definitions under evals/agents
    -f, --feature KEY                Filter evals by feature (module_name:feature_name)
    -j, --judge NAME                 LLM config used as a judge (defaults to gpt-4o when available)
        --agent-keys KEYS          Comma-separated list of agent keys (or repeat the flag) to run sequentially
        --compare MODE               Run comparisons (MODE: agents or llms)
        --dataset PATH               Path to a CSV dataset file (requires --feature)

To run evals you will need to configure API keys in your environment:

OPENAI_API_KEY=your_openai_api_key ANTHROPIC_API_KEY=your_anthropic_api_key GEMINI_API_KEY=your_gemini_api_key

Custom agents for evals

Eval runs can swap the built-in agents with YAML definitions stored in plugins/discourse-ai/evals/agents. Use --list-agents to discover available entries; the special key default always refers to the built-in agent prompt. Pass --agent-keys key1,key2 (or repeat --agent-keys key) to apply them:

./run --eval simple_summarization --models gpt-4o-mini --agent-keys topic_summary_eval,another_prompt

Each agent file only needs a system_prompt (and optional description). When specified, that prompt replaces the default system prompt of whichever agent the eval runner would normally use. Pass multiple keys (including default) to rerun the same evals with different prompts without restarting the CLI. Add new files under that directory to compare alternate prompts without touching the database.

When running agent comparisons (--compare agents) the CLI automatically prepends the built-in default agent so you can benchmark your YAML prompts against the stock behavior. Non-comparison runs still execute only the agents you list.

Dataset-driven evals

Supply --dataset path/to/file.csv along with --feature module:feature_name to generate eval cases from a CSV instead of YAML files. Each row must include content and expected_output columns; rows are converted into individual eval ids (prefixed with the dataset filename) that reuse the selected feature’s runner. Example:

./run --dataset evals/datasets/spam.csv --feature spam:inspect_posts --models gpt-4o-mini

Comparison matrix

Use the --compare flag to ask the CLI to judge multiple runs together:

• --compare agents: require a single --models value and at least one agent key (the built-in default agent is implicitly added). Each eval is executed for every agent; the judge LLM scores them side-by-side and announces the winner plus individual ratings.
• --compare llms: require at least two --models and exactly one agent (default unless you pass --agent-keys custom_agent). Every eval runs once and the judge compares the outputs from each LLM. Logs include the agent key (or default) so you can correlate recordings.

Both modes reuse the rubric declared under the eval’s judge block and stream the comparison summary to STDOUT. The structured log files continue to be written for each underlying run so you can drill into the raw outputs if the judge’s reasoning needs inspection.