Discourse/discourse
Discourse/discourse
0
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2026-08-08 17:53:55 +08:00
Code Packages Activity Actions
discourse/plugins/discourse-ai/spec/evals
History
Exact
Roman Rizzi 71c1cbffaf
 FEATURE: Format eval comparison results into a table (#36724) 
Personas comparison:

```

┌──────────────────────┬───────────────────────────────┬───────────────────────────────┬───────────────────────────────┐
│ input                │ default                       │ topic_summary_eval            │ judge                         │
├──────────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┤
│ simple_summarization │ [PASS]                        │ [PASS]                        │ [TIE]                         │
│                      │ 9/10 — Candidate 1 provides a │ 9/10 — Candidate 2            │ Tie — Both candidates         │
│                      │ comprehensive summary that    │ effectively summarizes the    │ effectively summarize the     │
│                      │ captures the central idea and │ conversation, highlighting    │ conversation, capturing the   │
│                      │ key attributes of dogs,       │ the loyalty, intelligence,    │ central idea and key          │
│                      │ including their roles as      │ and roles of dogs as service  │ attributes of dogs, including │
│                      │ service and therapy animals.  │ and therapy animals. The      │ their loyalty, intelligence,  │
│                      │ The response is clear and     │ response is concise and meets │ and roles as service and      │
│                      │ concise, meeting all rubric   │ all rubric criteria.          │ therapy animals. Both         │
│                      │ criteria.                     │                               │ responses meet the rubric     │
│                      │                               │                               │ criteria equally well.        │
└──────────────────────┴───────────────────────────────┴───────────────────────────────┴───────────────────────────────┘
Summary: default: 1/1 pass | topic_summary_eval: 1/1 pass | judge: 1/1 pass
Legend: [PASS]=pass, [FAIL]=fail, [SKIP]=skipped, [TIE]=tie
```

LLMs comparison against dataset:

```
┌──────────────────────────────┬────────┬─────────────┐
│ input                        │ GPT-4o │ GPT-4o-mini │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-1  │ [PASS] │ [PASS]      │
│                              │ true   │ true        │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-2  │ [PASS] │ [PASS]      │
│                              │ false  │ false       │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-3  │ [PASS] │ [PASS]      │
│                              │ true   │ true        │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-4  │ [PASS] │ [PASS]      │
│                              │ false  │ false       │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-5  │ [PASS] │ [PASS]      │
│                              │ true   │ true        │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-6  │ [PASS] │ [PASS]      │
│                              │ false  │ false       │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-7  │ [PASS] │ [PASS]      │
│                              │ true   │ true        │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-8  │ [PASS] │ [PASS]      │
│                              │ false  │ false       │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-9  │ [PASS] │ [PASS]      │
│                              │ true   │ true        │
├──────────────────────────────┼────────┼─────────────┤
│ dataset-spam_eval_dataset-10 │ [PASS] │ [PASS]      │
│                              │ false  │ false       │
└──────────────────────────────┴────────┴─────────────┘
Summary: GPT-4o: 10/10 pass | GPT-4o-mini: 10/10 pass
Legend: [PASS]=pass, [FAIL]=fail, [SKIP]=skipped, [TIE]=tie
```
2025-12-16 12:58:43 -03:00
..
runners FEATURE: Use evals to compare LLMs and Personas' prompts (#36027) 2025-11-18 10:39:52 -03:00
support
console_formatter_spec.rb FEATURE: Format eval comparison results into a table (#36724) 2025-12-16 12:58:43 -03:00
eval_spec.rb FEATURE: Run eval comparisons against a dataset (#36223) 2025-11-28 14:37:55 -03:00
features_spec.rb
judge_spec.rb FEATURE: Run eval comparisons against a dataset (#36223) 2025-11-28 14:37:55 -03:00
llm_repository_spec.rb
persona_prompt_loader_spec.rb FEATURE: Use evals to compare LLMs and Personas' prompts (#36027) 2025-11-18 10:39:52 -03:00
recorder_spec.rb FEATURE: Format eval comparison results into a table (#36724) 2025-12-16 12:58:43 -03:00
workbench_compare_spec.rb FEATURE: Format eval comparison results into a table (#36724) 2025-12-16 12:58:43 -03:00
workbench_spec.rb FEATURE: Format eval comparison results into a table (#36724) 2025-12-16 12:58:43 -03:00