0
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2026-08-05 19:38:04 +08:00
discourse/plugins/discourse-data-explorer/plugin.rb
Régis Hanol 527a13566f
DEV: Add usage metrics for workflows, automations, and data explorer (#41437)
Adds basic usage metrics for **workflows**, **automations**, and **data
explorer** so we can measure adoption across all sites, following the
pattern established for Kanban (dev topic `t/185911`).

Each plugin registers stats via the freeform `stat_type` API, so the
values flow automatically into our site statistics pipeline. No
infrastructure changes are needed.

## Metrics

For each feature: `total` (running count), `created`, `edited`,
`executed` (distinct objects run), and `executions` (total run count) —
each windowed over `last_day / 7_days / 30_days / previous_30_days`,
with a lifetime `count` on `total` and `executions`. Resulting columns:

| | Workflows | Automations | Data Explorer |
|---|---|---|---|
| total | `workflows_total_count` | `automations_total_count` |
`de_queries_total_count` |
| created | `workflows_created_*` | `automations_created_*` |
`de_queries_created_*` |
| edited | `workflows_edited_*` | `automations_edited_*` |
`de_queries_edited_*` |
| executed | `workflows_executed_*` | `automations_executed_*` |
`de_queries_executed_*` |
| executions | `workflows_executions_*` | `automations_executions_*` |
`de_executions_*` |

## How executions are counted

Automations already have a daily rollup (`discourse_automation_stats`),
so they reuse it.

Workflows and Data Explorer get a small daily rollup table each
(`discourse_workflows_execution_stats`, `data_explorer_query_stats`)
recording `total_runs` per object per day. This deliberately avoids two
traps:

- **Workflow executions are purged** after
`workflow_executions_retention_days` (default 30), and the purged rows
carry heavy payloads (full node-graph snapshot + step I/O). Counting
from a lightweight rollup keeps `previous_30_days`/lifetime metrics
valid without retaining those payloads, and decouples the metrics from a
per-site retention setting.
- **Data Explorer has no run log** — only a single `last_run_at`. The
rollup gives us real execution counts and a durable `previous_30_days`
window.

Workflow runs are recorded via an `after_create` on `Execution` (one row
per execution, rate-limited executions excluded). Data Explorer runs go
through a new `Query#record_run!` used by the three run sites.

## Data Explorer `updated_at` fix

Query runs previously did `query.update!(last_run_at:)`, which bumped
`updated_at` on every run and made "edited" indistinguishable from
"executed". `record_run!` now uses `update_columns`, so `updated_at`
reflects genuine edits again. Default (unpersisted) queries are still
persisted on first run as before, and the `DeleteHiddenQueries` job is
unaffected (it also gates on `last_run_at`).

## Core change

`register_stat` deduplicated by name only, so the three plugins couldn't
all register generic names like `total`/`executions` — whichever
activated first won and the rest were silently dropped. The identity
check now includes `stat_type`, so a name can be reused across stat
types (columns stay unique because they are prefixed with the stat
type). Kanban avoided this only because its names happened to be
globally unique.

## Notes for review

- **Single `de` stat_type** (yielding `de_queries_*` and
`de_executions_*`) rather than splitting into
`de_queries`/`de_executions` — happy to change if you'd rather query by
two separate `stat_type_freeform` values. cc measurement folks.
- `structure.sql` and model annotations were hand-updated (local box is
on PG17, which can't run the temp-DB dump/annotate tasks); CI will
verify.

## Tests

- New specs for all three `Statistics` modules and both rollup models
(`.log`, the `after_create`/`record_run!` hooks, the `updated_at`
behavior).
- New core spec covering same-name/different-`stat_type` registration.
2026-07-13 11:41:58 +02:00

234 lines
8.4 KiB
Ruby
Vendored

# frozen_string_literal: true
# name: discourse-data-explorer
# about: Allows you to make SQL queries against your live database, allowing for up-to-the-minute stats reporting.
# meta_topic_id: 32566
# version: 0.3
# authors: Riking
# url: https://github.com/discourse/discourse/tree/main/plugins/discourse-data-explorer
enabled_site_setting :data_explorer_enabled
register_asset "stylesheets/explorer.scss"
register_svg_icon "angle-down"
register_svg_icon "angle-right"
register_svg_icon "chart-line"
register_svg_icon "angle-left"
register_svg_icon "circle-exclamation"
register_svg_icon "info"
register_svg_icon "pencil"
register_svg_icon "upload"
add_admin_route "explorer.title", "discourse-data-explorer", use_new_show_route: true
module ::DiscourseDataExplorer
PLUGIN_NAME = "discourse-data-explorer"
# This should always match the max value for the
# data_explorer_query_result_limit site setting
QUERY_RESULT_MAX_LIMIT = 10_000
end
require_relative "lib/discourse_data_explorer/engine"
after_initialize do
GlobalSetting.add_default(:max_data_explorer_api_reqs_per_10_seconds, 2)
# Available options:
# - warn
# - warn+block
# - block
GlobalSetting.add_default(:max_data_explorer_api_req_mode, "warn")
if respond_to?(:register_discourse_workflows_node)
register_svg_icon "database"
register_discourse_workflows_node do
require_relative "lib/discourse_data_explorer/workflows/sql_action/v1"
DiscourseDataExplorer::Workflows::SqlAction::V1
end
end
add_to_class(:guardian, :user_is_a_member_of_group?) do |group|
return false if !current_user
return true if current_user.admin?
current_user.group_ids.include?(group.id)
end
add_to_class(:guardian, :user_can_access_query?) do |query|
return false if !current_user
return true if current_user.admin?
query.groups.any? { |group| user_is_a_member_of_group?(group) }
end
add_to_class(:guardian, :group_and_user_can_access_query?) do |group, query|
return false if !current_user
return true if current_user.admin?
user_is_a_member_of_group?(group) && query.groups.exists?(id: group.id)
end
add_to_serializer(
:group_show,
:has_visible_data_explorer_queries,
include_condition: -> { scope.user_is_a_member_of_group?(object) },
) { DiscourseDataExplorer::Query.for_group(object).exists? }
register_bookmarkable(DiscourseDataExplorer::QueryGroupBookmarkable)
register_admin_dashboard_report_source(DiscourseDataExplorer::AdminDashboardReportProvider)
add_api_key_scope(
:data_explorer,
{
run_queries: {
actions: %w[discourse_data_explorer/query#run discourse_data_explorer/query#public_run],
params: %i[id],
},
},
)
reloadable_patch do
if defined?(DiscourseAutomation)
add_automation_scriptable("recurring_data_explorer_result_pm") do
queries =
DiscourseDataExplorer::Query
.where(hidden: false)
.map { |q| { id: q.id, translated_name: q.name } }
field :recipients, component: :email_group_user, required: true
field :query_id, component: :choices, required: true, extra: { content: queries }
field :query_params, component: :"key-value", accepts_placeholders: true
field :skip_empty, component: :boolean
field :users_from_group, component: :boolean
field :attach_csv,
component: :boolean,
validator: ->(attach_csv) do
return if !attach_csv
extensions = SiteSetting.authorized_extensions.split("|")
if (extensions & %w[csv *]).empty?
I18n.t(
"discourse_automation.scriptables.recurring_data_explorer_result_pm.no_csv_allowed",
)
end
end
version 1
triggerables [:recurring]
script do |_, fields, automation|
recipients = Array(fields.dig("recipients", "value")).uniq
query_id = fields.dig("query_id", "value")
query_params = fields.dig("query_params", "value") || {}
skip_empty = fields.dig("skip_empty", "value") || false
users_from_group = fields.dig("users_from_group", "value") || false
attach_csv = fields.dig("attach_csv", "value") || false
unless SiteSetting.data_explorer_enabled
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - plugin must be enabled to run automation #{automation.id}"
next
end
if recipients.blank?
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - couldn't find any recipients for automation #{automation.id}"
next
end
DiscourseDataExplorer::ReportGenerator
.generate(
query_id,
query_params,
recipients,
{ skip_empty:, users_from_group:, attach_csv:, render_url_columns: true },
)
.each do |pm|
utils.send_pm(pm, automation_id: automation.id)
rescue ActiveRecord::RecordNotSaved => e
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - couldn't send PM for automation #{automation.id}: #{e.message}"
end
end
end
add_automation_scriptable("recurring_data_explorer_result_topic") do
queries =
DiscourseDataExplorer::Query
.where(hidden: false)
.map { |q| { id: q.id, translated_name: q.name } }
field :topic_id, component: :text, required: true
field :query_id, component: :choices, required: true, extra: { content: queries }
field :query_params, component: :"key-value", accepts_placeholders: true
field :skip_empty, component: :boolean
field :attach_csv, component: :boolean
version 1
triggerables [:recurring]
script do |_, fields, automation|
topic_id = fields.dig("topic_id", "value")
query_id = fields.dig("query_id", "value")
query_params = fields.dig("query_params", "value") || {}
skip_empty = fields.dig("skip_empty", "value") || false
attach_csv = fields.dig("attach_csv", "value") || false
unless SiteSetting.data_explorer_enabled
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - plugin must be enabled to run automation #{automation.id}"
next
end
topic = Topic.find_by(id: topic_id)
if topic.blank?
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - couldn't find topic ID (#{topic_id}) for automation #{automation.id}"
next
end
begin
post =
DiscourseDataExplorer::ReportGenerator.generate_post(
query_id,
query_params,
{ skip_empty:, attach_csv:, render_url_columns: true },
)
next if post.empty?
PostCreator.create!(
Discourse.system_user,
topic_id: topic.id,
raw: post["raw"],
skip_validations: true,
)
rescue ActiveRecord::RecordNotSaved => e
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - couldn't reply to topic ID #{topic_id} for automation #{automation.id}: #{e.message}"
end
end
end
end
end
if defined?(DiscourseAi)
require_relative "lib/discourse_data_explorer/ai_query_params"
require_relative "lib/discourse_data_explorer/tools/find_queries"
require_relative "lib/discourse_data_explorer/tools/run_sql"
require_relative "lib/discourse_data_explorer/tools/submit_query"
require_relative "lib/discourse_data_explorer/ai_query_generator"
DiscourseAi.register_feature(
module_name: :data_explorer,
feature: :query_generation,
agent_klass: DiscourseDataExplorer::AiQueryGenerator,
enabled_by_setting: "data_explorer_ai_queries_enabled",
plugin: self,
)
end
register_stat("queries_total", stat_type: :de) { DiscourseDataExplorer::Statistics.queries_total }
register_stat("queries_created", stat_type: :de) do
DiscourseDataExplorer::Statistics.queries_created
end
register_stat("queries_edited", stat_type: :de) do
DiscourseDataExplorer::Statistics.queries_edited
end
register_stat("queries_executed", stat_type: :de) do
DiscourseDataExplorer::Statistics.queries_executed
end
register_stat("executions", stat_type: :de) { DiscourseDataExplorer::Statistics.executions }
end