mirror of
https://github.com/discourse/discourse.git
synced 2026-08-05 19:38:04 +08:00
Adds basic usage metrics for **workflows**, **automations**, and **data explorer** so we can measure adoption across all sites, following the pattern established for Kanban (dev topic `t/185911`). Each plugin registers stats via the freeform `stat_type` API, so the values flow automatically into our site statistics pipeline. No infrastructure changes are needed. ## Metrics For each feature: `total` (running count), `created`, `edited`, `executed` (distinct objects run), and `executions` (total run count) — each windowed over `last_day / 7_days / 30_days / previous_30_days`, with a lifetime `count` on `total` and `executions`. Resulting columns: | | Workflows | Automations | Data Explorer | |---|---|---|---| | total | `workflows_total_count` | `automations_total_count` | `de_queries_total_count` | | created | `workflows_created_*` | `automations_created_*` | `de_queries_created_*` | | edited | `workflows_edited_*` | `automations_edited_*` | `de_queries_edited_*` | | executed | `workflows_executed_*` | `automations_executed_*` | `de_queries_executed_*` | | executions | `workflows_executions_*` | `automations_executions_*` | `de_executions_*` | ## How executions are counted Automations already have a daily rollup (`discourse_automation_stats`), so they reuse it. Workflows and Data Explorer get a small daily rollup table each (`discourse_workflows_execution_stats`, `data_explorer_query_stats`) recording `total_runs` per object per day. This deliberately avoids two traps: - **Workflow executions are purged** after `workflow_executions_retention_days` (default 30), and the purged rows carry heavy payloads (full node-graph snapshot + step I/O). Counting from a lightweight rollup keeps `previous_30_days`/lifetime metrics valid without retaining those payloads, and decouples the metrics from a per-site retention setting. - **Data Explorer has no run log** — only a single `last_run_at`. The rollup gives us real execution counts and a durable `previous_30_days` window. Workflow runs are recorded via an `after_create` on `Execution` (one row per execution, rate-limited executions excluded). Data Explorer runs go through a new `Query#record_run!` used by the three run sites. ## Data Explorer `updated_at` fix Query runs previously did `query.update!(last_run_at:)`, which bumped `updated_at` on every run and made "edited" indistinguishable from "executed". `record_run!` now uses `update_columns`, so `updated_at` reflects genuine edits again. Default (unpersisted) queries are still persisted on first run as before, and the `DeleteHiddenQueries` job is unaffected (it also gates on `last_run_at`). ## Core change `register_stat` deduplicated by name only, so the three plugins couldn't all register generic names like `total`/`executions` — whichever activated first won and the rest were silently dropped. The identity check now includes `stat_type`, so a name can be reused across stat types (columns stay unique because they are prefixed with the stat type). Kanban avoided this only because its names happened to be globally unique. ## Notes for review - **Single `de` stat_type** (yielding `de_queries_*` and `de_executions_*`) rather than splitting into `de_queries`/`de_executions` — happy to change if you'd rather query by two separate `stat_type_freeform` values. cc measurement folks. - `structure.sql` and model annotations were hand-updated (local box is on PG17, which can't run the temp-DB dump/annotate tasks); CI will verify. ## Tests - New specs for all three `Statistics` modules and both rollup models (`.log`, the `after_create`/`record_run!` hooks, the `updated_at` behavior). - New core spec covering same-name/different-`stat_type` registration.
234 lines
8.4 KiB
Ruby
Vendored
234 lines
8.4 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
# name: discourse-data-explorer
|
|
# about: Allows you to make SQL queries against your live database, allowing for up-to-the-minute stats reporting.
|
|
# meta_topic_id: 32566
|
|
# version: 0.3
|
|
# authors: Riking
|
|
# url: https://github.com/discourse/discourse/tree/main/plugins/discourse-data-explorer
|
|
|
|
enabled_site_setting :data_explorer_enabled
|
|
|
|
register_asset "stylesheets/explorer.scss"
|
|
|
|
register_svg_icon "angle-down"
|
|
register_svg_icon "angle-right"
|
|
register_svg_icon "chart-line"
|
|
register_svg_icon "angle-left"
|
|
register_svg_icon "circle-exclamation"
|
|
register_svg_icon "info"
|
|
register_svg_icon "pencil"
|
|
register_svg_icon "upload"
|
|
|
|
add_admin_route "explorer.title", "discourse-data-explorer", use_new_show_route: true
|
|
|
|
module ::DiscourseDataExplorer
|
|
PLUGIN_NAME = "discourse-data-explorer"
|
|
|
|
# This should always match the max value for the
|
|
# data_explorer_query_result_limit site setting
|
|
QUERY_RESULT_MAX_LIMIT = 10_000
|
|
end
|
|
|
|
require_relative "lib/discourse_data_explorer/engine"
|
|
|
|
after_initialize do
|
|
GlobalSetting.add_default(:max_data_explorer_api_reqs_per_10_seconds, 2)
|
|
|
|
# Available options:
|
|
# - warn
|
|
# - warn+block
|
|
# - block
|
|
GlobalSetting.add_default(:max_data_explorer_api_req_mode, "warn")
|
|
|
|
if respond_to?(:register_discourse_workflows_node)
|
|
register_svg_icon "database"
|
|
register_discourse_workflows_node do
|
|
require_relative "lib/discourse_data_explorer/workflows/sql_action/v1"
|
|
DiscourseDataExplorer::Workflows::SqlAction::V1
|
|
end
|
|
end
|
|
|
|
add_to_class(:guardian, :user_is_a_member_of_group?) do |group|
|
|
return false if !current_user
|
|
return true if current_user.admin?
|
|
current_user.group_ids.include?(group.id)
|
|
end
|
|
|
|
add_to_class(:guardian, :user_can_access_query?) do |query|
|
|
return false if !current_user
|
|
return true if current_user.admin?
|
|
query.groups.any? { |group| user_is_a_member_of_group?(group) }
|
|
end
|
|
|
|
add_to_class(:guardian, :group_and_user_can_access_query?) do |group, query|
|
|
return false if !current_user
|
|
return true if current_user.admin?
|
|
user_is_a_member_of_group?(group) && query.groups.exists?(id: group.id)
|
|
end
|
|
|
|
add_to_serializer(
|
|
:group_show,
|
|
:has_visible_data_explorer_queries,
|
|
include_condition: -> { scope.user_is_a_member_of_group?(object) },
|
|
) { DiscourseDataExplorer::Query.for_group(object).exists? }
|
|
|
|
register_bookmarkable(DiscourseDataExplorer::QueryGroupBookmarkable)
|
|
|
|
register_admin_dashboard_report_source(DiscourseDataExplorer::AdminDashboardReportProvider)
|
|
|
|
add_api_key_scope(
|
|
:data_explorer,
|
|
{
|
|
run_queries: {
|
|
actions: %w[discourse_data_explorer/query#run discourse_data_explorer/query#public_run],
|
|
params: %i[id],
|
|
},
|
|
},
|
|
)
|
|
|
|
reloadable_patch do
|
|
if defined?(DiscourseAutomation)
|
|
add_automation_scriptable("recurring_data_explorer_result_pm") do
|
|
queries =
|
|
DiscourseDataExplorer::Query
|
|
.where(hidden: false)
|
|
.map { |q| { id: q.id, translated_name: q.name } }
|
|
field :recipients, component: :email_group_user, required: true
|
|
field :query_id, component: :choices, required: true, extra: { content: queries }
|
|
field :query_params, component: :"key-value", accepts_placeholders: true
|
|
field :skip_empty, component: :boolean
|
|
field :users_from_group, component: :boolean
|
|
field :attach_csv,
|
|
component: :boolean,
|
|
validator: ->(attach_csv) do
|
|
return if !attach_csv
|
|
|
|
extensions = SiteSetting.authorized_extensions.split("|")
|
|
if (extensions & %w[csv *]).empty?
|
|
I18n.t(
|
|
"discourse_automation.scriptables.recurring_data_explorer_result_pm.no_csv_allowed",
|
|
)
|
|
end
|
|
end
|
|
|
|
version 1
|
|
triggerables [:recurring]
|
|
|
|
script do |_, fields, automation|
|
|
recipients = Array(fields.dig("recipients", "value")).uniq
|
|
query_id = fields.dig("query_id", "value")
|
|
query_params = fields.dig("query_params", "value") || {}
|
|
skip_empty = fields.dig("skip_empty", "value") || false
|
|
users_from_group = fields.dig("users_from_group", "value") || false
|
|
attach_csv = fields.dig("attach_csv", "value") || false
|
|
|
|
unless SiteSetting.data_explorer_enabled
|
|
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - plugin must be enabled to run automation #{automation.id}"
|
|
next
|
|
end
|
|
|
|
if recipients.blank?
|
|
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - couldn't find any recipients for automation #{automation.id}"
|
|
next
|
|
end
|
|
|
|
DiscourseDataExplorer::ReportGenerator
|
|
.generate(
|
|
query_id,
|
|
query_params,
|
|
recipients,
|
|
{ skip_empty:, users_from_group:, attach_csv:, render_url_columns: true },
|
|
)
|
|
.each do |pm|
|
|
utils.send_pm(pm, automation_id: automation.id)
|
|
rescue ActiveRecord::RecordNotSaved => e
|
|
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - couldn't send PM for automation #{automation.id}: #{e.message}"
|
|
end
|
|
end
|
|
end
|
|
|
|
add_automation_scriptable("recurring_data_explorer_result_topic") do
|
|
queries =
|
|
DiscourseDataExplorer::Query
|
|
.where(hidden: false)
|
|
.map { |q| { id: q.id, translated_name: q.name } }
|
|
field :topic_id, component: :text, required: true
|
|
field :query_id, component: :choices, required: true, extra: { content: queries }
|
|
field :query_params, component: :"key-value", accepts_placeholders: true
|
|
field :skip_empty, component: :boolean
|
|
field :attach_csv, component: :boolean
|
|
|
|
version 1
|
|
triggerables [:recurring]
|
|
|
|
script do |_, fields, automation|
|
|
topic_id = fields.dig("topic_id", "value")
|
|
query_id = fields.dig("query_id", "value")
|
|
query_params = fields.dig("query_params", "value") || {}
|
|
skip_empty = fields.dig("skip_empty", "value") || false
|
|
attach_csv = fields.dig("attach_csv", "value") || false
|
|
|
|
unless SiteSetting.data_explorer_enabled
|
|
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - plugin must be enabled to run automation #{automation.id}"
|
|
next
|
|
end
|
|
|
|
topic = Topic.find_by(id: topic_id)
|
|
if topic.blank?
|
|
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - couldn't find topic ID (#{topic_id}) for automation #{automation.id}"
|
|
next
|
|
end
|
|
|
|
begin
|
|
post =
|
|
DiscourseDataExplorer::ReportGenerator.generate_post(
|
|
query_id,
|
|
query_params,
|
|
{ skip_empty:, attach_csv:, render_url_columns: true },
|
|
)
|
|
|
|
next if post.empty?
|
|
|
|
PostCreator.create!(
|
|
Discourse.system_user,
|
|
topic_id: topic.id,
|
|
raw: post["raw"],
|
|
skip_validations: true,
|
|
)
|
|
rescue ActiveRecord::RecordNotSaved => e
|
|
Rails.logger.warn "#{DiscourseDataExplorer::PLUGIN_NAME} - couldn't reply to topic ID #{topic_id} for automation #{automation.id}: #{e.message}"
|
|
end
|
|
end
|
|
end
|
|
end
|
|
end
|
|
|
|
if defined?(DiscourseAi)
|
|
require_relative "lib/discourse_data_explorer/ai_query_params"
|
|
require_relative "lib/discourse_data_explorer/tools/find_queries"
|
|
require_relative "lib/discourse_data_explorer/tools/run_sql"
|
|
require_relative "lib/discourse_data_explorer/tools/submit_query"
|
|
require_relative "lib/discourse_data_explorer/ai_query_generator"
|
|
|
|
DiscourseAi.register_feature(
|
|
module_name: :data_explorer,
|
|
feature: :query_generation,
|
|
agent_klass: DiscourseDataExplorer::AiQueryGenerator,
|
|
enabled_by_setting: "data_explorer_ai_queries_enabled",
|
|
plugin: self,
|
|
)
|
|
end
|
|
|
|
register_stat("queries_total", stat_type: :de) { DiscourseDataExplorer::Statistics.queries_total }
|
|
register_stat("queries_created", stat_type: :de) do
|
|
DiscourseDataExplorer::Statistics.queries_created
|
|
end
|
|
register_stat("queries_edited", stat_type: :de) do
|
|
DiscourseDataExplorer::Statistics.queries_edited
|
|
end
|
|
register_stat("queries_executed", stat_type: :de) do
|
|
DiscourseDataExplorer::Statistics.queries_executed
|
|
end
|
|
register_stat("executions", stat_type: :de) { DiscourseDataExplorer::Statistics.executions }
|
|
end
|