mirror of
https://github.com/discourse/discourse.git
synced 2026-08-06 13:08:40 +08:00
Previously, CrawlerScorer wrote a score to every browser pageview event, but nothing consumed it, so the site traffic chart still counted automated sessions as logged-in or anonymous humans and overstated community traffic. Classify events scoring above CrawlerScorer::BOT_SCORE_THRESHOLD, as likely crawlers. Their daily counts are rolled up per logged-in state into a new browser_pageview_crawler_daily_rollups table, subtracted from the logged-in and anonymous series, and shown as their own "Likely crawlers" series ahead of the known-crawler one. All of this sits behind the new improved_crawler_detection upcoming change, which also replaces experimental_detect_crawler_pageviews as the gate on scoring. The rollup job backfills all existing history on its first run, then refreshes yesterday and today. Disabling the change restores the original counters and hides the series while leaving the rollups in place, so re-enabling takes effect without a backfill. <img width="993" height="755" alt="Screenshot 2026-07-31 at 12 22 13 pm" src="https://github.com/user-attachments/assets/6163b731-c2ba-4e53-8cc7-4eba78f4190d" />
38 lines
1.1 KiB
Ruby
Vendored
38 lines
1.1 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
class BrowserPageviewCrawlerDailyRollup < ActiveRecord::Base
|
|
def self.aggregate(start_date:, end_date:)
|
|
start_date = start_date.to_date
|
|
end_date = end_date.to_date + 1
|
|
|
|
DB.exec(<<~SQL, start_date:, end_date:, threshold: CrawlerScorer::BOT_SCORE_THRESHOLD)
|
|
INSERT INTO browser_pageview_crawler_daily_rollups (date, logged_in, count)
|
|
SELECT
|
|
created_at::date AS date,
|
|
user_id IS NOT NULL AS logged_in,
|
|
COUNT(*) AS count
|
|
FROM browser_pageview_events
|
|
WHERE created_at >= :start_date
|
|
AND created_at < :end_date
|
|
AND score > :threshold
|
|
AND #{BrowserPageviewEvent.rollup_source_condition}
|
|
GROUP BY date, logged_in
|
|
ON CONFLICT (date, logged_in) DO UPDATE
|
|
SET count = EXCLUDED.count
|
|
SQL
|
|
end
|
|
end
|
|
|
|
# == Schema Information
|
|
#
|
|
# Table name: browser_pageview_crawler_daily_rollups
|
|
#
|
|
# id :bigint not null, primary key
|
|
# count :bigint not null
|
|
# date :date not null
|
|
# logged_in :boolean not null
|
|
#
|
|
# Indexes
|
|
#
|
|
# idx_bpcrawler_rollups_date_logged_in_unique (date,logged_in) UNIQUE
|
|
#
|