0
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2026-08-06 09:44:06 +08:00
discourse/app/models/browser_pageview_crawler_daily_rollup.rb
Krzysztof Kotlarek bf8d011bdd
FEATURE: Reclassify likely crawlers in site traffic reporting (#42199)
Previously, CrawlerScorer wrote a score to every browser pageview event,
but nothing consumed it, so the site traffic chart still counted
automated sessions as logged-in or anonymous humans and overstated
community traffic.

Classify events scoring above CrawlerScorer::BOT_SCORE_THRESHOLD, as
likely crawlers. Their daily counts are rolled up per logged-in state
into a new browser_pageview_crawler_daily_rollups table, subtracted from
the logged-in and anonymous series, and shown as their own "Likely
crawlers" series ahead of the known-crawler one.

All of this sits behind the new improved_crawler_detection upcoming
change, which also replaces experimental_detect_crawler_pageviews as the
gate on scoring. The rollup job backfills all existing history on its
first run, then refreshes yesterday and today. Disabling the change
restores the original counters and hides the series while leaving the
rollups in place, so re-enabling takes effect without a backfill.

<img width="993" height="755" alt="Screenshot 2026-07-31 at 12 22 13 pm"
src="https://github.com/user-attachments/assets/6163b731-c2ba-4e53-8cc7-4eba78f4190d"
/>
2026-08-04 10:08:53 +08:00

38 lines
1.1 KiB
Ruby
Vendored

# frozen_string_literal: true
class BrowserPageviewCrawlerDailyRollup < ActiveRecord::Base
def self.aggregate(start_date:, end_date:)
start_date = start_date.to_date
end_date = end_date.to_date + 1
DB.exec(<<~SQL, start_date:, end_date:, threshold: CrawlerScorer::BOT_SCORE_THRESHOLD)
INSERT INTO browser_pageview_crawler_daily_rollups (date, logged_in, count)
SELECT
created_at::date AS date,
user_id IS NOT NULL AS logged_in,
COUNT(*) AS count
FROM browser_pageview_events
WHERE created_at >= :start_date
AND created_at < :end_date
AND score > :threshold
AND #{BrowserPageviewEvent.rollup_source_condition}
GROUP BY date, logged_in
ON CONFLICT (date, logged_in) DO UPDATE
SET count = EXCLUDED.count
SQL
end
end
# == Schema Information
#
# Table name: browser_pageview_crawler_daily_rollups
#
# id :bigint not null, primary key
# count :bigint not null
# date :date not null
# logged_in :boolean not null
#
# Indexes
#
# idx_bpcrawler_rollups_date_logged_in_unique (date,logged_in) UNIQUE
#