mirror of
https://gh.wpcy.net/https://github.com/discourse/discourse.git
synced 2026-06-18 20:55:39 +08:00
31ff0071f8
Previously, 437ab337d2 added the normalized_referrer column to the
browser_pageview_events table. That column stores the cleaned referrer
value used by the admin Top Referrers report, but existing
browser_pageview_events rows were not backfilled. Older rows still have
NULL normalized_referrer values, so historical dates can show little or
no referrer data even though the original referrer was recorded.
This PR backfills normalized_referrer for existing
browser_pageview_events rows and adds a mechanism to repeat that work if
the normalization rules change later.
Key technical changes:
1. Add the normalized_referrer_version column to the
browser_pageview_events table, which stores raw browser pageview events.
The new column records which version of the referrer normalization rules
processed each row. This gives the backfill a clear stopping condition
and lets a future rules change reprocess older rows by bumping the
version.
2. Replace the existing AggregateBrowserPageviewDailyRollups scheduled
job, which rebuilt browser pageview country and referrer daily rollups,
with the new MaintainBrowserPageviewRollups scheduled job. The new job
still keeps those rollups current, and it also backfills older
normalized referrers from the same execution path so separate jobs do
not rebuild the same rollups at the same time.
3. Backfill browser_pageview_events rows in batches so large sites do
not need to update all historical pageview events in one run. A day’s
referrer rollup is only rebuilt once every stale referrer row for that
day has been processed, so the Top Referrers report does not show
partial counts between batches.
4. Skip dates that no longer have source rows in the
browser_pageview_events table. Browser pageview events can be pruned by
cleanup, but daily rollups are the permanent report data. Before
rebuilding a date, the backfill checks that browser_pageview_events
still has events for that date. If no events remain, the existing rollup
is left untouched because the job can no longer safely reconstruct it.
14 lines
384 B
Ruby
Vendored
14 lines
384 B
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
Fabricator(:browser_pageview_event) do
|
|
url "https://example.com/"
|
|
ip_address "1.2.3.4"
|
|
user_agent "test"
|
|
session_id { SecureRandom.hex(16) }
|
|
end
|
|
|
|
Fabricator(:browser_pageview_event_with_unnormalized_referrer, from: :browser_pageview_event) do
|
|
referrer "https://www.example.com/"
|
|
normalized_referrer nil
|
|
normalized_referrer_version nil
|
|
end
|
