Discourse/discourse
Discourse/discourse
0
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2026-08-06 08:06:54 +08:00
Code Packages Activity Actions
discourse/app/jobs/scheduled
History
Exact
Krzysztof Kotlarek bf8d011bdd
 FEATURE: Reclassify likely crawlers in site traffic reporting (#42199) 
Previously, CrawlerScorer wrote a score to every browser pageview event,
but nothing consumed it, so the site traffic chart still counted
automated sessions as logged-in or anonymous humans and overstated
community traffic.

Classify events scoring above CrawlerScorer::BOT_SCORE_THRESHOLD, as
likely crawlers. Their daily counts are rolled up per logged-in state
into a new browser_pageview_crawler_daily_rollups table, subtracted from
the logged-in and anonymous series, and shown as their own "Likely
crawlers" series ahead of the known-crawler one.

All of this sits behind the new improved_crawler_detection upcoming
change, which also replaces experimental_detect_crawler_pageviews as the
gate on scoring. The rollup job backfills all existing history on its
first run, then refreshes yesterday and today. Disabling the change
restores the original counters and hides the series while leaving the
rollups in place, so re-enabling takes effect without a backfill.

<img width="993" height="755" alt="Screenshot 2026-07-31 at 12 22 13 pm"
src="https://github.com/user-attachments/assets/6163b731-c2ba-4e53-8cc7-4eba78f4190d"
/>
2026-08-04 10:08:53 +08:00
..
about_stats.rb
activation_reminder_emails.rb
aggregate_web_hooks_events.rb DEV: Enable Style/RedundantParentheses rubocop rule (#40095) 2026-05-19 15:48:09 +02:00
auto_queue_handler.rb DEV: replace .each with .find_each for paginated queries (#27159) 2024-05-23 15:42:21 -07:00
backfill_dominant_colors.rb
backfill_nested_reply_stats.rb PERF: Add fast nested reply stats preparation (#41870) 2026-07-21 10:45:53 -05:00
badge_grant.rb DEV: Enable Style/RedundantSelf rubocop rule (#40098) 2026-05-19 19:27:45 +02:00
bookmark_reminder_notifications.rb
calculate_scores.rb DEV: Split weekly job into multiple smaller jobs (#31260) 2025-02-11 10:26:45 +02:00
call_discourse_hub.rb FEATURE: Improve display of installed/available version on dashboard (#41165) 2026-06-24 13:07:05 +01:00
category_stats.rb
check_new_features.rb UX: suppress new feature notifications on brand new sites (#40135) 2026-05-19 09:38:57 +10:00
check_out_of_date_themes.rb
check_translation_overrides.rb FIX: Ensure JsLocaleHelper to not output deprecated translations (#28037) 2024-07-29 15:21:25 +08:00
check_upcoming_changes.rb FIX: Do not notify new sites or plugins enabled of old upcoming changes (#42045) 2026-07-30 09:51:08 +10:00
clean_dismissed_topic_users.rb
clean_up_api_keys_max_life.rb
clean_up_associated_accounts.rb
clean_up_associated_groups.rb
clean_up_bookmarks.rb DEV: Split weekly job into multiple smaller jobs (#31260) 2025-02-11 10:26:45 +02:00
clean_up_browser_pageview_events.rb DEV: Track browser session engagement for crawler detection (#41215) 2026-07-01 09:17:48 +08:00
clean_up_crawler_stats.rb
clean_up_drafts.rb DEV: Split weekly job into multiple smaller jobs (#31260) 2025-02-11 10:26:45 +02:00
clean_up_email_change_requests.rb
clean_up_email_login_codes.rb FEATURE: Log in with a one-time email code (#40804) 2026-06-17 12:34:48 -07:00
clean_up_email_logs.rb
clean_up_email_tokens.rb
clean_up_exports.rb
clean_up_inactive_users.rb DEV: add clean_up_inactive_users_query modifier for plugins to extend query for inactive users (#41372) 2026-07-06 09:39:11 -07:00
clean_up_post_reply_keys.rb
clean_up_search_logs.rb
clean_up_stylesheet_cache.rb
clean_up_tags.rb FIX: sort tag names in clean up tags staff log entry (#37909) 2026-02-19 14:43:58 +08:00
clean_up_unmatched_emails.rb
clean_up_unmatched_ips.rb
clean_up_unsubscribe_keys.rb
clean_up_unused_api_keys.rb
clean_up_unused_registered_user_api_key_clients.rb DEV: Fix assigned but unused variable Prism warnings (#39436) 2026-04-22 12:42:14 +02:00
clean_up_unused_staged_users.rb DEV: Enable Style/RedundantBegin rubocop rule (#40096) 2026-05-19 18:44:54 +02:00
clean_up_unused_user_api_keys.rb
clean_up_uploads.rb FIX: Clean up not secure uploads with access_control_post_id (#31189) 2025-02-06 11:26:34 +10:00
clean_up_user_api_keys_max_life.rb
clean_up_user_auth_tokens.rb DEV: Split weekly job into multiple smaller jobs (#31260) 2025-02-11 10:26:45 +02:00
cleanup_problem_check_trackers.rb FIX: Clear disabled/deleted ProblemChecks automatically (#42243) 2026-07-31 21:18:28 +01:00
cleanup_redelivering_web_hook_events.rb DEV: Update rubocop-discourse to 3.13 and autofix issues (#35073) 2025-10-06 16:11:01 +02:00
clear_expired_impersonations.rb DEV: Allow impersonation without session swapping (#34213) 2025-08-21 14:18:15 +08:00
create_missing_avatars.rb
create_recent_post_search_indexes.rb DEV: Deprecate the assignment of nil to site settings (#36093) 2025-12-01 15:04:23 +08:00
delete_rejected_emails.rb DEV: Split weekly job into multiple smaller jobs (#31260) 2025-02-11 10:26:45 +02:00
destroy_old_deletion_stubs.rb
destroy_old_hidden_posts.rb
detect_crawler_pageviews.rb FEATURE: Reclassify likely crawlers in site traffic reporting (#42199) 2026-08-04 10:08:53 +08:00
directory_refresh.rb DEV: refresh user directory more often for sites with few users (#39852) 2026-05-08 12:04:03 -04:00
drop_backup_schema.rb
enqueue_digest_emails.rb FIX: send activity summaries based on "last seen" (#27035) 2024-05-22 10:23:03 +02:00
enqueue_onceoffs.rb
enqueue_suspect_users.rb FIX: Exclude suspended users from suspect users review queue (#37796) 2026-02-16 10:46:52 +08:00
ensure_badges_consistency.rb PERF: Enqueue Job::BackfillBadge in Jobs::BadgeGrant (#30945) 2025-01-24 09:35:01 +08:00
ensure_db_consistency.rb DEV: Rescue from each step of EnsureDbConsistency (#31313) 2025-02-13 12:07:04 +02:00
ensure_s3_uploads_existence.rb FIX: Jobs::EnsureS3UploadsExistence broken for multisite (#27401) 2024-06-10 16:26:39 +08:00
fix_user_usernames_and_groups_names_clash.rb DEV: Hand-pick Rails/WhereNot autofixes (#35117) 2025-10-03 13:29:22 +02:00
flush_browser_pageview_events.rb PERF: Batch BrowserPageviewEvent inserts through Redis (#40328) 2026-06-22 10:19:01 +08:00
grant_anniversary_badges.rb
grant_new_user_of_the_month_badges.rb
heartbeat.rb FIX: Redo Sidekiq monitoring to restart stuck sidekiq processes (#30198) 2024-12-18 12:48:50 +08:00
ignored_users_summary.rb
invalidate_inactive_admins.rb
maintain_browser_pageview_rollups.rb FEATURE: Reclassify likely crawlers in site traffic reporting (#42199) 2026-08-04 10:08:53 +08:00
maintain_category_activity_daily_rollups.rb PERF: Serve the activity_by_category report from daily rollups (#42046) 2026-07-28 12:53:29 +08:00
maintain_user_visit_daily_rollups.rb PERF: Use daily rollups for DAU/MAU reports (#41958) 2026-07-24 09:52:40 +08:00
migrate_upload_scheme.rb
notify_admins_of_available_upcoming_changes.rb FIX: Do not send notifications for upcoming changes not displayed (#40933) 2026-06-17 09:10:49 +10:00
notify_admins_of_problems.rb FEATURE: Add problem checks page to admin panel and allow ignoring problem checks (#39103) 2026-04-23 08:28:33 +08:00
old_keys_reminder.rb DEV: Update rubocop-discourse to 3.13 and autofix issues (#35073) 2025-10-06 16:11:01 +02:00
pending_queued_posts_reminder.rb
pending_reviewables_reminder.rb
pending_users_reminder.rb Revert "FEATURE: discourse-workflows (#39704)" (#40366) 2026-05-28 13:59:42 +01:00
periodical_updates.rb FEATURE: Support markdown in tag descriptions (#41996) 2026-07-29 14:43:47 +01:00
poll_mailbox.rb DEV: Database backed admin notices (#26192) 2024-05-23 09:29:08 +08:00
presence_channel_auto_leave.rb
process_badge_backlog.rb
process_shelved_notifications.rb
process_user_notification_schedules.rb DEV: Enable Style/RedundantBegin rubocop rule (#40096) 2026-05-19 18:44:54 +02:00
purge_deleted_uploads.rb
purge_expired_ignored_users.rb
purge_old_mini_scheduler_stat.rb DEV: Split weekly job into multiple smaller jobs (#31260) 2025-02-11 10:26:45 +02:00
purge_old_notifications.rb DEV: Split weekly job into multiple smaller jobs (#31260) 2025-02-11 10:26:45 +02:00
purge_old_web_hook_events.rb
purge_old_web_hook_events_daily_aggregate.rb FEATURE: Add WebHookEventsDailyAggregate (#27542) 2024-06-25 13:56:47 -03:00
purge_unactivated.rb
rebuild_category_activity_daily_rollups.rb PERF: Serve the activity_by_category report from daily rollups (#42046) 2026-07-28 12:53:29 +08:00
recalculate_nested_hot_scores.rb FEATURE: Hot algorithm for nested replies (simple) (#41742) 2026-07-20 10:56:05 -05:00
redeliver_web_hook_events.rb FIX: Match webhook-related messagebus audience to controller audience (#40693) 2026-06-09 18:22:26 +01:00
regenerate_sitemaps.rb
reindex_search.rb DEV: Move private method to bottom of file 2026-02-09 18:41:22 +00:00
reviewable_priorities.rb
run_problem_checks.rb FEATURE: Add Site Advice section to the redesigned admin dashboard (#40969) 2026-06-23 11:57:49 +02:00
schedule_backup.rb DEV: Rename Backuper to Creator 2026-01-22 18:46:02 +00:00
tl3_promotions.rb DEV: Add tl3_custom_promotions plugin modifier to tl3_promotions.rb (#29834) 2024-11-22 15:28:43 -05:00
top_refresh_older.rb
top_refresh_today.rb
topic_timer_enqueuer.rb DEV: Enable Style/RedundantBegin rubocop rule (#40096) 2026-05-19 18:44:54 +02:00
unsilence_users.rb
update_animated_uploads.rb DEV: Remove logical OR assignment of constants (#29201) 2024-10-16 10:09:07 +08:00
update_heat_settings.rb
update_topic_hot_scores.rb DEV: Update rubocop-discourse to 3.13 and autofix issues (#35073) 2025-10-06 16:11:01 +02:00