0
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2026-08-06 02:51:10 +08:00
discourse/spec/jobs
Krzysztof Kotlarek bf8d011bdd
FEATURE: Reclassify likely crawlers in site traffic reporting (#42199)
Previously, CrawlerScorer wrote a score to every browser pageview event,
but nothing consumed it, so the site traffic chart still counted
automated sessions as logged-in or anonymous humans and overstated
community traffic.

Classify events scoring above CrawlerScorer::BOT_SCORE_THRESHOLD, as
likely crawlers. Their daily counts are rolled up per logged-in state
into a new browser_pageview_crawler_daily_rollups table, subtracted from
the logged-in and anonymous series, and shown as their own "Likely
crawlers" series ahead of the known-crawler one.

All of this sits behind the new improved_crawler_detection upcoming
change, which also replaces experimental_detect_crawler_pageviews as the
gate on scoring. The rollup job backfills all existing history on its
first run, then refreshes yesterday and today. Disabling the change
restores the original counters and hides the series while leaving the
rollups in place, so re-enabling takes effect without a backfill.

<img width="993" height="755" alt="Screenshot 2026-07-31 at 12 22 13 pm"
src="https://github.com/user-attachments/assets/6163b731-c2ba-4e53-8cc7-4eba78f4190d"
/>
2026-08-04 10:08:53 +08:00
..
regular FEATURE: Support users in DAccessControl and backend (#41358) 2026-07-15 09:18:06 +10:00
scheduled FIX: Do not notify new sites or plugins enabled of old upcoming changes (#42045) 2026-07-30 09:51:08 +10:00
about_stats_spec.rb DEV: Enable Style/RedundantBegin rubocop rule (#40096) 2026-05-19 18:44:54 +02:00
activation_reminder_emails_spec.rb
auto_queue_handler_spec.rb
automatic_group_membership_spec.rb
backfill_sidebar_site_settings_spec.rb
bookmark_reminder_notifications_spec.rb DEV: Enable Style/RedundantBegin rubocop rule (#40096) 2026-05-19 18:44:54 +02:00
bulk_grant_trust_level_spec.rb PERF: Add Users to Group Pt. 1 (#38737) 2026-03-26 09:11:22 -05:00
bulk_invite_spec.rb
bump_topic_spec.rb
call_discourse_hub_spec.rb FIX: Handle boolean values in version check response (#38786) 2026-03-23 15:39:38 +00:00
change_display_name_spec.rb FIX: Preserve display name in quotes when using the rich text editor (#38078) 2026-02-26 21:04:07 +01:00
check_new_features_spec.rb UX: suppress new feature notifications on brand new sites (#40135) 2026-05-19 09:38:57 +10:00
check_translation_overrides_spec.rb DEV: Refresh translation override status when updating (#31233) 2025-02-07 14:12:28 +08:00
clean_dismissed_topic_users_spec.rb DEV: Update rubocop-discourse to 3.13 and autofix issues (#35073) 2025-10-06 16:11:01 +02:00
clean_up_associated_accounts_spec.rb
clean_up_browser_pageview_events_spec.rb DEV: Track browser session engagement for crawler detection (#41215) 2026-07-01 09:17:48 +08:00
clean_up_crawler_stats_spec.rb
clean_up_email_change_requests_spec.rb
clean_up_email_logs_spec.rb DEV: add shortcut fab!(:variable, :fabricator) to specs (#33577) 2025-07-11 11:16:34 -03:00
clean_up_inactive_users_spec.rb DEV: add clean_up_inactive_users_query modifier for plugins to extend query for inactive users (#41372) 2026-07-06 09:39:11 -07:00
clean_up_post_reply_keys_spec.rb
clean_up_tags_spec.rb DEV: fix a large amount of typos (#37428) 2026-02-02 16:31:58 +11:00
clean_up_unused_staged_users_spec.rb
clean_up_unused_user_api_key_clients_spec.rb
clean_up_unused_user_api_keys_spec.rb DEV: add shortcut fab!(:variable, :fabricator) to specs (#33577) 2025-07-11 11:16:34 -03:00
clean_up_uploads_spec.rb DEV: Prefer RSpec primitives for test setup (#42122) 2026-07-30 11:17:22 +08:00
clean_up_user_api_keys_max_life_spec.rb DEV: Update rubocop-discourse to 3.13 and autofix issues (#35073) 2025-10-06 16:11:01 +02:00
clean_up_user_export_topics_spec.rb
cleanup_problem_check_trackers_spec.rb FIX: Clear disabled/deleted ProblemChecks automatically (#42243) 2026-07-31 21:18:28 +01:00
cleanup_redelivering_web_hook_events_spec.rb
close_topic_spec.rb FEATURE: Allow topic_timers_allowed_groups to manage topic timers (#41126) 2026-06-23 12:46:56 -05:00
correct_missing_dualstack_urls_spec.rb
crawl_topic_link_spec.rb DEV: Clean up scope resolution operators in plugins (#34979) 2025-09-30 14:36:34 +02:00
create_linked_topic_spec.rb DEV: add shortcut fab!(:variable, :fabricator) to specs (#33577) 2025-07-11 11:16:34 -03:00
create_missing_avatars_spec.rb
create_recent_post_search_indexes_spec.rb
create_user_reviewable_spec.rb
delete_replies_spec.rb FIX: DeleteReplies should use the guardian instead of checking for staff (#35443) 2025-10-16 18:26:41 +02:00
delete_topic_spec.rb
delete_user_posts_spec.rb FIX: return nil correctly for nil check (#37460) 2026-02-02 15:43:56 -08:00
deliver_push_notification_spec.rb DEV: Prefer RSpec primitives for test setup (#42122) 2026-07-30 11:17:22 +08:00
detect_crawler_pageviews_spec.rb FEATURE: Reclassify likely crawlers in site traffic reporting (#42199) 2026-08-04 10:08:53 +08:00
download_avatar_from_url_spec.rb
download_backup_email_spec.rb FEATURE: better email subject lines (#36040) 2026-04-24 15:14:10 +04:00
download_profile_background_from_url_spec.rb
emit_web_hook_event_spec.rb DEV: always load the excon gem 2026-01-15 17:13:11 -05:00
enqueue_digest_emails_spec.rb
enqueue_suspect_users_spec.rb FIX: Exclude suspended users from suspect users review queue (#37796) 2026-02-16 10:46:52 +08:00
ensure_db_consistency_spec.rb
ensure_s3_uploads_existence_multisite_spec.rb
ensure_s3_uploads_existence_spec.rb DEV: Deprecate the assignment of nil to site settings (#36093) 2025-12-01 15:04:23 +08:00
export_csv_file_spec.rb PERF: Use daily rollups for DAU/MAU reports (#41958) 2026-07-24 09:52:40 +08:00
export_user_archive_spec.rb DEV: Prefer RSpec primitives for test setup (#42122) 2026-07-30 11:17:22 +08:00
feature_topic_users_spec.rb
fix_out_of_sync_user_uploaded_avatar_spec.rb
fix_primary_emails_for_staged_users_spec.rb
fix_s3_etags_spec.rb
fix_user_usernames_and_groups_names_clash_spec.rb
flush_browser_pageview_events_spec.rb DEV: Track beacon browser pageviews behind dashboard_improvements (#40934) 2026-06-23 10:21:39 +08:00
grant_anniversary_badges_spec.rb
grant_new_user_of_the_month_badges_spec.rb
ignored_users_summary_spec.rb
index_user_fields_for_search_spec.rb
invalidate_inactive_admins_spec.rb
invite_email_spec.rb DEV: Tighten protections when disabling allow_email_invites (#37453) 2026-02-02 11:40:04 -05:00
jobs_base_spec.rb DEV: Enable Rails/FilePath rubocop rule (#40097) 2026-05-19 19:07:54 +02:00
jobs_scheduled_spec.rb
jobs_spec.rb
maintain_browser_pageview_rollups_spec.rb FEATURE: Reclassify likely crawlers in site traffic reporting (#42199) 2026-08-04 10:08:53 +08:00
mass_award_badge_spec.rb
notify_category_change_spec.rb DEV: add shortcut fab!(:variable, :fabricator) to specs (#33577) 2025-07-11 11:16:34 -03:00
notify_mailing_list_subscribers_spec.rb DEV: add shortcut fab!(:variable, :fabricator) to specs (#33577) 2025-07-11 11:16:34 -03:00
notify_moved_posts_spec.rb DEV: add shortcut fab!(:variable, :fabricator) to specs (#33577) 2025-07-11 11:16:34 -03:00
notify_reviewable_spec.rb FIX: Resolve all of a deleted spammer's reviewables (#40780) 2026-06-15 13:34:15 +08:00
notify_tag_change_spec.rb FIX: Allow "watching first post" notification to fire when tags edit notifications are disabled (#37935) 2026-02-20 16:59:53 +08:00
old_keys_reminder_spec.rb DEV: Update rubocop-discourse to 3.13 and autofix issues (#35073) 2025-10-06 16:11:01 +02:00
open_topic_spec.rb FEATURE: Allow topic_timers_allowed_groups to manage topic timers (#41126) 2026-06-23 12:46:56 -05:00
pending_queued_posts_reminder_spec.rb
pending_reviewables_reminder_spec.rb
pending_users_reminder_spec.rb Revert "FEATURE: discourse-workflows (#39704)" (#40366) 2026-05-28 13:59:42 +01:00
periodical_updates_spec.rb
poll_mailbox_spec.rb FIX: Allow only sanitized strings in admin notices (#39796) 2026-05-06 13:40:20 -04:00
post_update_topic_tracking_state_spec.rb FIX: PM unread count shows (1) but message list is empty (#38733) 2026-03-20 09:50:38 +01:00
post_uploads_recovery_spec.rb
prepare_nested_reply_stats_spec.rb PERF: Add fast nested reply stats preparation (#41870) 2026-07-21 10:45:53 -05:00
process_bulk_invite_emails_spec.rb
process_email_spec.rb
process_localized_cooked_spec.rb FEATURE: Localize local oneboxes (#40493) 2026-06-08 11:49:55 +08:00
process_post_spec.rb FIX: Sync category description when post content changes outside PostRevisor (#39184) 2026-05-11 16:13:23 +02:00
process_shelved_notifications_spec.rb DEV: Update rubocop-discourse to 3.13 and autofix issues (#35073) 2025-10-06 16:11:01 +02:00
publish_topic_to_category_spec.rb FIX: TL4 PM participants can schedule system publication of private messages (#39872) 2026-05-12 14:50:04 +10:00
pull_hotlinked_images_spec.rb SECURITY: Authorize secure-upload hotlink downloads against post user 2026-06-30 16:10:52 +02:00
pull_user_profile_hotlinked_images_spec.rb
purge_expired_ignored_users_spec.rb
push_notification_spec.rb DEV: Centralize push notification delivery into a single job (#39207) 2026-04-14 11:55:45 -03:00
rebake_custom_emoji_posts_spec.rb
redeliver_web_hook_events_spec.rb FIX: Match webhook-related messagebus audience to controller audience (#40693) 2026-06-09 18:22:26 +01:00
refresh_users_reviewable_counts_spec.rb
reindex_search_spec.rb FEATURE: Add full-text search for chat messages (#34704) 2025-10-22 11:30:35 +02:00
remap_category_hashtag_spec.rb FIX: Remap category hashtags when slugs change (#41075) 2026-06-23 09:28:30 +10:00
remove_banner_spec.rb
reviewable_priorities_spec.rb DEV: add shortcut fab!(:variable, :fabricator) to specs (#33577) 2025-07-11 11:16:34 -03:00
run_problem_check_spec.rb FIX: Fix scheduled targeted problem checks (#35696) 2025-11-10 10:09:14 +08:00
run_problem_checks_spec.rb FEATURE: Add Site Advice section to the redesigned admin dashboard (#40969) 2026-06-23 11:57:49 +02:00
send_push_notification_spec.rb DEV: Centralize push notification delivery into a single job (#39207) 2026-04-14 11:55:45 -03:00
send_system_message_spec.rb
suspicious_login_spec.rb DEV: add shortcut fab!(:variable, :fabricator) to specs (#33577) 2025-07-11 11:16:34 -03:00
sync_access_control_for_uploads_spec.rb DEV: Prefer RSpec primitives for test setup (#42122) 2026-07-30 11:17:22 +08:00
sync_topic_user_bookmarked_spec.rb
tl3_promotions_spec.rb
toggle_topic_closed_spec.rb DEV: rename topic_id to timerable_id for BaseTimer (#34667) 2025-09-17 13:19:17 +08:00
topic_timer_enqueuer_spec.rb DEV: Update rubocop-discourse to 3.13 and autofix issues (#35073) 2025-10-06 16:11:01 +02:00
truncate_user_flag_stats_spec.rb
unsilence_users_spec.rb
update_animated_uploads_spec.rb
update_gravatar_spec.rb
update_topic_hot_scores_spec.rb DEV: Clean up scope resolution operators in plugins (#34979) 2025-09-30 14:36:34 +02:00
update_username_spec.rb FIX: Sync category description when post content changes outside PostRevisor (#39184) 2026-05-11 16:13:23 +02:00
user_email_spec.rb DEV: add shortcut fab!(:variable, :fabricator) to specs (#33577) 2025-07-11 11:16:34 -03:00