0
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2026-08-06 13:08:40 +08:00
discourse/app/jobs/regular
Mark VanLandingham c85593eda8
PERF: Add fast nested reply stats preparation (#41870)
## Problem

Nested reply statistics currently start backfilling only after nested
replies are enabled. At the default 100 topics every five minutes, that
is 28,800 topics per day—about 174 days for a site with five million
topics. Large sites therefore have to expose the feature before its
historical stats are ready.

## Solution

Add a backend-only `nested_replies:prepare_stats` rake task. For each
site, it:

- enables the existing realtime stats callbacks while the nested-replies
UI remains disabled;
- captures the current regular-topic high-water mark;
- enqueues low-priority, keyset-paginated batches which immediately
enqueue the next batch instead of waiting five minutes;
- reuses the existing per-topic backfill SQL; and
- isolates an individual failed topic into its own job, where normal
Sidekiq retries apply, so later topics can continue.

The task leaves the hidden maintenance flag enabled, so replies created,
deleted, or reparented after preparation begins keep the same stats
current before nested replies are enabled. On multisite it prepares
every database by default; `RAILS_DB` scopes it to one database.

The existing scheduled backfill keeps its original feature gate, topic
selection, batch schedule, and calculations. This adds no UI,
migrations, stats version, readiness marker, durable cursor setting,
lock, or watchdog.

Run it with:

```sh
RAILS_ENV=production bin/rake nested_replies:prepare_stats
```

Completion is logged with the captured high-water topic ID. Re-running
the task is safe because it uses the existing idempotent upsert
behavior.

Tests cover disabled-mode preparation, bounded continuation and
high-water behavior, isolated retries, rake-task enqueueing,
preservation of the old scheduler gate, and realtime maintenance while
the UI is disabled.
2026-07-21 10:45:53 -05:00
..
admin_confirmation_email.rb
anonymize_user.rb FIX: Block associated account emails on user delete and anonymize (#37638) 2026-02-09 18:26:35 +01:00
automatic_group_membership.rb
backfill_badge.rb PERF: Set cluster_concurrency 1 for Jobs::BackfillBadge (#32121) 2025-04-02 11:23:32 +08:00
backfill_sidebar_site_settings.rb
backup_chunks_merger.rb
bulk_grant_trust_level.rb PERF: Add Users to Group Pt. 1 (#38737) 2026-03-26 09:11:22 -05:00
bulk_invite.rb
bulk_user_title_update.rb
bump_topic.rb FEATURE: Allow topic_timers_allowed_groups to manage topic timers (#41126) 2026-06-23 12:46:56 -05:00
change_display_name.rb FIX: Sync category description when post content changes outside PostRevisor (#39184) 2026-05-11 16:13:23 +02:00
check_video_conversion_status.rb DEV: Update mediaconvert to use a subdirectory for output (#36049) 2025-11-20 14:03:14 -07:00
cleanup_acls_for_deleted.rb FEATURE: Support users in DAccessControl and backend (#41358) 2026-07-15 09:18:06 +10:00
clear_slow_mode.rb
close_topic.rb FEATURE: Allow topic_timers_allowed_groups to manage topic timers (#41126) 2026-06-23 12:46:56 -05:00
confirm_sns_subscription.rb SECURITY: Prevent any signed AWS SNS TopicARN from being accepted via webhooks 2026-06-30 16:10:52 +02:00
convert_video.rb FEATURE: Add support for aws MediaConvert (#33092) 2025-07-23 11:58:33 -06:00
crawl_topic_link.rb DEV: always load the excon gem 2026-01-15 17:13:11 -05:00
create_avatar_thumbnails.rb
create_backup.rb
create_linked_topic.rb
create_user_reviewable.rb
critical_user_email.rb
delete_inaccessible_notifications.rb
delete_replies.rb FIX: DeleteReplies should use the guardian instead of checking for staff (#35443) 2025-10-16 18:26:41 +02:00
delete_topic.rb
delete_user_posts.rb FIX: return nil correctly for nil check (#37460) 2026-02-02 15:43:56 -08:00
deliver_push_notification.rb DEV: Centralize push notification delivery into a single job (#39207) 2026-04-14 11:55:45 -03:00
download_avatar_from_url.rb
download_backup_email.rb
download_profile_background_from_url.rb
emit_web_hook_event.rb DEV: always load the excon gem 2026-01-15 17:13:11 -05:00
export_csv_file.rb FIX: re-validate permissions when running CSV export job (#41464) 2026-07-06 16:50:06 +05:30
export_user_archive.rb DEV: Enable Style/RedundantBegin rubocop rule (#40096) 2026-05-19 18:44:54 +02:00
feature_topic_users.rb
generate_topic_og_image.rb FEATURE: generated OpenGraph images for topics (#39194) 2026-06-26 10:37:10 -04:00
generate_topic_thumbnails.rb
group_pm_alert.rb
group_pm_update_summary.rb
group_smtp_email.rb FIX: use next instead of return in sidekiq_retry_in blocks (#39529) 2026-04-24 16:59:49 +02:00
index_category_for_search.rb
index_user_fields_for_search.rb
invite_email.rb DEV: Tighten protections when disabling allow_email_invites (#37453) 2026-02-02 11:40:04 -05:00
invite_password_instructions_email.rb
make_embedded_topic_visible.rb
mass_award_badge.rb
merge_user.rb DEV: Fix assigned but unused variable Prism warnings (#39436) 2026-04-22 12:42:14 +02:00
notify_category_change.rb
notify_mailing_list_subscribers.rb FIX: use next instead of return in sidekiq_retry_in blocks (#39529) 2026-04-24 16:59:49 +02:00
notify_moved_posts.rb DEV: Hand-pick Rails/WhereNot autofixes (#35117) 2025-10-03 13:29:22 +02:00
notify_post_revision.rb
notify_reviewable.rb FIX: Resolve all of a deleted spammer's reviewables (#40780) 2026-06-15 13:34:15 +08:00
notify_tag_change.rb FIX: Allow "watching first post" notification to fire when tags edit notifications are disabled (#37935) 2026-02-20 16:59:53 +08:00
notify_users_added_to_group.rb PERF: Add Users to Group Improvements Pt. 2 (#38901) 2026-04-06 09:31:50 -05:00
open_topic.rb FEATURE: Allow topic_timers_allowed_groups to manage topic timers (#41126) 2026-06-23 12:46:56 -05:00
post_alert.rb
post_update_topic_tracking_state.rb FEATURE: Exclude small actions from topic counters and unread tracking (#40481) 2026-06-08 08:03:55 +02:00
prepare_nested_reply_stats.rb PERF: Add fast nested reply stats preparation (#41870) 2026-07-21 10:45:53 -05:00
process_bulk_invite_emails.rb
process_email.rb
process_localized_cooked.rb FEATURE: Localize local oneboxes (#40493) 2026-06-08 11:49:55 +08:00
process_post.rb FIX: Sync category description when post content changes outside PostRevisor (#39184) 2026-05-11 16:13:23 +02:00
process_sns_notification.rb SECURITY: Prevent any signed AWS SNS TopicARN from being accepted via webhooks 2026-06-30 16:10:52 +02:00
publish_group_membership_updates.rb
publish_topic_to_category.rb FEATURE: Allow topic_timers_allowed_groups to manage topic timers (#41126) 2026-06-23 12:46:56 -05:00
pull_hotlinked_images.rb SECURITY: Authorize secure-upload hotlink downloads against post user 2026-06-30 16:10:52 +02:00
pull_user_profile_hotlinked_images.rb
push_notification.rb DEV: Centralize push notification delivery into a single job (#39207) 2026-04-14 11:55:45 -03:00
rebake_custom_emoji_posts.rb
rebake_posts_for_upload.rb
rebake_quoted_posts_for_user.rb FIX: Move quoted post rebaking on avatar upload to a background job (#40231) 2026-05-22 13:27:52 -03:00
refresh_users_reviewable_counts.rb
remap_category_hashtag.rb FIX: Remap category hashtags when slugs change (#41075) 2026-06-23 09:28:30 +10:00
remove_banner.rb DEV: Update rubocop (#38721) 2026-03-20 00:39:52 +01:00
retrieve_topic.rb
run_heartbeat.rb
run_problem_check.rb FIX: Fix scheduled targeted problem checks (#35696) 2025-11-10 10:09:14 +08:00
send_email_login_code.rb DEV: Introduce enable_local_logins_via_code as an alpha upcoming change (#41624) 2026-07-10 10:19:27 -07:00
send_push_notification.rb DEV: Centralize push notification delivery into a single job (#39207) 2026-04-14 11:55:45 -03:00
send_system_message.rb
site_setting_update_default_categories.rb FIX: Timeout issue when updating a large collection of users when changing the default_categories_* and default_tags_* SiteSettings (#33665) 2025-08-20 12:55:53 -05:00
site_setting_update_default_tags.rb FIX: Timeout issue when updating a large collection of users when changing the default_categories_* and default_tags_* SiteSettings (#33665) 2025-08-20 12:55:53 -05:00
suspicious_login.rb
sync_access_control_for_uploads.rb DEV: Enable Style/RedundantBegin rubocop rule (#40096) 2026-05-19 18:44:54 +02:00
sync_topic_user_bookmarked.rb
toggle_topic_closed.rb FEATURE: Allow topic_timers_allowed_groups to manage topic timers (#41126) 2026-06-23 12:46:56 -05:00
topic_action_converter.rb
topic_timer_base.rb
truncate_user_flag_stats.rb
unpin_topic.rb DEV: Update rubocop (#38721) 2026-03-20 00:39:52 +01:00
update_gravatar.rb
update_group_mentions.rb
update_hotlinked_raw.rb DEV: Update rubocop-discourse to 3.13 and autofix issues (#35073) 2025-10-06 16:11:01 +02:00
update_post_uploads_secure_status.rb
update_top_redirection.rb
update_topic_upload_security.rb
update_username.rb FIX: Sync category description when post content changes outside PostRevisor (#39184) 2026-05-11 16:13:23 +02:00
user_email.rb DEV: Enable Style/RedundantParentheses rubocop rule (#40095) 2026-05-19 15:48:09 +02:00