mirror of
https://github.com/discourse/discourse.git
synced 2026-08-10 22:51:02 +08:00
36a8a51ef0GitHub oneboxes and the discourse-github plugin talked to GitHub's REST
and
GraphQL API with no rate-limit awareness. On busy instances this
exhausted
GitHub's limits (60 requests/hour unauthenticated, 5000 authenticated),
and
because there was no backoff every render kept hitting GitHub and
re-failing
-- which GitHub's docs warn can get an integration banned. The recently
added PR-status onebox multiplied the number of calls and made it far
worse.
GitHub access was also fragmented: the core onebox engines used OpenURI,
the
discourse-github plugin used Octokit, and the discourse-ai bot tools
used
FinalDestination::HTTP -- three HTTP stacks, three tokens, and
inconsistent
(or entirely missing) error and rate-limit handling.
This introduces a single client, Discourse::GithubApi, that every GitHub
data-API request now flows through. It is built on Faraday with the
SSRF-safe
FinalDestination adapter and:
- authenticates per token (Bearer) and returns plain string-keyed Hashes
(get/post) or raw bodies (raw_get) -- one response shape, no
Octokit/Sawyer
- only ever sends the access token to api.github.com and
raw.githubusercontent.com, rejecting any other absolute URL, so a
user-derived path can never leak a token to an arbitrary host
- backs off on rate limits both reactively (403/429) and proactively
(when
X-RateLimit-Remaining hits 0), honouring Retry-After /
X-RateLimit-Reset,
via a shared Redis flag (GithubRateLimit) keyed per token so each
token's
budget and the shared unauthenticated/IP budget back off independently
- short-circuits while backing off without ever sleeping, so onebox
rendering
and post baking degrade to a plain link instead of blocking a request
- caches ETags and sends If-None-Match, so unchanged resources return
304s
that do not count against the rate limit
Every caller was moved onto it:
- the 6 core GitHub onebox engines, via a slimmed
Onebox::Mixins::GithubApi
adapter that keeps their public methods and translates client errors
back
to the OpenURI::HTTPError vocabulary they already rescue (engines
unchanged)
- the github_blob raw.githubusercontent.com fetch
- the discourse-github plugin (badges, linkback, permalinks, token
validator),
which no longer uses the octokit and sawyer gems (they stay in the
Gemfile for
the discourse-code-review official plugin, which still depends on them)
- the discourse-ai bot's GitHub tools (search code, diff, file content,
search files)
Also adds a GithubOneboxBackoff admin problem check that surfaces while
one of
the onebox token identities is backing off -- scoped to the tokens
resolved by
Onebox::GithubAccess (each configured github_onebox_access_tokens entry
plus the
unauthenticated client) so a backoff on the AI bot or linkback token is
not
misattributed to onebox. Its message points admins at the relevant
setting with
the {{setting:...}} link marker, which problem-check messages now expand
too.
Onebox token resolution is centralised in Onebox::GithubAccess, and the
onebox
cache TTL for transient GitHub failures is shortened so they recover
quickly.
GitHub OAuth login, theme git-clone, the inbound webhook, and the
Oneboxer
FinalDestination URL-resolution special-cases for github.com are
intentionally
out of scope -- they are different concerns, not the rate-limited data
API.
|
||
|---|---|---|
| .. | ||
| admin/search | ||
| admin_notices | ||
| categories | ||
| category | ||
| concerns | ||
| discourse_id | ||
| flags | ||
| groups | ||
| nested_topic | ||
| notification/action | ||
| notifications | ||
| problem_check | ||
| site_setting | ||
| spam_rule | ||
| tags | ||
| themes | ||
| upcoming_changes | ||
| user | ||
| user_api_key | ||
| video_conversion | ||
| admin_dashboard_engagement.rb | ||
| admin_dashboard_highlights.rb | ||
| admin_dashboard_search.rb | ||
| admin_dashboard_section_configuration.rb | ||
| admin_dashboard_site_traffic.rb | ||
| anonymous_shadow_creator.rb | ||
| badge_granter.rb | ||
| base_bookmarkable.rb | ||
| category_hashtag_data_source.rb | ||
| color_scheme_revisor.rb | ||
| destroy_task.rb | ||
| email_settings_exception_handler.rb | ||
| email_settings_validator.rb | ||
| email_style_updater.rb | ||
| external_upload_manager.rb | ||
| group_action_logger.rb | ||
| group_mentions_updater.rb | ||
| group_message.rb | ||
| handle_chunk_upload.rb | ||
| hashtag_autocomplete_service.rb | ||
| heat_settings_updater.rb | ||
| hub_push_notification_pusher.rb | ||
| inline_uploads.rb | ||
| locale_normalizer.rb | ||
| notification_emailer.rb | ||
| post_action_notifier.rb | ||
| post_alerter.rb | ||
| post_bookmarkable.rb | ||
| post_owner_changer.rb | ||
| push_notification_pusher.rb | ||
| random_topic_selector.rb | ||
| registered_bookmarkable.rb | ||
| search_indexer.rb | ||
| sidebar_section_links_updater.rb | ||
| sidebar_site_settings_backfiller.rb | ||
| site_setting_update_existing_users.rb | ||
| site_settings_task.rb | ||
| staff_action_logger.rb | ||
| tag_hashtag_data_source.rb | ||
| tag_settings_updater.rb | ||
| theme_settings_migrations_runner.rb | ||
| themes_install_task.rb | ||
| topic_bookmarkable.rb | ||
| topic_status_updater.rb | ||
| topic_timestamp_changer.rb | ||
| tracked_topics_updater.rb | ||
| trust_level_granter.rb | ||
| user_action_manager.rb | ||
| user_activator.rb | ||
| user_anonymizer.rb | ||
| user_authenticator.rb | ||
| user_destroyer.rb | ||
| user_merger.rb | ||
| user_notification_renderer.rb | ||
| user_notification_schedule_processor.rb | ||
| user_password_expirer.rb | ||
| user_silencer.rb | ||
| user_stat_count_updater.rb | ||
| user_suspender.rb | ||
| user_updater.rb | ||
| username_changer.rb | ||
| username_checker_service.rb | ||
| web_hook_emitter.rb | ||
| wildcard_domain_checker.rb | ||
| wildcard_url_checker.rb | ||
| word_watcher.rb | ||
