mirror of
https://github.com/discourse/discourse.git
synced 2026-08-09 21:45:25 +08:00
GitHub oneboxes and the discourse-github plugin talked to GitHub's REST
and
GraphQL API with no rate-limit awareness. On busy instances this
exhausted
GitHub's limits (60 requests/hour unauthenticated, 5000 authenticated),
and
because there was no backoff every render kept hitting GitHub and
re-failing
-- which GitHub's docs warn can get an integration banned. The recently
added PR-status onebox multiplied the number of calls and made it far
worse.
GitHub access was also fragmented: the core onebox engines used OpenURI,
the
discourse-github plugin used Octokit, and the discourse-ai bot tools
used
FinalDestination::HTTP -- three HTTP stacks, three tokens, and
inconsistent
(or entirely missing) error and rate-limit handling.
This introduces a single client, Discourse::GithubApi, that every GitHub
data-API request now flows through. It is built on Faraday with the
SSRF-safe
FinalDestination adapter and:
- authenticates per token (Bearer) and returns plain string-keyed Hashes
(get/post) or raw bodies (raw_get) -- one response shape, no
Octokit/Sawyer
- only ever sends the access token to api.github.com and
raw.githubusercontent.com, rejecting any other absolute URL, so a
user-derived path can never leak a token to an arbitrary host
- backs off on rate limits both reactively (403/429) and proactively
(when
X-RateLimit-Remaining hits 0), honouring Retry-After /
X-RateLimit-Reset,
via a shared Redis flag (GithubRateLimit) keyed per token so each
token's
budget and the shared unauthenticated/IP budget back off independently
- short-circuits while backing off without ever sleeping, so onebox
rendering
and post baking degrade to a plain link instead of blocking a request
- caches ETags and sends If-None-Match, so unchanged resources return
304s
that do not count against the rate limit
Every caller was moved onto it:
- the 6 core GitHub onebox engines, via a slimmed
Onebox::Mixins::GithubApi
adapter that keeps their public methods and translates client errors
back
to the OpenURI::HTTPError vocabulary they already rescue (engines
unchanged)
- the github_blob raw.githubusercontent.com fetch
- the discourse-github plugin (badges, linkback, permalinks, token
validator),
which no longer uses the octokit and sawyer gems (they stay in the
Gemfile for
the discourse-code-review official plugin, which still depends on them)
- the discourse-ai bot's GitHub tools (search code, diff, file content,
search files)
Also adds a GithubOneboxBackoff admin problem check that surfaces while
one of
the onebox token identities is backing off -- scoped to the tokens
resolved by
Onebox::GithubAccess (each configured github_onebox_access_tokens entry
plus the
unauthenticated client) so a backoff on the AI bot or linkback token is
not
misattributed to onebox. Its message points admins at the relevant
setting with
the {{setting:...}} link marker, which problem-check messages now expand
too.
Onebox token resolution is centralised in Onebox::GithubAccess, and the
onebox
cache TTL for transient GitHub failures is shortened so they recover
quickly.
GitHub OAuth login, theme git-clone, the inbound webhook, and the
Oneboxer
FinalDestination URL-resolution special-cases for github.com are
intentionally
out of scope -- they are different concerns, not the rate-limited data
API.
86 lines
2.5 KiB
Ruby
Vendored
86 lines
2.5 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
module Jobs
|
|
class ReplaceGithubNonPermalinks < ::Jobs::Base
|
|
sidekiq_options queue: "low"
|
|
|
|
def execute(args)
|
|
return unless SiteSetting.enable_discourse_github_plugin?
|
|
return unless SiteSetting.github_permalinks_enabled?
|
|
|
|
post_id = args[:post_id]
|
|
raise Discourse::InvalidParameters.new(:post_id) if post_id.blank?
|
|
|
|
post = Post.find_by(id: post_id)
|
|
return if post.blank?
|
|
|
|
client = Discourse::GithubApi.for(token: SiteSetting.github_linkback_access_token)
|
|
return if client.backing_off?
|
|
|
|
raw = post.raw.dup
|
|
start_raw = raw.dup
|
|
|
|
regex =
|
|
%r{github\.com/(?<user>[^/]+)/(?<repo>[^/\s]+)/blob/(?<sha1>[^/\s]+)/(?<file>[^#\s]+)(?<from-to>#(L([^-\s]*)(-L(\d*))?))?}i
|
|
|
|
matches = post.raw.scan(regex)
|
|
matches.each do |user, repo, sha1, file, from_to|
|
|
next if excluded?(user, repo, file)
|
|
|
|
begin
|
|
commit = client.get("/repos/#{user}/#{repo}/commits/#{sha1}")
|
|
if commit && commit["sha"] != sha1
|
|
new_sha = commit["sha"]
|
|
old_url = "github.com/#{user}/#{repo}/blob/#{sha1}/#{file}#{from_to}"
|
|
new_url = "github.com/#{user}/#{repo}/blob/#{new_sha}/#{file}#{from_to}"
|
|
raw.sub!(old_url, new_url)
|
|
end
|
|
rescue Discourse::GithubApi::NotFound
|
|
next
|
|
rescue => e
|
|
log(
|
|
:error,
|
|
"Failed to replace Github link with permalink in post #{post_id}\n" + e.message + "\n" +
|
|
e.backtrace.join("\n"),
|
|
)
|
|
end
|
|
end
|
|
|
|
post.reload
|
|
|
|
if start_raw == post.raw && raw != post.raw
|
|
changes = { raw: raw, edit_reason: I18n.t("replace_github_link.edit_reason") }
|
|
post.revise(Discourse.system_user, changes, bypass_bump: true)
|
|
end
|
|
end
|
|
|
|
def excluded?(user, repo, file)
|
|
excluded = SiteSetting.github_permalinks_exclude.split("|")
|
|
|
|
excluded.each do |e|
|
|
path_parts = e.split("/")
|
|
# when only filename is provided
|
|
if path_parts.length == 1
|
|
return true if file == e
|
|
next
|
|
end
|
|
|
|
path_parts.each { |p| p.sub!("*", "\\S+") }
|
|
|
|
regex = Regexp.new(path_parts.join("\/"))
|
|
return true if "#{user}/#{repo}/#{file}".match(regex)
|
|
end
|
|
|
|
false
|
|
end
|
|
|
|
private
|
|
|
|
def log(log_level, message)
|
|
Rails.logger.public_send(
|
|
log_level,
|
|
"#{RailsMultisite::ConnectionManagement.current_db}: #{message}",
|
|
)
|
|
end
|
|
end
|
|
end
|