mirror of
https://github.com/discourse/discourse.git
synced 2026-08-14 13:58:53 +08:00
GitHub oneboxes and the discourse-github plugin talked to GitHub's REST
and
GraphQL API with no rate-limit awareness. On busy instances this
exhausted
GitHub's limits (60 requests/hour unauthenticated, 5000 authenticated),
and
because there was no backoff every render kept hitting GitHub and
re-failing
-- which GitHub's docs warn can get an integration banned. The recently
added PR-status onebox multiplied the number of calls and made it far
worse.
GitHub access was also fragmented: the core onebox engines used OpenURI,
the
discourse-github plugin used Octokit, and the discourse-ai bot tools
used
FinalDestination::HTTP -- three HTTP stacks, three tokens, and
inconsistent
(or entirely missing) error and rate-limit handling.
This introduces a single client, Discourse::GithubApi, that every GitHub
data-API request now flows through. It is built on Faraday with the
SSRF-safe
FinalDestination adapter and:
- authenticates per token (Bearer) and returns plain string-keyed Hashes
(get/post) or raw bodies (raw_get) -- one response shape, no
Octokit/Sawyer
- only ever sends the access token to api.github.com and
raw.githubusercontent.com, rejecting any other absolute URL, so a
user-derived path can never leak a token to an arbitrary host
- backs off on rate limits both reactively (403/429) and proactively
(when
X-RateLimit-Remaining hits 0), honouring Retry-After /
X-RateLimit-Reset,
via a shared Redis flag (GithubRateLimit) keyed per token so each
token's
budget and the shared unauthenticated/IP budget back off independently
- short-circuits while backing off without ever sleeping, so onebox
rendering
and post baking degrade to a plain link instead of blocking a request
- caches ETags and sends If-None-Match, so unchanged resources return
304s
that do not count against the rate limit
Every caller was moved onto it:
- the 6 core GitHub onebox engines, via a slimmed
Onebox::Mixins::GithubApi
adapter that keeps their public methods and translates client errors
back
to the OpenURI::HTTPError vocabulary they already rescue (engines
unchanged)
- the github_blob raw.githubusercontent.com fetch
- the discourse-github plugin (badges, linkback, permalinks, token
validator),
which no longer uses the octokit and sawyer gems (they stay in the
Gemfile for
the discourse-code-review official plugin, which still depends on them)
- the discourse-ai bot's GitHub tools (search code, diff, file content,
search files)
Also adds a GithubOneboxBackoff admin problem check that surfaces while
one of
the onebox token identities is backing off -- scoped to the tokens
resolved by
Onebox::GithubAccess (each configured github_onebox_access_tokens entry
plus the
unauthenticated client) so a backoff on the AI bot or linkback token is
not
misattributed to onebox. Its message points admins at the relevant
setting with
the {{setting:...}} link marker, which problem-check messages now expand
too.
Onebox token resolution is centralised in Onebox::GithubAccess, and the
onebox
cache TTL for transient GitHub failures is shortened so they recover
quickly.
GitHub OAuth login, theme git-clone, the inbound webhook, and the
Oneboxer
FinalDestination URL-resolution special-cases for github.com are
intentionally
out of scope -- they are different concerns, not the rate-limited data
API.
212 lines
6 KiB
Ruby
Vendored
212 lines
6 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
require_dependency "pretty_text"
|
|
require "digest/sha1"
|
|
|
|
class GithubLinkback
|
|
class Link
|
|
attr_reader :url, :project, :type
|
|
attr_accessor :sha, :pr_number, :issue_number
|
|
|
|
def initialize(url, project, type)
|
|
@url = url
|
|
@project = project
|
|
@type = type
|
|
end
|
|
end
|
|
|
|
def initialize(post)
|
|
@post = post
|
|
end
|
|
|
|
def should_enqueue?
|
|
return false if ignored_category?
|
|
|
|
!!(
|
|
SiteSetting.github_linkback_enabled? && SiteSetting.enable_discourse_github_plugin? &&
|
|
@post.present? && @post.post_type == Post.types[:regular] && @post.raw =~ /github\.com/ &&
|
|
Guardian.new.can_see?(@post) && @post.topic.visible?
|
|
)
|
|
end
|
|
|
|
def enqueue
|
|
Jobs.enqueue(:create_github_linkback, post_id: @post.id) if should_enqueue?
|
|
end
|
|
|
|
def github_links
|
|
projects = SiteSetting.github_linkback_projects.split("|")
|
|
|
|
return [] if projects.blank?
|
|
|
|
result = {}
|
|
PrettyText
|
|
.extract_links(@post.cooked)
|
|
.map(&:url)
|
|
.each do |l|
|
|
if l =~ %r{https?://github\.com/([^/]+)/([^/]+)/commit/([0-9a-f]+)}
|
|
url, org, repo, sha = Regexp.last_match.to_a
|
|
project = "#{org}/#{repo}"
|
|
|
|
next if result[url]
|
|
next if @post.custom_fields[GithubLinkback.field_for(url)].present?
|
|
next unless is_allowed_project_link?(projects, project)
|
|
|
|
link = Link.new(url, project, :commit)
|
|
link.sha = sha
|
|
result[url] = link
|
|
elsif l =~ %r{https?://github.com/([^/]+)/([^/]+)/pull/(\d+)}
|
|
url, org, repo, pr_number = Regexp.last_match.to_a
|
|
project = "#{org}/#{repo}"
|
|
|
|
next if result[url]
|
|
next if @post.custom_fields[GithubLinkback.field_for(url)].present?
|
|
next unless is_allowed_project_link?(projects, project)
|
|
|
|
link = Link.new(url, project, :pr)
|
|
link.pr_number = pr_number.to_i
|
|
result[url] = link
|
|
elsif l =~ %r{https?://github.com/([^/]+)/([^/]+)/issues/(\d+)}
|
|
url, org, repo, issue_number = Regexp.last_match.to_a
|
|
project = "#{org}/#{repo}"
|
|
|
|
next if result[url]
|
|
next if @post.custom_fields[GithubLinkback.field_for(url)].present?
|
|
next unless is_allowed_project_link?(projects, project)
|
|
|
|
link = Link.new(url, project, :issue)
|
|
link.issue_number = issue_number.to_i
|
|
result[url] = link
|
|
end
|
|
end
|
|
result.values
|
|
end
|
|
|
|
def is_allowed_project_link?(projects, project)
|
|
return true if projects.include?(project)
|
|
|
|
check_user = project.split("/")[0]
|
|
projects.any? do |allowed_project|
|
|
allowed_user, allowed_all_projects = allowed_project.split("/")
|
|
(allowed_user == check_user) && (allowed_all_projects == "*")
|
|
end
|
|
end
|
|
|
|
def create
|
|
return [] if SiteSetting.github_linkback_access_token.blank?
|
|
return [] if client.backing_off?
|
|
|
|
links = []
|
|
|
|
DistributedMutex.synchronize("github_linkback_#{@post.id}") do
|
|
links = github_links
|
|
return [] if links.length() > SiteSetting.github_linkback_maximum_links
|
|
|
|
links.each do |link|
|
|
begin
|
|
case link.type
|
|
when :commit
|
|
post_commit(link)
|
|
when :pr
|
|
post_pr_or_issue(link, :pr)
|
|
when :issue
|
|
post_pr_or_issue(link, :issue)
|
|
else
|
|
next
|
|
end
|
|
rescue Discourse::GithubApi::Error => e
|
|
Rails.logger.warn("Failed to post GitHub linkback for #{link.url}: #{e.message}")
|
|
next
|
|
end
|
|
|
|
# Don't post the same link twice
|
|
@post.custom_fields[GithubLinkback.field_for(link.url)] = "true"
|
|
end
|
|
@post.save_custom_fields
|
|
end
|
|
|
|
links
|
|
end
|
|
|
|
def self.field_for(url)
|
|
"github-linkback:#{Digest::SHA1.hexdigest(url)[0..15]}"
|
|
end
|
|
|
|
private
|
|
|
|
def ignored_category?
|
|
return false if @post&.topic&.category_id.blank?
|
|
|
|
SiteSetting.github_linkback_ignored_categories_map.include?(@post.topic.category_id)
|
|
end
|
|
|
|
def post_pr_or_issue(link, type)
|
|
pr_or_issue_number = link.pr_number || link.issue_number
|
|
|
|
return if topic_already_linked_on_github?(link)
|
|
|
|
comment =
|
|
I18n.t(
|
|
type == :pr ? "github_linkback.pr_template" : "github_linkback.issue_template",
|
|
title: SiteSetting.title,
|
|
post_url: "#{Discourse.base_url}#{@post.url}",
|
|
)
|
|
|
|
client.post("/repos/#{link.project}/issues/#{pr_or_issue_number}/comments", { body: comment })
|
|
end
|
|
|
|
def post_commit(link)
|
|
return if topic_already_linked_on_github?(link)
|
|
|
|
comment =
|
|
I18n.t(
|
|
"github_linkback.commit_template",
|
|
title: SiteSetting.title,
|
|
post_url: "#{Discourse.base_url}#{@post.url}",
|
|
)
|
|
|
|
client.post("/repos/#{link.project}/commits/#{link.sha}/comments", { body: comment })
|
|
end
|
|
|
|
def topic_already_linked_on_github?(link)
|
|
texts =
|
|
if link.type == :commit
|
|
fetch_commit_comment_texts(link.project, link.sha)
|
|
else
|
|
fetch_pr_or_issue_texts(link.project, link.pr_number || link.issue_number)
|
|
end
|
|
|
|
texts.any? { |text| text_links_to_topic?(text) }
|
|
rescue Discourse::GithubApi::Error
|
|
false
|
|
end
|
|
|
|
def fetch_commit_comment_texts(project, sha)
|
|
client
|
|
.get("/repos/#{project}/commits/#{sha}/comments", per_page: 100)
|
|
.map { |comment| comment["body"].to_s }
|
|
end
|
|
|
|
def fetch_pr_or_issue_texts(project, number)
|
|
[
|
|
client.get("/repos/#{project}/issues/#{number}")["body"].to_s,
|
|
*client
|
|
.get("/repos/#{project}/issues/#{number}/comments", per_page: 100)
|
|
.map { |comment| comment["body"].to_s },
|
|
]
|
|
end
|
|
|
|
def text_links_to_topic?(text)
|
|
base_url = Discourse.base_url
|
|
text
|
|
.scan(%r{#{Regexp.escape(base_url)}/t/\S+})
|
|
.any? do |url|
|
|
route = Discourse.route_for(url)
|
|
route && route[:controller] == "topics" && route[:action] == "show" &&
|
|
(route[:id] || route[:topic_id]).to_i == @post.topic_id
|
|
end
|
|
end
|
|
|
|
def client
|
|
@client ||= Discourse::GithubApi.for(token: SiteSetting.github_linkback_access_token)
|
|
end
|
|
end
|