0
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2026-08-06 13:08:40 +08:00
discourse/lib/onebox/engine/github_pull_request_onebox.rb
Régis Hanol 36a8a51ef0
FEATURE: Route all GitHub API requests through one rate-limited client (#40637)
GitHub oneboxes and the discourse-github plugin talked to GitHub's REST
and
GraphQL API with no rate-limit awareness. On busy instances this
exhausted
GitHub's limits (60 requests/hour unauthenticated, 5000 authenticated),
and
because there was no backoff every render kept hitting GitHub and
re-failing
-- which GitHub's docs warn can get an integration banned. The recently
added PR-status onebox multiplied the number of calls and made it far
worse.

GitHub access was also fragmented: the core onebox engines used OpenURI,
the
discourse-github plugin used Octokit, and the discourse-ai bot tools
used
FinalDestination::HTTP -- three HTTP stacks, three tokens, and
inconsistent
(or entirely missing) error and rate-limit handling.

This introduces a single client, Discourse::GithubApi, that every GitHub
data-API request now flows through. It is built on Faraday with the
SSRF-safe
FinalDestination adapter and:

- authenticates per token (Bearer) and returns plain string-keyed Hashes
(get/post) or raw bodies (raw_get) -- one response shape, no
Octokit/Sawyer
- only ever sends the access token to api.github.com and
  raw.githubusercontent.com, rejecting any other absolute URL, so a
  user-derived path can never leak a token to an arbitrary host
- backs off on rate limits both reactively (403/429) and proactively
(when
X-RateLimit-Remaining hits 0), honouring Retry-After /
X-RateLimit-Reset,
via a shared Redis flag (GithubRateLimit) keyed per token so each
token's
  budget and the shared unauthenticated/IP budget back off independently
- short-circuits while backing off without ever sleeping, so onebox
rendering
  and post baking degrade to a plain link instead of blocking a request
- caches ETags and sends If-None-Match, so unchanged resources return
304s
  that do not count against the rate limit

Every caller was moved onto it:

- the 6 core GitHub onebox engines, via a slimmed
Onebox::Mixins::GithubApi
adapter that keeps their public methods and translates client errors
back
to the OpenURI::HTTPError vocabulary they already rescue (engines
unchanged)
- the github_blob raw.githubusercontent.com fetch
- the discourse-github plugin (badges, linkback, permalinks, token
validator),
which no longer uses the octokit and sawyer gems (they stay in the
Gemfile for
the discourse-code-review official plugin, which still depends on them)
- the discourse-ai bot's GitHub tools (search code, diff, file content,
  search files)

Also adds a GithubOneboxBackoff admin problem check that surfaces while
one of
the onebox token identities is backing off -- scoped to the tokens
resolved by
Onebox::GithubAccess (each configured github_onebox_access_tokens entry
plus the
unauthenticated client) so a backoff on the AI bot or linkback token is
not
misattributed to onebox. Its message points admins at the relevant
setting with
the {{setting:...}} link marker, which problem-check messages now expand
too.
Onebox token resolution is centralised in Onebox::GithubAccess, and the
onebox
cache TTL for transient GitHub failures is shortened so they recover
quickly.

GitHub OAuth login, theme git-clone, the inbound webhook, and the
Oneboxer
FinalDestination URL-resolution special-cases for github.com are
intentionally
out of scope -- they are different concerns, not the rate-limited data
API.
2026-06-15 10:59:10 +02:00

198 lines
6.4 KiB
Ruby
Vendored

# frozen_string_literal: true
require_relative "../mixins/github_body"
require_relative "../mixins/github_api"
module Onebox
module Engine
class GithubPullRequestOnebox
include Engine
include LayoutSupport
include JSON
include Onebox::Mixins::GithubBody
include Onebox::Mixins::GithubApi
matches_domain("github.com", "www.github.com")
always_https
def self.matches_path(path)
path.match?(%r{.*/pull})
end
def url
"https://api.github.com/repos/#{match[:org]}/#{match[:repository]}/pulls/#{match[:number]}"
end
def inline_data
return if !github_token? && !SiteSetting.github_pr_status_enabled
if commit_sha = @url[%r{/commits?/(\h+)}, 1]
commit =
load_json(
"https://api.github.com/repos/#{match[:org]}/#{match[:repository]}/commits/#{commit_sha}",
)
message = commit["commit"]["message"].split("\n").first
return(
{
title:
"#{message} - #{match[:org]}/#{match[:repository]}@#{commit["sha"][0...7]} - GitHub",
}
)
end
pr_data = raw
result = {
title:
"#{pr_data["title"]} - Pull Request ##{match[:number]} - #{match[:org]}/#{match[:repository]} - GitHub",
}
if SiteSetting.github_pr_status_enabled
status = fetch_pr_status(pr_data)&.dig(:status)
result[:css_class] = "--gh-status-#{status}" if status
end
result
rescue StandardError => e
Rails.logger.warn("Inline GitHub PR onebox error for #{@url}: #{e.message}")
nil
end
private
def match
@match ||=
@url.match(%r{github\.com/(?<org>[^/]+)/(?<repository>[^/]+)/pull/(?<number>[^/]+)})
end
def data
result = raw.clone
result["link"] = link
status_data = fetch_pr_status(result)
result["pr_status"] = status_data&.dig(:status)
result["pr_status_title"] = pr_status_title(result["pr_status"])
status_timestamp = status_data&.dig(:timestamp) || result["created_at"]
status_date = Time.parse(status_timestamp)
result["status_date"] = status_date.strftime("%I:%M%p - %d %b %y %Z")
result["status_date_date"] = status_date.strftime("%F")
result["status_date_time"] = status_date.strftime("%T")
ulink = URI(link)
_, org, repo = ulink.path.split("/")
result["domain"] = "#{ulink.host}/#{org}/#{repo}"
result["body"], result["excerpt"] = compute_body(result["body"])
if result["commit"] = load_commit(link)
result["body"], result["excerpt"] =
compute_body(result["commit"]["commit"]["message"].lines[1..].join)
elsif result["comment"] = load_comment(link)
result["body"], result["excerpt"] = compute_body(result["comment"]["body"])
elsif result["discussion"] = load_review(link)
result["body"], result["excerpt"] = compute_body(result["discussion"]["body"])
else
result["pr"] = true
end
result["number"] = match[:number]
result["i18n"] = i18n
result["i18n"]["status_date_label"] = status_date_label(result["pr_status"])
result["i18n"]["pr_summary"] = I18n.t(
"onebox.github.pr_summary",
{
commits: result["commits"],
changed_files: result["changed_files"],
additions: result["additions"],
deletions: result["deletions"],
},
)
result["is_private"] = result.dig("base", "repo", "private")
result["base"]["label"].sub!(/\A#{org}:/, "")
result["head"]["label"].sub!(/\A#{org}:/, "")
result
end
def i18n
{
opened: I18n.t("onebox.github.opened"),
commit_by: I18n.t("onebox.github.commit_by"),
comment_by: I18n.t("onebox.github.comment_by"),
review_by: I18n.t("onebox.github.review_by"),
}
end
def status_date_label(status)
key = status.presence || "open"
I18n.t("onebox.github.status_date.#{key}")
end
def pr_status_title(status)
key = status.presence || "default"
I18n.t("onebox.github.pr_title.#{key}")
end
def load_commit(link)
if commit_match = link.match(%r{commits/(\h+)})
load_json(
"https://api.github.com/repos/#{match[:org]}/#{match[:repository]}/commits/#{commit_match[1]}",
)
end
end
def load_comment(link)
if comment_match = link.match(/#issuecomment-(\d+)/)
load_json(
"https://api.github.com/repos/#{match[:org]}/#{match[:repository]}/issues/comments/#{comment_match[1]}",
)
end
end
def load_review(link)
if review_match = link.match(/#discussion_r(\d+)/)
load_json(
"https://api.github.com/repos/#{match[:org]}/#{match[:repository]}/pulls/comments/#{review_match[1]}",
)
end
end
def fetch_pr_status(pr_data)
return unless SiteSetting.github_pr_status_enabled
return { status: "merged", timestamp: pr_data["merged_at"] } if pr_data["merged"]
return { status: "closed", timestamp: pr_data["closed_at"] } if pr_data["state"] == "closed"
return { status: "draft", timestamp: pr_data["created_at"] } if pr_data["draft"]
reviews_data = load_json(url + "/reviews")
latest_reviews = latest_review_states_with_timestamps(reviews_data)
%w[CHANGES_REQUESTED APPROVED].each do |state|
reviews = latest_reviews.select { |r| r[:state] == state }
if reviews.present?
return { status: state.downcase, timestamp: reviews.map { |r| r[:timestamp] }.max }
end
end
{ status: "open", timestamp: pr_data["created_at"] }
rescue StandardError => e
Rails.logger.warn("GitHub PR status fetch error: #{e.message}")
nil
end
def latest_review_states_with_timestamps(reviews)
return [] if reviews.blank?
reviews
.reject do |r|
r.dig("user", "id").nil? || !%w[CHANGES_REQUESTED APPROVED].include?(r["state"])
end
.group_by { |r| r.dig("user", "id") }
.transform_values { |rs| rs.max_by { |r| r["submitted_at"] } }
.values
.map { |r| { state: r["state"], timestamp: r["submitted_at"] } }
end
end
end
end