mirror of
https://github.com/discourse/discourse.git
synced 2026-08-06 13:08:40 +08:00
36a8a51ef0GitHub oneboxes and the discourse-github plugin talked to GitHub's REST
and
GraphQL API with no rate-limit awareness. On busy instances this
exhausted
GitHub's limits (60 requests/hour unauthenticated, 5000 authenticated),
and
because there was no backoff every render kept hitting GitHub and
re-failing
-- which GitHub's docs warn can get an integration banned. The recently
added PR-status onebox multiplied the number of calls and made it far
worse.
GitHub access was also fragmented: the core onebox engines used OpenURI,
the
discourse-github plugin used Octokit, and the discourse-ai bot tools
used
FinalDestination::HTTP -- three HTTP stacks, three tokens, and
inconsistent
(or entirely missing) error and rate-limit handling.
This introduces a single client, Discourse::GithubApi, that every GitHub
data-API request now flows through. It is built on Faraday with the
SSRF-safe
FinalDestination adapter and:
- authenticates per token (Bearer) and returns plain string-keyed Hashes
(get/post) or raw bodies (raw_get) -- one response shape, no
Octokit/Sawyer
- only ever sends the access token to api.github.com and
raw.githubusercontent.com, rejecting any other absolute URL, so a
user-derived path can never leak a token to an arbitrary host
- backs off on rate limits both reactively (403/429) and proactively
(when
X-RateLimit-Remaining hits 0), honouring Retry-After /
X-RateLimit-Reset,
via a shared Redis flag (GithubRateLimit) keyed per token so each
token's
budget and the shared unauthenticated/IP budget back off independently
- short-circuits while backing off without ever sleeping, so onebox
rendering
and post baking degrade to a plain link instead of blocking a request
- caches ETags and sends If-None-Match, so unchanged resources return
304s
that do not count against the rate limit
Every caller was moved onto it:
- the 6 core GitHub onebox engines, via a slimmed
Onebox::Mixins::GithubApi
adapter that keeps their public methods and translates client errors
back
to the OpenURI::HTTPError vocabulary they already rescue (engines
unchanged)
- the github_blob raw.githubusercontent.com fetch
- the discourse-github plugin (badges, linkback, permalinks, token
validator),
which no longer uses the octokit and sawyer gems (they stay in the
Gemfile for
the discourse-code-review official plugin, which still depends on them)
- the discourse-ai bot's GitHub tools (search code, diff, file content,
search files)
Also adds a GithubOneboxBackoff admin problem check that surfaces while
one of
the onebox token identities is backing off -- scoped to the tokens
resolved by
Onebox::GithubAccess (each configured github_onebox_access_tokens entry
plus the
unauthenticated client) so a backoff on the AI bot or linkback token is
not
misattributed to onebox. Its message points admins at the relevant
setting with
the {{setting:...}} link marker, which problem-check messages now expand
too.
Onebox token resolution is centralised in Onebox::GithubAccess, and the
onebox
cache TTL for transient GitHub failures is shortened so they recover
quickly.
GitHub OAuth login, theme git-clone, the inbound webhook, and the
Oneboxer
FinalDestination URL-resolution special-cases for github.com are
intentionally
out of scope -- they are different concerns, not the rate-limited data
API.
91 lines
2.7 KiB
Ruby
Vendored
91 lines
2.7 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
require_relative "../mixins/github_body"
|
|
require_relative "../mixins/github_api"
|
|
|
|
module Onebox
|
|
module Engine
|
|
class GithubIssueOnebox
|
|
#Author Lidlanca 2014
|
|
include Engine
|
|
include LayoutSupport
|
|
include JSON
|
|
include Onebox::Mixins::GithubBody
|
|
include Onebox::Mixins::GithubApi
|
|
|
|
matches_regexp(
|
|
%r{^https?://(?:www\.)?(?:(?:\w)+\.)?github\.com/(?<org>.+)/(?<repo>.+)/issues/([[:digit:]]+)},
|
|
)
|
|
always_https
|
|
|
|
def url
|
|
m = match
|
|
"https://api.github.com/repos/#{m["org"]}/#{m["repo"]}/issues/#{m["item_id"]}"
|
|
end
|
|
|
|
def inline_data
|
|
return unless github_token?
|
|
|
|
result = raw
|
|
{
|
|
title:
|
|
"#{result["title"]} - Issue ##{match["item_id"]} - #{match["org"]}/#{match["repo"]} - GitHub",
|
|
}
|
|
rescue StandardError => e
|
|
Rails.logger.warn("Inline GitHub issue onebox error for #{@url}: #{e.message}")
|
|
nil
|
|
end
|
|
|
|
private
|
|
|
|
def match
|
|
@match ||=
|
|
@url.match(
|
|
%r{^http(?:s)?://(?:www\.)?(?:(?:\w)+\.)?github\.com/(?<org>.+)/(?<repo>.+)/(?<type>issues)/(?<item_id>[\d]+)},
|
|
)
|
|
end
|
|
|
|
def i18n
|
|
{ opened: I18n.t("onebox.github.opened"), closed: I18n.t("onebox.github.closed") }
|
|
end
|
|
|
|
def data
|
|
result = raw.clone
|
|
repo = load_repo
|
|
created_at = Time.parse(result["created_at"])
|
|
closed_at = Time.parse(result["closed_at"]) if result["closed_at"]
|
|
body, excerpt = compute_body(result["body"])
|
|
ulink = URI(link)
|
|
|
|
labels =
|
|
result["labels"].map do |l|
|
|
{ name: Emoji.codes_to_img(Onebox::Helpers.sanitize(l["name"])) }
|
|
end
|
|
|
|
{
|
|
link: @url,
|
|
title: result["title"],
|
|
body: body,
|
|
excerpt: excerpt,
|
|
labels: labels,
|
|
user: result["user"],
|
|
created_at: created_at.strftime("%I:%M%p - %d %b %y %Z"),
|
|
created_at_date: created_at.strftime("%F"),
|
|
created_at_time: created_at.strftime("%T"),
|
|
closed_at: closed_at&.strftime("%I:%M%p - %d %b %y %Z"),
|
|
closed_at_date: closed_at&.strftime("%F"),
|
|
closed_at_time: closed_at&.strftime("%T"),
|
|
closed_by: result["closed_by"],
|
|
avatar: "https://avatars1.githubusercontent.com/u/#{result["user"]["id"]}?v=2&s=96",
|
|
domain: "#{ulink.host}/#{ulink.path.split("/")[1]}/#{ulink.path.split("/")[2]}",
|
|
i18n: i18n,
|
|
is_private: repo["private"],
|
|
}
|
|
end
|
|
|
|
def load_repo
|
|
load_json("https://api.github.com/repos/#{match[:org]}/#{match[:repo]}")
|
|
end
|
|
end
|
|
end
|
|
end
|
