mirror of
https://github.com/discourse/discourse.git
synced 2026-08-04 10:39:43 +08:00
GitHub oneboxes and the discourse-github plugin talked to GitHub's REST
and
GraphQL API with no rate-limit awareness. On busy instances this
exhausted
GitHub's limits (60 requests/hour unauthenticated, 5000 authenticated),
and
because there was no backoff every render kept hitting GitHub and
re-failing
-- which GitHub's docs warn can get an integration banned. The recently
added PR-status onebox multiplied the number of calls and made it far
worse.
GitHub access was also fragmented: the core onebox engines used OpenURI,
the
discourse-github plugin used Octokit, and the discourse-ai bot tools
used
FinalDestination::HTTP -- three HTTP stacks, three tokens, and
inconsistent
(or entirely missing) error and rate-limit handling.
This introduces a single client, Discourse::GithubApi, that every GitHub
data-API request now flows through. It is built on Faraday with the
SSRF-safe
FinalDestination adapter and:
- authenticates per token (Bearer) and returns plain string-keyed Hashes
(get/post) or raw bodies (raw_get) -- one response shape, no
Octokit/Sawyer
- only ever sends the access token to api.github.com and
raw.githubusercontent.com, rejecting any other absolute URL, so a
user-derived path can never leak a token to an arbitrary host
- backs off on rate limits both reactively (403/429) and proactively
(when
X-RateLimit-Remaining hits 0), honouring Retry-After /
X-RateLimit-Reset,
via a shared Redis flag (GithubRateLimit) keyed per token so each
token's
budget and the shared unauthenticated/IP budget back off independently
- short-circuits while backing off without ever sleeping, so onebox
rendering
and post baking degrade to a plain link instead of blocking a request
- caches ETags and sends If-None-Match, so unchanged resources return
304s
that do not count against the rate limit
Every caller was moved onto it:
- the 6 core GitHub onebox engines, via a slimmed
Onebox::Mixins::GithubApi
adapter that keeps their public methods and translates client errors
back
to the OpenURI::HTTPError vocabulary they already rescue (engines
unchanged)
- the github_blob raw.githubusercontent.com fetch
- the discourse-github plugin (badges, linkback, permalinks, token
validator),
which no longer uses the octokit and sawyer gems (they stay in the
Gemfile for
the discourse-code-review official plugin, which still depends on them)
- the discourse-ai bot's GitHub tools (search code, diff, file content,
search files)
Also adds a GithubOneboxBackoff admin problem check that surfaces while
one of
the onebox token identities is backing off -- scoped to the tokens
resolved by
Onebox::GithubAccess (each configured github_onebox_access_tokens entry
plus the
unauthenticated client) so a backoff on the AI bot or linkback token is
not
misattributed to onebox. Its message points admins at the relevant
setting with
the {{setting:...}} link marker, which problem-check messages now expand
too.
Onebox token resolution is centralised in Onebox::GithubAccess, and the
onebox
cache TTL for transient GitHub failures is shortened so they recover
quickly.
GitHub OAuth login, theme git-clone, the inbound webhook, and the
Oneboxer
FinalDestination URL-resolution special-cases for github.com are
intentionally
out of scope -- they are different concerns, not the rate-limited data
API.
160 lines
4.3 KiB
Ruby
Vendored
160 lines
4.3 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
class InlineOneboxer
|
|
MIN_TITLE_LENGTH = 2
|
|
|
|
def initialize(urls, opts = nil)
|
|
@urls = urls
|
|
@opts = opts || {}
|
|
end
|
|
|
|
def process
|
|
@urls.map { |url| InlineOneboxer.lookup(url, @opts) }.compact
|
|
end
|
|
|
|
def self.invalidate(url)
|
|
Discourse.cache.delete(cache_key(url))
|
|
end
|
|
|
|
def self.cache_lookup(url)
|
|
Discourse.cache.read(cache_key(url))
|
|
end
|
|
|
|
def self.local_handlers
|
|
@local_handlers ||= {}
|
|
end
|
|
|
|
def self.register_local_handler(controller, &handler)
|
|
local_handlers[controller] = handler
|
|
end
|
|
|
|
def self.lookup(url, opts = nil)
|
|
opts ||= {}
|
|
opts = opts.with_indifferent_access
|
|
|
|
unless opts[:skip_cache] || opts[:invalidate]
|
|
cached = cache_lookup(url)
|
|
return cached if cached.present?
|
|
end
|
|
|
|
return unless url
|
|
|
|
if route = Discourse.route_for(url)
|
|
if route[:controller] == "topics"
|
|
if topic = Oneboxer.local_topic(url, route, opts)
|
|
opts[:skip_cache] = true
|
|
post_number = [route[:post_number].to_i, topic.highest_post_number].min
|
|
if post_number > 1
|
|
opts[:post_number] = post_number
|
|
opts[:post_author] = post_author_for_title(topic, post_number)
|
|
end
|
|
return onebox_for(url, topic.title, opts)
|
|
else
|
|
# not permitted to see topic
|
|
return nil
|
|
end
|
|
elsif handler = local_handlers[route[:controller]]
|
|
return handler.call(url, route)
|
|
end
|
|
end
|
|
|
|
always_allow = SiteSetting.enable_inline_onebox_on_all_domains
|
|
allowed_domains = SiteSetting.allowed_inline_onebox_domains&.split("|") unless always_allow
|
|
|
|
return nil unless always_allow || allowed_domains
|
|
|
|
uri =
|
|
begin
|
|
URI(url)
|
|
rescue URI::Error
|
|
nil
|
|
end
|
|
|
|
return nil if uri.blank? || uri.hostname.blank?
|
|
return nil unless always_allow || allowed_domains.include?(uri.hostname)
|
|
return nil if Onebox::DomainChecker.is_blocked?(uri.hostname)
|
|
|
|
if data = Oneboxer.inline_data_for(url)
|
|
return onebox_for(url, data[:title], opts, css_class: data[:css_class])
|
|
end
|
|
|
|
max_redirects = 0 if SiteSetting.block_onebox_on_redirect
|
|
headers = { "Accept-Language" => Oneboxer.accept_language }
|
|
|
|
if SiteSetting.inline_onebox_user_agent.present?
|
|
headers["User-Agent"] = SiteSetting.inline_onebox_user_agent
|
|
end
|
|
|
|
title =
|
|
RetrieveTitle.crawl(
|
|
url,
|
|
max_redirects: max_redirects,
|
|
initial_https_redirect_ignore_limit: SiteSetting.block_onebox_on_redirect,
|
|
headers: headers,
|
|
)
|
|
title = nil if title && title.length < MIN_TITLE_LENGTH
|
|
|
|
onebox_for(url, title, opts)
|
|
end
|
|
|
|
def self.is_previewing?(user_id)
|
|
Discourse.redis.get(preview_key(user_id)) == "1"
|
|
end
|
|
|
|
def self.preview!(user_id)
|
|
Discourse.redis.setex(preview_key(user_id), 1.minute, "1")
|
|
end
|
|
|
|
def self.finish_preview!(user_id)
|
|
Discourse.redis.del(preview_key(user_id))
|
|
end
|
|
|
|
private
|
|
|
|
def self.onebox_for(url, title, opts, css_class: nil)
|
|
title = title && Emoji.gsub_emoji_to_unicode(title)
|
|
if title && opts[:post_number]
|
|
title += " - "
|
|
if opts[:post_author]
|
|
title +=
|
|
I18n.t(
|
|
"inline_oneboxer.topic_page_title_post_number_by_user",
|
|
post_number: opts[:post_number],
|
|
username: opts[:post_author],
|
|
)
|
|
else
|
|
title +=
|
|
I18n.t("inline_oneboxer.topic_page_title_post_number", post_number: opts[:post_number])
|
|
end
|
|
end
|
|
|
|
title = title && Emoji.gsub_emoji_to_unicode(title)
|
|
title = WordWatcher.censor_text(title) if title.present?
|
|
|
|
onebox = { url: url, title: title }
|
|
onebox[:css_class] = css_class if css_class.present?
|
|
|
|
if !opts[:skip_cache]
|
|
ttl = onebox[:title].blank? ? 1.minute : 1.day
|
|
Discourse.cache.write(cache_key(url), onebox, expires_in: ttl)
|
|
end
|
|
onebox
|
|
end
|
|
|
|
def self.cache_key(url)
|
|
"inline_onebox:#{Oneboxer.onebox_locale}:#{url}"
|
|
end
|
|
|
|
def self.post_author_for_title(topic, post_number)
|
|
guardian = Guardian.new
|
|
post = topic.posts.find_by(post_number: post_number)
|
|
author = post&.user
|
|
if author && guardian.can_see_post?(post) && post.post_type == Post.types[:regular]
|
|
author.username
|
|
end
|
|
end
|
|
|
|
def self.preview_key(user_id)
|
|
"inline-onebox:preview:#{user_id}"
|
|
end
|
|
end
|