mirror of
https://github.com/discourse/discourse.git
synced 2026-08-09 21:45:25 +08:00
GitHub oneboxes and the discourse-github plugin talked to GitHub's REST
and
GraphQL API with no rate-limit awareness. On busy instances this
exhausted
GitHub's limits (60 requests/hour unauthenticated, 5000 authenticated),
and
because there was no backoff every render kept hitting GitHub and
re-failing
-- which GitHub's docs warn can get an integration banned. The recently
added PR-status onebox multiplied the number of calls and made it far
worse.
GitHub access was also fragmented: the core onebox engines used OpenURI,
the
discourse-github plugin used Octokit, and the discourse-ai bot tools
used
FinalDestination::HTTP -- three HTTP stacks, three tokens, and
inconsistent
(or entirely missing) error and rate-limit handling.
This introduces a single client, Discourse::GithubApi, that every GitHub
data-API request now flows through. It is built on Faraday with the
SSRF-safe
FinalDestination adapter and:
- authenticates per token (Bearer) and returns plain string-keyed Hashes
(get/post) or raw bodies (raw_get) -- one response shape, no
Octokit/Sawyer
- only ever sends the access token to api.github.com and
raw.githubusercontent.com, rejecting any other absolute URL, so a
user-derived path can never leak a token to an arbitrary host
- backs off on rate limits both reactively (403/429) and proactively
(when
X-RateLimit-Remaining hits 0), honouring Retry-After /
X-RateLimit-Reset,
via a shared Redis flag (GithubRateLimit) keyed per token so each
token's
budget and the shared unauthenticated/IP budget back off independently
- short-circuits while backing off without ever sleeping, so onebox
rendering
and post baking degrade to a plain link instead of blocking a request
- caches ETags and sends If-None-Match, so unchanged resources return
304s
that do not count against the rate limit
Every caller was moved onto it:
- the 6 core GitHub onebox engines, via a slimmed
Onebox::Mixins::GithubApi
adapter that keeps their public methods and translates client errors
back
to the OpenURI::HTTPError vocabulary they already rescue (engines
unchanged)
- the github_blob raw.githubusercontent.com fetch
- the discourse-github plugin (badges, linkback, permalinks, token
validator),
which no longer uses the octokit and sawyer gems (they stay in the
Gemfile for
the discourse-code-review official plugin, which still depends on them)
- the discourse-ai bot's GitHub tools (search code, diff, file content,
search files)
Also adds a GithubOneboxBackoff admin problem check that surfaces while
one of
the onebox token identities is backing off -- scoped to the tokens
resolved by
Onebox::GithubAccess (each configured github_onebox_access_tokens entry
plus the
unauthenticated client) so a backoff on the AI bot or linkback token is
not
misattributed to onebox. Its message points admins at the relevant
setting with
the {{setting:...}} link marker, which problem-check messages now expand
too.
Onebox token resolution is centralised in Onebox::GithubAccess, and the
onebox
cache TTL for transient GitHub failures is shortened so they recover
quickly.
GitHub OAuth login, theme git-clone, the inbound webhook, and the
Oneboxer
FinalDestination URL-resolution special-cases for github.com are
intentionally
out of scope -- they are different concerns, not the rate-limited data
API.
271 lines
10 KiB
Ruby
Vendored
271 lines
10 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
module Onebox
|
|
module Mixins
|
|
module GitBlobOnebox
|
|
def self.included(klass)
|
|
klass.include(Onebox::Engine)
|
|
klass.include(Onebox::LayoutSupport)
|
|
klass.matches_regexp(klass.git_regexp)
|
|
klass.always_https
|
|
klass.include(InstanceMethods)
|
|
end
|
|
|
|
EXPAND_AFTER = 0b001
|
|
EXPAND_BEFORE = 0b010
|
|
EXPAND_NONE = 0b0
|
|
|
|
DEFAULTS = {
|
|
EXPAND_ONE_LINER: EXPAND_AFTER | EXPAND_BEFORE, #set how to expand a one liner. user EXPAND_NONE to disable expand
|
|
LINES_BEFORE: 10,
|
|
LINES_AFTER: 10,
|
|
SHOW_LINE_NUMBER: true,
|
|
MAX_LINES: 20,
|
|
MAX_CHARS: 5000,
|
|
}
|
|
|
|
module InstanceMethods
|
|
def initialize(url, timeout = nil)
|
|
super url, timeout
|
|
# merge engine options from global Onebox.options interface
|
|
# self.options = Onebox.options["GithubBlobOnebox"] # self.class.name.split("::").last.to_s
|
|
# self.options = Onebox.options[self.class.name.split("::").last.to_s] #We can use this a more generic approach. extract the engine class name automatically
|
|
|
|
self.options = DEFAULTS
|
|
|
|
@selected_lines_array = nil
|
|
@selected_one_liner = 0
|
|
@model_file = nil
|
|
|
|
# Define constant after merging options set in Onebox.options
|
|
# We can define constant automatically.
|
|
options.each_pair do |constant_name, value|
|
|
constant_name_u = constant_name.to_s.upcase
|
|
if constant_name_u == constant_name.to_s
|
|
#define a constant if not already defined
|
|
unless self.class.const_defined? constant_name_u.to_sym
|
|
Onebox::Mixins::GitBlobOnebox.const_set constant_name_u.to_sym,
|
|
options[constant_name_u.to_sym]
|
|
end
|
|
end
|
|
end
|
|
end
|
|
|
|
private
|
|
|
|
def calc_range(m, contents_lines_size)
|
|
truncated = false
|
|
from = /\d+/.match(m[:from]) #get numeric should only match a positive interger
|
|
to = /\d+/.match(m[:to]) #get numeric should only match a positive interger
|
|
range_provided = !(from.nil? && to.nil?) #true if "from" or "to" provided in URL
|
|
from = from.nil? ? 1 : from[0].to_i #if from not provided default to 1st line
|
|
to = to.nil? ? -1 : to[0].to_i #if to not provided default to undefiend to be handled later in the logic
|
|
|
|
if to === -1 && range_provided #case "from" exists but no valid "to". aka ONE_LINER
|
|
one_liner = true
|
|
to = from
|
|
else
|
|
one_liner = false
|
|
end
|
|
|
|
unless range_provided #case no range provided default to 1..MAX_LINES
|
|
from = 1
|
|
to = MAX_LINES
|
|
truncated = true if contents_lines_size > MAX_LINES
|
|
#we can technically return here
|
|
end
|
|
|
|
from, to = [from, to].sort #enforce valid range. [from < to]
|
|
from = 1 if from > contents_lines_size #if "from" out of TOP bound set to 1st line
|
|
to = contents_lines_size if to > contents_lines_size #if "to" is out of TOP bound set to last line.
|
|
|
|
if one_liner
|
|
@selected_one_liner = from
|
|
if EXPAND_ONE_LINER != EXPAND_NONE
|
|
if (EXPAND_ONE_LINER & EXPAND_BEFORE != 0) # check if EXPAND_BEFORE flag is on
|
|
from = [1, from - LINES_BEFORE].max # make sure expand before does not go out of bound
|
|
end
|
|
|
|
if (EXPAND_ONE_LINER & EXPAND_AFTER != 0) # check if EXPAND_FLAG flag is on
|
|
to = [to + LINES_AFTER, contents_lines_size].min # make sure expand after does not go out of bound
|
|
end
|
|
|
|
from = contents_lines_size if from > contents_lines_size #if "from" is out of the content top bound
|
|
# to = contents_lines_size if to > contents_lines_size #if "to" is out of the content top bound
|
|
else
|
|
#no expand show the one liner solely
|
|
end
|
|
end
|
|
|
|
if to - from > MAX_LINES && !one_liner #if exceed the MAX_LINES limit correct unless range was produced by one_liner which it expand setting will allow exceeding the line limit
|
|
truncated = true
|
|
to = from + MAX_LINES - 1
|
|
end
|
|
|
|
{
|
|
from: from, #calculated from
|
|
from_minus_one: from - 1, #used for getting currect ol>li numbering with css used in template
|
|
to: to, #calculated to
|
|
one_liner: one_liner, #boolean if a one-liner
|
|
selected_one_liner: @selected_one_liner, #if a one liner is provided we create a reference for it.
|
|
range_provided: range_provided, #boolean if range provided
|
|
truncated: truncated,
|
|
}
|
|
end
|
|
|
|
#minimize/compact leading indentation while preserving overall indentation
|
|
def removeLeadingIndentation(str)
|
|
min_space = 100
|
|
a_lines = str.lines
|
|
a_lines.each do |l|
|
|
l = l.chomp("\n") # remove new line
|
|
m = l.match(/\A[ ]*/) # find leading spaces 0 or more
|
|
if m.nil? || l.size == m[0].size || l.size == 0
|
|
next # SKIP no match or line is only spaces
|
|
else # no match | only spaces in line | empty line
|
|
m_str_length = m[0].size
|
|
if m_str_length <= 1 # minimum space is 1 or nothing we can break we found our minimum
|
|
min_space = m_str_length
|
|
break #stop iteration
|
|
end
|
|
min_space = m_str_length if m_str_length < min_space
|
|
end
|
|
end
|
|
a_lines.each do |l|
|
|
re = Regexp.new "^[ ]{#{min_space}}" #match the minimum spaces of the line
|
|
l.gsub!(re, "")
|
|
end
|
|
a_lines.join
|
|
end
|
|
|
|
def line_number_helper(lines, start, selected)
|
|
lines = removeLeadingIndentation(lines.join).lines # A little ineffeicent we could modify removeLeadingIndentation to accept array and return array, but for now it is only working with a string
|
|
hash_builder = []
|
|
output_builder = []
|
|
lines.map.with_index do |line, i|
|
|
lnum = (i.to_i + start)
|
|
hash_builder.push(
|
|
line_number: lnum,
|
|
data: line.gsub("\n", ""),
|
|
selected: (selected == lnum) ? true : false,
|
|
)
|
|
output_builder.push "#{lnum}: #{line}"
|
|
end
|
|
{ output: output_builder.join(), array: hash_builder }
|
|
end
|
|
|
|
def match
|
|
@match ||= @url.match(raw_regexp)
|
|
end
|
|
|
|
def sha1
|
|
full = match[:sha1]
|
|
|
|
# We don't actually know if this is an sha1, or a branch/tag name
|
|
# So we only truncate if it's exactly 40 characters long, which is fairly unlikely to be a branch/tag name.
|
|
if full.length == 40
|
|
full[0..8]
|
|
else
|
|
full
|
|
end
|
|
end
|
|
|
|
def title
|
|
Sanitize.fragment(match[:file])
|
|
end
|
|
|
|
def raw
|
|
return @raw if defined?(@raw)
|
|
|
|
m = match
|
|
|
|
if m
|
|
from = /\d+/.match(m[:from]) #get numeric should only match a positive interger
|
|
to = /\d+/.match(m[:to]) #get numeric should only match a positive interger
|
|
|
|
@file = m[:file]
|
|
@lang = Onebox::FileTypeFinder.from_file_name(m[:file])
|
|
|
|
if @lang == "stl" && link.match?(%r{\Ahttps?://(www\.)?github\.com.*/blob/})
|
|
@model_file = @lang.dup
|
|
@raw = "https://render.githubusercontent.com/view/solid?url=" + raw_template(m)
|
|
else
|
|
contents = fetch_raw(raw_template(m), auth_headers(m)).to_s
|
|
utf8_contents = contents.dup.force_encoding(Encoding::UTF_8)
|
|
|
|
if contents.bytes.include?(0) || !utf8_contents.valid_encoding?
|
|
@raw = nil
|
|
@binary = true
|
|
return
|
|
end
|
|
|
|
contents = utf8_contents
|
|
|
|
contents_lines = contents.lines #get contents lines
|
|
contents_lines_size = contents_lines.size #get number of lines
|
|
|
|
cr = calc_range(m, contents_lines_size) #calculate the range of lines for output
|
|
selected_one_liner = cr[:selected_one_liner] #if url is a one-liner calc_range will return it
|
|
from = cr[:from]
|
|
to = cr[:to]
|
|
@truncated = cr[:truncated]
|
|
range_provided = cr[:range_provided]
|
|
@cr_results = cr
|
|
|
|
if range_provided #if a range provided (single line or more)
|
|
if SHOW_LINE_NUMBER
|
|
lines_result =
|
|
line_number_helper(
|
|
contents_lines[(from - 1)..(to - 1)],
|
|
from,
|
|
selected_one_liner,
|
|
) #print code with prefix line numbers in case range provided
|
|
contents = lines_result[:output]
|
|
@selected_lines_array = lines_result[:array]
|
|
else
|
|
contents = contents_lines[(from - 1)..(to - 1)].join()
|
|
end
|
|
else
|
|
contents = contents_lines[(from - 1)..(to - 1)].join()
|
|
end
|
|
|
|
if contents.length > MAX_CHARS #truncate content chars to limits
|
|
contents = contents[0..MAX_CHARS]
|
|
@truncated = true
|
|
end
|
|
|
|
@raw = contents
|
|
end
|
|
end
|
|
end
|
|
|
|
def fetch_raw(url, headers)
|
|
URI.parse(url).open({ read_timeout: timeout }.merge(headers)).read
|
|
end
|
|
|
|
def data
|
|
@data ||= {
|
|
title: title,
|
|
link: link,
|
|
i18n: i18n,
|
|
# IMPORTANT NOTE: All of the other class variables are populated
|
|
# as *side effects* of the `raw` method! They must all appear
|
|
# AFTER the call to `raw`! Don't get bitten by this like I did!
|
|
content: raw,
|
|
binary: @binary,
|
|
lang: "lang-#{@lang}",
|
|
lines: @selected_lines_array,
|
|
has_lines: !@selected_lines_array.nil?,
|
|
selected_one_liner: @selected_one_liner,
|
|
cr_results: @cr_results,
|
|
truncated: @truncated,
|
|
model_file: @model_file,
|
|
width: 480,
|
|
height: 360,
|
|
sha1: sha1,
|
|
}
|
|
end
|
|
end
|
|
end
|
|
end
|
|
end
|