mirror of
https://gh.wpcy.net/https://github.com/discourse/discourse.git
synced 2026-06-19 07:43:46 +08:00
36a8a51ef0
GitHub oneboxes and the discourse-github plugin talked to GitHub's REST
and
GraphQL API with no rate-limit awareness. On busy instances this
exhausted
GitHub's limits (60 requests/hour unauthenticated, 5000 authenticated),
and
because there was no backoff every render kept hitting GitHub and
re-failing
-- which GitHub's docs warn can get an integration banned. The recently
added PR-status onebox multiplied the number of calls and made it far
worse.
GitHub access was also fragmented: the core onebox engines used OpenURI,
the
discourse-github plugin used Octokit, and the discourse-ai bot tools
used
FinalDestination::HTTP -- three HTTP stacks, three tokens, and
inconsistent
(or entirely missing) error and rate-limit handling.
This introduces a single client, Discourse::GithubApi, that every GitHub
data-API request now flows through. It is built on Faraday with the
SSRF-safe
FinalDestination adapter and:
- authenticates per token (Bearer) and returns plain string-keyed Hashes
(get/post) or raw bodies (raw_get) -- one response shape, no
Octokit/Sawyer
- only ever sends the access token to api.github.com and
raw.githubusercontent.com, rejecting any other absolute URL, so a
user-derived path can never leak a token to an arbitrary host
- backs off on rate limits both reactively (403/429) and proactively
(when
X-RateLimit-Remaining hits 0), honouring Retry-After /
X-RateLimit-Reset,
via a shared Redis flag (GithubRateLimit) keyed per token so each
token's
budget and the shared unauthenticated/IP budget back off independently
- short-circuits while backing off without ever sleeping, so onebox
rendering
and post baking degrade to a plain link instead of blocking a request
- caches ETags and sends If-None-Match, so unchanged resources return
304s
that do not count against the rate limit
Every caller was moved onto it:
- the 6 core GitHub onebox engines, via a slimmed
Onebox::Mixins::GithubApi
adapter that keeps their public methods and translates client errors
back
to the OpenURI::HTTPError vocabulary they already rescue (engines
unchanged)
- the github_blob raw.githubusercontent.com fetch
- the discourse-github plugin (badges, linkback, permalinks, token
validator),
which no longer uses the octokit and sawyer gems (they stay in the
Gemfile for
the discourse-code-review official plugin, which still depends on them)
- the discourse-ai bot's GitHub tools (search code, diff, file content,
search files)
Also adds a GithubOneboxBackoff admin problem check that surfaces while
one of
the onebox token identities is backing off -- scoped to the tokens
resolved by
Onebox::GithubAccess (each configured github_onebox_access_tokens entry
plus the
unauthenticated client) so a backoff on the AI bot or linkback token is
not
misattributed to onebox. Its message points admins at the relevant
setting with
the {{setting:...}} link marker, which problem-check messages now expand
too.
Onebox token resolution is centralised in Onebox::GithubAccess, and the
onebox
cache TTL for transient GitHub failures is shortened so they recover
quickly.
GitHub OAuth login, theme git-clone, the inbound webhook, and the
Oneboxer
FinalDestination URL-resolution special-cases for github.com are
intentionally
out of scope -- they are different concerns, not the rate-limited data
API.
91 lines
2.6 KiB
Ruby
Vendored
91 lines
2.6 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
module DiscourseAi
|
|
module Agents
|
|
module Tools
|
|
class GithubSearchFiles < Tool
|
|
def self.signature
|
|
{
|
|
name: name,
|
|
description:
|
|
"Searches for files in a GitHub repository containing specific keywords in their paths or names",
|
|
parameters: [
|
|
{
|
|
name: "repo",
|
|
description: "The repository name in the format 'owner/repo'",
|
|
type: "string",
|
|
required: true,
|
|
},
|
|
{
|
|
name: "keywords",
|
|
description: "An array of keywords to match in file paths or names",
|
|
type: "array",
|
|
item_type: "string",
|
|
required: true,
|
|
},
|
|
{
|
|
name: "branch",
|
|
description:
|
|
"The branch or commit SHA to search within (default: repository's default branch)",
|
|
type: "string",
|
|
required: false,
|
|
},
|
|
],
|
|
}
|
|
end
|
|
|
|
def self.name
|
|
"github_search_files"
|
|
end
|
|
|
|
def repo
|
|
parameters[:repo]
|
|
end
|
|
|
|
def keywords
|
|
parameters[:keywords]
|
|
end
|
|
|
|
def branch
|
|
parameters[:branch]
|
|
end
|
|
|
|
def description_args
|
|
{ repo: repo, keywords: keywords.join(", "), branch: @branch_name }
|
|
end
|
|
|
|
MAX_FILE_SEARCH_RESULTS = 25
|
|
|
|
def invoke
|
|
# Fetch the default branch if no branch is specified
|
|
branch_name = branch || fetch_default_branch(repo)
|
|
@branch_name = branch_name
|
|
|
|
api_url = "https://api.github.com/repos/#{repo}/git/trees/#{branch_name}?recursive=1"
|
|
|
|
begin
|
|
tree_data = github_client.get(api_url)
|
|
rescue Discourse::GithubApi::Error => e
|
|
return { error: "Failed to perform file search. #{e.message}" }
|
|
end
|
|
|
|
matching_files =
|
|
tree_data["tree"]
|
|
.select do |item|
|
|
item["type"] == "blob" && keywords.any? { |keyword| item["path"].include?(keyword) }
|
|
end
|
|
.map { |item| { path: item["path"], size: item["size"] } }
|
|
.take(MAX_FILE_SEARCH_RESULTS)
|
|
|
|
result = { matching_files: matching_files, branch: branch_name }
|
|
if matching_files.length == MAX_FILE_SEARCH_RESULTS
|
|
result[
|
|
:note
|
|
] = "Result limit reached (#{MAX_FILE_SEARCH_RESULTS} files). There may be more matching files."
|
|
end
|
|
result
|
|
end
|
|
end
|
|
end
|
|
end
|
|
end
|
