mirror of
https://gh.wpcy.net/https://github.com/discourse/discourse.git
synced 2026-06-19 09:24:23 +08:00
GitHub oneboxes and the discourse-github plugin talked to GitHub's REST
and
GraphQL API with no rate-limit awareness. On busy instances this
exhausted
GitHub's limits (60 requests/hour unauthenticated, 5000 authenticated),
and
because there was no backoff every render kept hitting GitHub and
re-failing
-- which GitHub's docs warn can get an integration banned. The recently
added PR-status onebox multiplied the number of calls and made it far
worse.
GitHub access was also fragmented: the core onebox engines used OpenURI,
the
discourse-github plugin used Octokit, and the discourse-ai bot tools
used
FinalDestination::HTTP -- three HTTP stacks, three tokens, and
inconsistent
(or entirely missing) error and rate-limit handling.
This introduces a single client, Discourse::GithubApi, that every GitHub
data-API request now flows through. It is built on Faraday with the
SSRF-safe
FinalDestination adapter and:
- authenticates per token (Bearer) and returns plain string-keyed Hashes
(get/post) or raw bodies (raw_get) -- one response shape, no
Octokit/Sawyer
- only ever sends the access token to api.github.com and
raw.githubusercontent.com, rejecting any other absolute URL, so a
user-derived path can never leak a token to an arbitrary host
- backs off on rate limits both reactively (403/429) and proactively
(when
X-RateLimit-Remaining hits 0), honouring Retry-After /
X-RateLimit-Reset,
via a shared Redis flag (GithubRateLimit) keyed per token so each
token's
budget and the shared unauthenticated/IP budget back off independently
- short-circuits while backing off without ever sleeping, so onebox
rendering
and post baking degrade to a plain link instead of blocking a request
- caches ETags and sends If-None-Match, so unchanged resources return
304s
that do not count against the rate limit
Every caller was moved onto it:
- the 6 core GitHub onebox engines, via a slimmed
Onebox::Mixins::GithubApi
adapter that keeps their public methods and translates client errors
back
to the OpenURI::HTTPError vocabulary they already rescue (engines
unchanged)
- the github_blob raw.githubusercontent.com fetch
- the discourse-github plugin (badges, linkback, permalinks, token
validator),
which no longer uses the octokit and sawyer gems (they stay in the
Gemfile for
the discourse-code-review official plugin, which still depends on them)
- the discourse-ai bot's GitHub tools (search code, diff, file content,
search files)
Also adds a GithubOneboxBackoff admin problem check that surfaces while
one of
the onebox token identities is backing off -- scoped to the tokens
resolved by
Onebox::GithubAccess (each configured github_onebox_access_tokens entry
plus the
unauthenticated client) so a backoff on the AI bot or linkback token is
not
misattributed to onebox. Its message points admins at the relevant
setting with
the {{setting:...}} link marker, which problem-check messages now expand
too.
Onebox token resolution is centralised in Onebox::GithubAccess, and the
onebox
cache TTL for transient GitHub failures is shortened so they recover
quickly.
GitHub OAuth login, theme git-clone, the inbound webhook, and the
Oneboxer
FinalDestination URL-resolution special-cases for github.com are
intentionally
out of scope -- they are different concerns, not the rate-limited data
API.
306 lines
9.4 KiB
Ruby
Vendored
306 lines
9.4 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
RSpec.describe DiscourseAi::Agents::Tools::GithubDiff do
|
|
let(:bot_user) { Fabricate(:user) }
|
|
fab!(:llm_model)
|
|
let(:llm) { DiscourseAi::Completions::Llm.proxy(llm_model) }
|
|
|
|
before { enable_current_plugin }
|
|
|
|
context "with #sort_and_shorten_diff" do
|
|
it "sorts and shortens the diff without dropping data" do
|
|
diff = <<~DIFF
|
|
diff --git a/src/lib.rs b/src/lib.rs
|
|
index b466edd..66b068f 100644
|
|
--- a/src/lib.rs
|
|
+++ b/src/lib.rs
|
|
this is a longer diff
|
|
this is a longer diff
|
|
this is a longer diff
|
|
diff --git a/tests/test_encoding.py b/tests/test_encoding.py
|
|
index 27b2192..9f31319 100644
|
|
--- a/tests/test_encoding.py
|
|
+++ b/tests/test_encoding.py
|
|
short diff
|
|
DIFF
|
|
|
|
sorted_diff = described_class.sort_and_shorten_diff(diff)
|
|
|
|
expect(sorted_diff).to eq(<<~DIFF)
|
|
diff --git a/tests/test_encoding.py b/tests/test_encoding.py
|
|
index 27b2192..9f31319 100644
|
|
--- a/tests/test_encoding.py
|
|
+++ b/tests/test_encoding.py
|
|
short diff
|
|
|
|
diff --git a/src/lib.rs b/src/lib.rs
|
|
index b466edd..66b068f 100644
|
|
--- a/src/lib.rs
|
|
+++ b/src/lib.rs
|
|
this is a longer diff
|
|
this is a longer diff
|
|
this is a longer diff
|
|
DIFF
|
|
end
|
|
end
|
|
|
|
context "with parameter validation" do
|
|
let(:repo) { "owner/repo" }
|
|
|
|
it "prioritizes sha when both pull_id and sha provided" do
|
|
tool =
|
|
described_class.new(
|
|
{ repo: repo, pull_id: 123, sha: "abc123" },
|
|
bot_user: bot_user,
|
|
llm: llm,
|
|
)
|
|
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/commits/abc123").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github+json",
|
|
},
|
|
).to_return(status: 404)
|
|
|
|
result = tool.invoke
|
|
expect(result[:error]).to include("commit")
|
|
end
|
|
|
|
it "returns error when neither pull_id nor sha provided" do
|
|
tool = described_class.new({ repo: repo }, bot_user: bot_user, llm: llm)
|
|
result = tool.invoke
|
|
expect(result[:error]).to eq("Must provide either pull_id or sha")
|
|
end
|
|
end
|
|
|
|
context "with a pull request" do
|
|
let(:repo) { "discourse/discourse-automation" }
|
|
let(:pull_id) { 253 }
|
|
let(:tool) do
|
|
described_class.new({ repo: repo, pull_id: pull_id }, bot_user: bot_user, llm: llm)
|
|
end
|
|
let(:diff) { <<~DIFF }
|
|
diff --git a/lib/discourse_automation/automation.rb b/lib/discourse_automation/automation.rb
|
|
index 3e3e3e3..4f4f4f4 100644
|
|
--- a/lib/discourse_automation/automation.rb
|
|
+++ b/lib/discourse_automation/automation.rb
|
|
@@ -1,3 +1,3 @@
|
|
-module DiscourseAutomation
|
|
DIFF
|
|
|
|
let(:pr_info) do
|
|
{
|
|
"title" => "Test PR",
|
|
"state" => "open",
|
|
"user" => {
|
|
"login" => "test-user",
|
|
},
|
|
"created_at" => "2023-01-01T00:00:00Z",
|
|
"updated_at" => "2023-01-02T00:00:00Z",
|
|
"head" => {
|
|
"repo" => {
|
|
"full_name" => "test/repo",
|
|
},
|
|
"ref" => "feature-branch",
|
|
"sha" => "abc123",
|
|
},
|
|
"base" => {
|
|
"repo" => {
|
|
"full_name" => "main/repo",
|
|
},
|
|
"ref" => "main",
|
|
},
|
|
}
|
|
end
|
|
|
|
it "retrieves both PR info and diff" do
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/pulls/#{pull_id}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github+json",
|
|
},
|
|
).to_return(status: 200, body: pr_info.to_json)
|
|
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/pulls/#{pull_id}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github.v3.diff",
|
|
},
|
|
).to_return(status: 200, body: diff)
|
|
|
|
result = tool.invoke
|
|
expect(result[:type]).to eq("pull_request")
|
|
expect(result[:diff]).to eq(diff)
|
|
expect(result[:pr_info]).to include(title: "Test PR", state: "open", author: "test-user")
|
|
expect(result[:error]).to be_nil
|
|
end
|
|
|
|
it "uses the github access token if present" do
|
|
SiteSetting.ai_bot_github_access_token = "ABC"
|
|
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/pulls/#{pull_id}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github+json",
|
|
"Authorization" => "Bearer ABC",
|
|
},
|
|
).to_return(status: 200, body: pr_info.to_json)
|
|
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/pulls/#{pull_id}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github.v3.diff",
|
|
"Authorization" => "Bearer ABC",
|
|
},
|
|
).to_return(status: 200, body: diff)
|
|
|
|
result = tool.invoke
|
|
expect(result[:diff]).to eq(diff)
|
|
expect(result[:error]).to be_nil
|
|
end
|
|
|
|
it "returns an error for invalid pull request" do
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/pulls/#{pull_id}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github+json",
|
|
},
|
|
).to_return(status: 404)
|
|
|
|
result = tool.invoke
|
|
expect(result[:diff]).to be_nil
|
|
expect(result[:error]).to include("Failed to retrieve the PR information")
|
|
end
|
|
|
|
it "handles PRs from deleted forks gracefully" do
|
|
pr_info_deleted_fork = {
|
|
"title" => "Test PR from deleted fork",
|
|
"state" => "closed",
|
|
"user" => nil,
|
|
"created_at" => "2023-01-01T00:00:00Z",
|
|
"updated_at" => "2023-01-02T00:00:00Z",
|
|
"head" => {
|
|
"repo" => nil,
|
|
"ref" => "feature-branch",
|
|
"sha" => "abc123",
|
|
},
|
|
"base" => {
|
|
"repo" => nil,
|
|
"ref" => "main",
|
|
},
|
|
}
|
|
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/pulls/#{pull_id}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github+json",
|
|
},
|
|
).to_return(status: 200, body: pr_info_deleted_fork.to_json)
|
|
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/pulls/#{pull_id}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github.v3.diff",
|
|
},
|
|
).to_return(status: 200, body: diff)
|
|
|
|
result = tool.invoke
|
|
expect(result[:type]).to eq("pull_request")
|
|
expect(result[:diff]).to eq(diff)
|
|
expect(result[:pr_info][:title]).to eq("Test PR from deleted fork")
|
|
expect(result[:pr_info][:author]).to be_nil
|
|
expect(result[:pr_info][:target][:repo]).to be_nil
|
|
expect(result[:error]).to be_nil
|
|
end
|
|
end
|
|
|
|
context "with a commit" do
|
|
let(:repo) { "owner/repo" }
|
|
let(:sha) { "abc123def456" }
|
|
let(:tool) { described_class.new({ repo: repo, sha: sha }, bot_user: bot_user, llm: llm) }
|
|
let(:diff) { <<~DIFF }
|
|
diff --git a/lib/feature.rb b/lib/feature.rb
|
|
index 1234567..89abcdef 100644
|
|
--- a/lib/feature.rb
|
|
+++ b/lib/feature.rb
|
|
@@ -10,3 +10,5 @@
|
|
+def new_method
|
|
+end
|
|
DIFF
|
|
|
|
let(:commit_info) do
|
|
{
|
|
"sha" => sha,
|
|
"commit" => {
|
|
"message" => "Fix bug in feature X\n\nThis fixes an issue with...",
|
|
"author" => {
|
|
"name" => "Test Author",
|
|
"date" => "2023-01-01T00:00:00Z",
|
|
},
|
|
},
|
|
"author" => {
|
|
"login" => "test-user",
|
|
},
|
|
"stats" => {
|
|
"additions" => 10,
|
|
"deletions" => 5,
|
|
"total" => 15,
|
|
},
|
|
"files" => [{ "filename" => "lib/feature.rb" }, { "filename" => "spec/feature_spec.rb" }],
|
|
}
|
|
end
|
|
|
|
it "retrieves commit info and diff" do
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/commits/#{sha}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github+json",
|
|
},
|
|
).to_return(status: 200, body: commit_info.to_json)
|
|
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/commits/#{sha}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github.v3.diff",
|
|
},
|
|
).to_return(status: 200, body: diff)
|
|
|
|
result = tool.invoke
|
|
expect(result[:type]).to eq("commit")
|
|
expect(result[:diff]).to eq(diff)
|
|
expect(result[:commit_info]).to include(
|
|
sha: sha,
|
|
message: "Fix bug in feature X\n\nThis fixes an issue with...",
|
|
author: "Test Author",
|
|
author_login: "test-user",
|
|
files_changed: 2,
|
|
)
|
|
expect(result[:commit_info][:stats]).to include(additions: 10, deletions: 5, total: 15)
|
|
expect(result[:error]).to be_nil
|
|
end
|
|
|
|
it "uses the github access token if present" do
|
|
SiteSetting.ai_bot_github_access_token = "ABC"
|
|
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/commits/#{sha}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github+json",
|
|
"Authorization" => "Bearer ABC",
|
|
},
|
|
).to_return(status: 200, body: commit_info.to_json)
|
|
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/commits/#{sha}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github.v3.diff",
|
|
"Authorization" => "Bearer ABC",
|
|
},
|
|
).to_return(status: 200, body: diff)
|
|
|
|
result = tool.invoke
|
|
expect(result[:diff]).to eq(diff)
|
|
expect(result[:error]).to be_nil
|
|
end
|
|
|
|
it "returns an error for invalid commit" do
|
|
stub_request(:get, "https://api.github.com/repos/#{repo}/commits/#{sha}").with(
|
|
headers: {
|
|
"Accept" => "application/vnd.github+json",
|
|
},
|
|
).to_return(status: 404)
|
|
|
|
result = tool.invoke
|
|
expect(result[:diff]).to be_nil
|
|
expect(result[:error]).to include("Failed to retrieve the commit information")
|
|
end
|
|
end
|
|
end
|