mirror of
https://gh.wpcy.net/https://github.com/discourse/discourse.git
synced 2026-05-25 04:34:04 +08:00
## Overview
This PR introduces comprehensive search functionality for chat messages,
enabling users to search through their chat history both globally across
all accessible channels and within specific channels.
### Search Capabilities
**All-Channel Search**: When no channel is specified, users can search
across all channels they have access to. The search respects channel
permissions through `ChannelFetcher.all_secured_channel_ids`, ensuring
users only see results from channels they can view.
**Per-Channel Search**: Users can scope their search to a specific
channel by providing a `channel_id` parameter, useful for finding
messages within a particular conversation context.
**Search Features**:
- Full-text search using PostgreSQL's tsvector/tsquery
- Advanced filters: `@username` to filter by author, `#channel` to
filter by channel slug
- Sort options: relevance (default) or latest
- Pagination support
- Search data weighted by relevance
## Site Setting: `chat_search_enabled`
This feature is gated behind the `chat_search_enabled` site setting,
which is currently:
- **Default**: `false`
- **Hidden**: `true`
- **Client-accessible**: `true`
### Deployment Strategy
Due to the need for chat messages to be indexed before search becomes
useful, we're implementing a two-phase deployment:
**Phase 1 (Initial Merge)**:
- `chat_search_enabled` remains `false` and hidden
- The `register_search_index` uses default (true) instead of `chat_search_enabled` value
- This allows the reindexing infrastructure to begin indexing existing
chat messages even if we don't show the UI yet
**Wait Period**:
- Wait at least one week after Phase 1 deployment
- `Jobs::ReindexSearch` runs every 2 hours and will progressively index
all chat messages
- This ensures most sites have a significant part of their chat history indexed
**Phase 2 (Follow-up Merge)**:
- Set `chat_search_enabled` default to `true` and unhide it
- Update the `register_search_index` enabled proc uses the default
(true) instead of using the `chat_search_enabled` setting
- Users can now access search with pre-indexed data
**Rationale**: Without this phased approach, users would see the search
UI immediately but receive no results until the reindexing job runs,
creating a confusing experience. By pre-indexing while the UI is hidden,
we ensure search works immediately when enabled.
## New Plugin API: `register_search_index`
This PR introduces a new plugin API that allows plugins to register
custom search indexes that integrate seamlessly with Discourse's search
infrastructure.
### API Signature
```ruby
register_search_index(
model_class:, # The ActiveRecord model to index
search_data_class:, # The model for storing search data
index_version:, # Version number for re-indexing
search_data:, # Proc that returns weighted search data
load_unindexed_record_ids:,# Proc that finds records needing indexing
enabled: # Optional proc to enable/disable (default: -> { true })
)
```
### How It Works
**Integration with SearchIndexer**: When `SearchIndexer.index(obj)` is
called, it checks registered search handlers for the object's type. If a
handler matches, it:
1. Calls the `search_data` proc with the object and an `IndexerHelper`
instance
2. Receives weighted search data (`:a_weight`, `:b_weight`, `:c_weight`,
`:d_weight`)
3. Updates the corresponding search data table with PostgreSQL's
tsvector
**Integration with Jobs::ReindexSearch**: The scheduled job (runs every
2 hours) calls `rebuild_registered_search_handlers`, which:
1. Iterates through all registered search handlers
2. Skips handlers where `enabled` proc returns `false`
3. Calls `load_unindexed_record_ids` to find records needing indexing
4. Indexes up to `limit` records per handler (default: 10,000)
### Chat Implementation Example
```ruby
register_search_index(
model_class: Chat::Message,
search_data_class: Chat::MessageSearchData,
index_version: 1,
search_data: proc { |message, indexer_helper|
{
a_weight: message.message,
d_weight: indexer_helper.scrub_html(message.cooked)[0..600_000]
}
},
load_unindexed_record_ids: proc { |limit:, index_version:|
Chat::Message
.joins("LEFT JOIN chat_message_search_data ON chat_message_id = chat_messages.id")
.where(
"chat_message_search_data.locale IS NULL OR
chat_message_search_data.locale != ? OR
chat_message_search_data.version != ?",
SiteSetting.default_locale,
index_version
)
.order("chat_messages.id ASC")
.limit(limit)
.pluck(:id)
}
)
```
Co-authored-by: Martin Brennan <mjrbrennan@gmail.com>
Co-authored-by: Loïc Guitaut <5648+Flink@users.noreply.github.com>
384 lines
12 KiB
Ruby
Vendored
384 lines
12 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
require "highline/import"
|
|
|
|
module SystemHelpers
|
|
PLATFORM_KEY_MODIFIER = RUBY_PLATFORM =~ /darwin/i ? :meta : :control
|
|
|
|
def pause_test
|
|
msg = "Test paused. Press enter to resume, or `d` + enter to start debugger.\n\n"
|
|
msg += "Browser inspection URLs:\n"
|
|
|
|
response =
|
|
Net::HTTP.get(CHROME_REMOTE_DEBUGGING_ADDRESS, "/json/list", CHROME_REMOTE_DEBUGGING_PORT)
|
|
|
|
socat_pid = nil
|
|
|
|
if exposed_port =
|
|
ENV["PLAYWRIGHT_FORWARD_DEVTOOLS_TO_PORT"].presence ||
|
|
ENV["SELENIUM_FORWARD_DEVTOOLS_TO_PORT"].presence
|
|
socat_pid =
|
|
fork do
|
|
exec "socat tcp-listen:#{exposed_port},reuseaddr,fork tcp:localhost:#{CHROME_REMOTE_DEBUGGING_PORT}"
|
|
end
|
|
end
|
|
|
|
JSON
|
|
.parse(response)
|
|
.each do |result|
|
|
devtools_url = result["devtoolsFrontendUrl"]
|
|
|
|
devtools_url.gsub!(":#{CHROME_REMOTE_DEBUGGING_PORT}", ":#{exposed_port}") if exposed_port
|
|
|
|
if ENV["CODESPACE_NAME"]
|
|
devtools_url =
|
|
devtools_url
|
|
.gsub(
|
|
"localhost:#{exposed_port}",
|
|
"#{ENV["CODESPACE_NAME"]}-#{exposed_port}.#{ENV["GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN"]}",
|
|
)
|
|
.gsub("http://", "https://")
|
|
.gsub("ws=", "wss=")
|
|
end
|
|
|
|
msg += " - (#{result["type"]}) #{devtools_url} (#{URI(result["url"]).path})\n"
|
|
end
|
|
|
|
result = ask("\n\e[33m#{msg}\e[0m")
|
|
debugger if result == "d" # rubocop:disable Lint/Debugger
|
|
puts "\e[33mResuming...\e[0m"
|
|
Process.kill("TERM", socat_pid) if socat_pid
|
|
self
|
|
end
|
|
|
|
def sign_in(user)
|
|
visit File.join(
|
|
GlobalSetting.relative_url_root || "",
|
|
"/session/#{user.encoded_username}/become.json?redirect=false",
|
|
)
|
|
|
|
expect(page).to have_content("Signed in to #{user.encoded_username} successfully")
|
|
end
|
|
|
|
def setup_system_test
|
|
SiteSetting.login_required = false
|
|
SiteSetting.has_login_hint = false
|
|
SiteSetting.force_hostname = Capybara.server_host
|
|
SiteSetting.port = Capybara.server_port
|
|
SiteSetting.external_system_avatars_url = ""
|
|
SiteSetting.enable_user_tips = false
|
|
SiteSetting.splash_screen = false
|
|
SiteSetting.allowed_internal_hosts =
|
|
(
|
|
SiteSetting.allowed_internal_hosts.to_s.split("|") +
|
|
MinioRunner.config.minio_urls.map { |url| URI.parse(url).host }
|
|
).join("|")
|
|
end
|
|
|
|
def try_until_success(timeout: Capybara.default_max_wait_time, frequency: 0.01, reason: nil)
|
|
start ||= Time.zone.now
|
|
backoff ||= frequency
|
|
yield
|
|
rescue RSpec::Expectations::ExpectationNotMetError,
|
|
Capybara::ExpectationNotMet,
|
|
Capybara::ElementNotFound
|
|
raise if Time.zone.now >= start + timeout.seconds
|
|
sleep backoff
|
|
backoff += frequency
|
|
retry
|
|
end
|
|
|
|
def wait_for_attribute(
|
|
element,
|
|
attribute,
|
|
value,
|
|
timeout: Capybara.default_max_wait_time,
|
|
frequency: 0.01
|
|
)
|
|
try_until_success(timeout: timeout, frequency: frequency) do
|
|
expect(element[attribute.to_sym]).to eq(value)
|
|
end
|
|
end
|
|
|
|
# Waits for an element to stop animating up to timeout seconds,
|
|
# then raises a Capybara error if it does not stop.
|
|
#
|
|
# This is based on getBoundingClientRect, where Y is the distance
|
|
# from the top of the element to the top of the viewport, and X
|
|
# is the distance from the leftmost edge of the element to the
|
|
# left of the viewport. The viewpoint origin (0, 0) is at the
|
|
# top left of the page.
|
|
#
|
|
# Once X and Y stop changing based on the current vs previous position,
|
|
# then we know the animation has stopped and the element is stabilised,
|
|
# at which point we can click on it without fear of Capybara mis-clicking.
|
|
#
|
|
# c.f. https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect
|
|
def wait_for_animation(element, timeout: Capybara.default_max_wait_time)
|
|
old_element_x = nil
|
|
old_element_y = nil
|
|
|
|
try_until_success(timeout: timeout) do
|
|
current_element_x = element.rect[:x]
|
|
current_element_y = element.rect[:y]
|
|
|
|
stopped_moving = current_element_x == old_element_x && current_element_y == old_element_y
|
|
|
|
old_element_x = current_element_x
|
|
old_element_y = current_element_y
|
|
|
|
raise Capybara::ExpectationNotMet if !stopped_moving
|
|
end
|
|
end
|
|
|
|
def resize_window(width: nil, height: nil)
|
|
original_size = Capybara.current_session.current_window.size
|
|
Capybara.current_session.current_window.resize_to(
|
|
width || original_size[0],
|
|
height || original_size[1],
|
|
)
|
|
yield
|
|
ensure
|
|
Capybara.current_session.current_window.resize_to(original_size[0], original_size[1])
|
|
end
|
|
|
|
def using_browser_timezone(timezone, &example)
|
|
using_session(timezone) do
|
|
page.driver.with_playwright_page do |pw_page|
|
|
cdp_session = pw_page.context.new_cdp_session(pw_page)
|
|
cdp_session.send_message("Emulation.setTimezoneOverride", params: { timezoneId: timezone })
|
|
freeze_time(&example)
|
|
end
|
|
end
|
|
end
|
|
|
|
def select_text_range(selector, start = 0, offset = 5)
|
|
js = <<-JS
|
|
const node = document.querySelector(arguments[0]).childNodes[0];
|
|
const selection = window.getSelection();
|
|
const range = document.createRange();
|
|
range.selectNodeContents(node);
|
|
range.setStart(node, arguments[1]);
|
|
range.setEnd(node, arguments[1] + arguments[2]);
|
|
selection.removeAllRanges();
|
|
selection.addRange(range);
|
|
JS
|
|
|
|
page.execute_script(js, selector, start, offset)
|
|
end
|
|
|
|
def current_active_element
|
|
{
|
|
classes: page.evaluate_script("document.activeElement.className"),
|
|
id: page.evaluate_script("document.activeElement.id"),
|
|
}
|
|
end
|
|
|
|
def fake_scroll_down_long(selector_to_make_tall = "#main-outlet")
|
|
find(selector_to_make_tall)
|
|
execute_script(<<~JS)
|
|
(function() {
|
|
const el = document.querySelector("#{selector_to_make_tall}");
|
|
if (!el) {
|
|
throw new Error("Element '#{selector_to_make_tall}' not found");
|
|
}
|
|
el.style.minHeight = "10000px";
|
|
|
|
const sentinel = document.createElement("div");
|
|
sentinel.id = "scroll-sentinel";
|
|
sentinel.style.width = "1px";
|
|
sentinel.style.height = "1px";
|
|
document.body.appendChild(sentinel);
|
|
})();
|
|
JS
|
|
find("#scroll-sentinel")
|
|
execute_script('document.getElementById("scroll-sentinel").scrollIntoView()')
|
|
end
|
|
|
|
def setup_or_skip_s3_system_test(enable_secure_uploads: false, enable_direct_s3_uploads: true)
|
|
skip_unless_s3_system_specs_enabled!
|
|
|
|
SiteSetting.enable_s3_uploads = true
|
|
|
|
SiteSetting.s3_upload_bucket = "discoursetest"
|
|
SiteSetting.enable_upload_debug_mode = true
|
|
|
|
SiteSetting.s3_access_key_id = MinioRunner.config.minio_root_user
|
|
SiteSetting.s3_secret_access_key = MinioRunner.config.minio_root_password
|
|
SiteSetting.s3_endpoint = MinioRunner.config.minio_server_url
|
|
|
|
SiteSetting.enable_direct_s3_uploads = enable_direct_s3_uploads
|
|
SiteSetting.secure_uploads = enable_secure_uploads
|
|
|
|
# On CI, the minio binary is preinstalled in the docker image so there is no need for us to check for a new binary
|
|
MinioRunner.start(install: ENV["CI"] ? false : true)
|
|
end
|
|
|
|
def skip_unless_s3_system_specs_enabled!
|
|
if !ENV["CI"] && !ENV["RUN_S3_SYSTEM_SPECS"]
|
|
skip(
|
|
"S3 system specs are disabled in this environment, set CI=1 or RUN_S3_SYSTEM_SPECS=1 to enable them.",
|
|
)
|
|
end
|
|
end
|
|
|
|
def skip_on_ci!(message = "Flaky on CI")
|
|
skip(message) if ENV["CI"]
|
|
end
|
|
|
|
def click_logo
|
|
PageObjects::Components::Logo.new.click
|
|
end
|
|
|
|
def is_mobile?
|
|
!!RSpec.current_example.metadata[:mobile]
|
|
end
|
|
|
|
def with_logs
|
|
playwright_logger = nil
|
|
page.driver.with_playwright_page { |pw_page| playwright_logger = PlaywrightLogger.new(pw_page) }
|
|
|
|
yield(playwright_logger)
|
|
end
|
|
|
|
# This method can be used to run a system test with a user that has a physical security key by adding a virtual
|
|
# authenticator to the browser. It will automatically remove the virtual authenticator after the block is executed.
|
|
#
|
|
# Example:
|
|
# with_security_key(user, options) do
|
|
# <your system test code here>
|
|
# end
|
|
#
|
|
def with_security_key(user)
|
|
# The public and private keys are complicated to generate programmatically, so we generate it by running the
|
|
# `spec/user_preferences/security_keys_spec.rb` test and uncommenting the lines that print the keys.
|
|
public_key_base64 =
|
|
"pQECAyYgASFYIJhY+jDNJM8g0lyKP3ivDxs+mrKXqfKUY3f7Uo4pWTPDIlggj03xktSm0JTSqbDefhu5WAKH7VRQmWXotjtI/8ka/P0="
|
|
private_key_base64 =
|
|
"MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQg2AWg10o6aoM0s55halZvcQLnpM2tVO2D8Ugw7wFCjzyhRANCAASYWPowzSTPINJcij94rw8bPpqyl6nylGN3-1KOKVkzw49N8ZLUptCU0qmw3n4buVgCh-1UUJll6LY7SP_JGvz9"
|
|
credential_id_base64 = Base64.strict_encode64(SecureRandom.random_bytes(32))
|
|
credential_id_bytes = Base64.urlsafe_decode64(credential_id_base64)
|
|
private_key_bytes = Base64.urlsafe_decode64(private_key_base64)
|
|
|
|
with_virtual_authenticator do |cdp_client, authenticator_id|
|
|
cdp_client.send_message(
|
|
"WebAuthn.addCredential",
|
|
params: {
|
|
authenticatorId: authenticator_id,
|
|
credential: {
|
|
credentialId: Base64.strict_encode64(credential_id_bytes),
|
|
isResidentCredential: false,
|
|
rpId: DiscourseWebauthn.rp_id,
|
|
privateKey: Base64.strict_encode64(private_key_bytes),
|
|
signCount: 1,
|
|
},
|
|
},
|
|
)
|
|
|
|
Fabricate(
|
|
:user_security_key,
|
|
user:,
|
|
public_key: public_key_base64,
|
|
credential_id: credential_id_base64,
|
|
name: "First Key",
|
|
)
|
|
|
|
yield
|
|
end
|
|
end
|
|
|
|
def with_virtual_authenticator(options = {})
|
|
page.driver.with_playwright_page do |pw_page|
|
|
cdp_client = pw_page.context.new_cdp_session(pw_page)
|
|
cdp_client.send_message("WebAuthn.enable")
|
|
|
|
authenticator_options = {
|
|
protocol: "ctap2",
|
|
transport: "usb",
|
|
hasResidentKey: false,
|
|
hasUserVerification: false,
|
|
automaticPresenceSimulation: true,
|
|
}.merge(options)
|
|
|
|
response =
|
|
cdp_client.send_message(
|
|
"WebAuthn.addVirtualAuthenticator",
|
|
params: {
|
|
options: authenticator_options,
|
|
},
|
|
)
|
|
|
|
authenticator_id = response["authenticatorId"]
|
|
|
|
begin
|
|
yield(cdp_client, authenticator_id)
|
|
ensure
|
|
cdp_client.send_message(
|
|
"WebAuthn.removeVirtualAuthenticator",
|
|
params: {
|
|
authenticatorId: authenticator_id,
|
|
},
|
|
)
|
|
|
|
cdp_client.send_message("WebAuthn.disable")
|
|
end
|
|
end
|
|
end
|
|
|
|
def add_cookie(options = {})
|
|
page.driver.with_playwright_page do |playwright_page|
|
|
playwright_page.context.add_cookies(
|
|
[{ domain: Discourse.current_hostname, path: "/" }.merge(options)],
|
|
)
|
|
end
|
|
end
|
|
|
|
def get_style(element, key)
|
|
script = "window.getComputedStyle(arguments[0]).getPropertyValue(arguments[1])"
|
|
page.evaluate_script(script, element, key)
|
|
end
|
|
|
|
def expect_no_alert
|
|
opened_dialog = false
|
|
|
|
page.driver.with_playwright_page do |pw_page|
|
|
pw_page.on("dialog", ->(dialog) { opened_dialog = true })
|
|
|
|
yield
|
|
|
|
expect(opened_dialog).to eq(false)
|
|
end
|
|
end
|
|
|
|
def get_rgb_color(element, property = "backgroundColor")
|
|
css_property = property.underscore.dasherize
|
|
|
|
try_until_success do
|
|
style_hash = element.style(css_property)
|
|
color = style_hash[css_property]
|
|
raise Capybara::ExpectationNotMet if color.blank?
|
|
color
|
|
end
|
|
end
|
|
|
|
# should be used only on very rare occasion when you need to wait for something
|
|
# that is not visually changing on the page
|
|
def wait_for_timeout(ms = 100)
|
|
page.driver.with_playwright_page { |pw_page| pw_page.wait_for_timeout(ms) }
|
|
end
|
|
|
|
def wait_until_hidden(element)
|
|
element.with_playwright_element_handle do |playwright_element|
|
|
playwright_element.wait_for_element_state("hidden")
|
|
rescue Playwright::Error => e
|
|
raise e unless e.message.match?(/Element is not attached to the DOM/)
|
|
end
|
|
end
|
|
|
|
def locator(selector, locator = nil)
|
|
if locator
|
|
locator.locator(selector)
|
|
else
|
|
page.driver.with_playwright_page { |pw_page| pw_page.locator(selector) }
|
|
end
|
|
end
|
|
end
|