mirror of
https://gh.wpcy.net/https://github.com/discourse/discourse.git
synced 2026-06-19 03:23:50 +08:00
fa54f62348
Document attachments (doc, docx, xls, xlsx, rtf, csv, md, txt) are now converted to text before being included in LLM prompts, instead of being forwarded as raw base64 payloads. PDFs remain the only format sent as a raw upload, capped at 10MB. New converters under lib/completions: - DocToText shells out to antiword - DocxToText parses OOXML directly with size and depth limits - XlsToText shells out to xls2csv - XlsxToText parses OOXML and shared strings into CSV-style text - RtfToText is a custom RTF tokenizer with destination/group handling Plain text formats (csv, md, txt) are read with a 1MB byte cap and UTF-8 normalization. Extracted text is truncated to 100k characters, with a preamble noting the original filename and size. Dialect trimming now uses token-aware truncation against a per-message budget so large extracted documents collapse cleanly under the prompt limit, rather than the previous step-based slicing of raw content. Other changes: - LlmModel.normalize_attachment_types is shared with UploadEncoder and collapses "markdown" to "md" so the canonical extension is consistent across model config, UI defaults, and encoder output - ai-llm-attachment-types adds csv, xls, xlsx to the default choices - Locale strings clarify that vision controls images and allowed_attachment_types controls documents --------- Co-authored-by: Rafael Silva <xfalcox@gmail.com>
55 lines
1.5 KiB
Ruby
Vendored
55 lines
1.5 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
require "discourse/safe_exec"
|
|
|
|
module DiscourseAi
|
|
module Completions
|
|
class XlsToText
|
|
XLS2CSV_TIMEOUT_SECONDS = 5
|
|
MAX_CONVERSION_OUTPUT_BYTES = 4 * 100_001
|
|
SAFE_EXEC_ENV = { "PATH" => ENV["PATH"].to_s }.freeze
|
|
XLS2CSV_RLIMITS = {
|
|
cpu_seconds: XLS2CSV_TIMEOUT_SECONDS,
|
|
memory_bytes: 256 * 1024 * 1024,
|
|
file_size_bytes: 1 * 1024 * 1024,
|
|
open_files: 64,
|
|
processes: 0,
|
|
}
|
|
|
|
def self.convert(path)
|
|
return if !xls2csv_installed?
|
|
|
|
Discourse::SafeExec.capture(
|
|
"xls2csv",
|
|
path,
|
|
read: sandbox_read_paths(path),
|
|
execute: Discourse::SafeExec.default_execute_paths,
|
|
timeout: XLS2CSV_TIMEOUT_SECONDS,
|
|
env: SAFE_EXEC_ENV,
|
|
unsetenv_others: true,
|
|
rlimits: XLS2CSV_RLIMITS,
|
|
seccomp_deny_network: true,
|
|
max_output_bytes: MAX_CONVERSION_OUTPUT_BYTES,
|
|
truncate_output: true,
|
|
failure_message: "Failed to convert .xls upload to text",
|
|
)
|
|
end
|
|
|
|
def self.xls2csv_installed?
|
|
return @xls2csv_installed if defined?(@xls2csv_installed)
|
|
|
|
@xls2csv_installed =
|
|
begin
|
|
Discourse::Utils.execute_command("which", "xls2csv")
|
|
true
|
|
rescue Discourse::Utils::CommandError
|
|
false
|
|
end
|
|
end
|
|
|
|
def self.sandbox_read_paths(path)
|
|
Discourse::SafeExec.default_read_paths + [File.realpath(path)]
|
|
end
|
|
end
|
|
end
|
|
end
|
