mirror of
https://github.com/discourse/discourse.git
synced 2026-08-11 02:59:07 +08:00
This is part of the uploads rework series. `disco upload` currently writes its results into `uploads.db`, a hand-written schema where each upload is stashed as a JSON blob (`uploads.upload`, `optimized_images.optimized_images`). That's hard to query and doesn't line up with the schema DSL the other databases use. The replacement is `files.db`, and I want it to store the real Discourse upload columns instead of a blob, so an ad-hoc `SELECT` actually tells you something. So I added a `files_db` schema config under the schema DSL and generated its SQL and models from it. A few things are deliberately different from `intermediate_db`: - The conventions are minimal: no `id -> original_id` rename and no `*upload*_id -> text` rule. In `files.db` the ids are the real staging `Upload#id`/`optimized_images.id` and stay integers, and `upload_id` is a real integer foreign key, not a content hash. I only ignore `updated_at` globally and make `created_at` not required. - `uploads` and `optimized_images` mirror the live Discourse tables. On `uploads` I ignore `user_id` (always the system user on the staging site; the import step assigns the real owner), `access_control_post_id` (staging post id, not portable) and `retain_hours` (not meaningful for imported files). - `upload_results` is synthetic: one row per IntermediateDB `upload_sources` row, keyed by the same XXH3 hash. It carries the precomputed per-source markdown, the `upload_id` (nullable, since several results dedup onto one upload via sha1, and it's NULL when a source was skipped or failed), plus `status` and `skip_reason`. Those two are string enums on purpose so `skip_reason = 'download_error'` stays readable in plain SQL. - `downloads` is the synthetic download-cache index. Nothing consumes `files.db` yet. A later PR switches `disco upload` over and removes the old `uploads_db` schema; here I only add the new schema alongside it. The generated SQL, models and enums come straight from `migrations/bin/disco schema generate --db files_db`.
107 lines
2.8 KiB
Ruby
Vendored
107 lines
2.8 KiB
Ruby
Vendored
# frozen_string_literal: true
|
|
|
|
require "active_support"
|
|
require "active_support/core_ext"
|
|
require "zeitwerk"
|
|
|
|
module Migrations
|
|
class NoSettingsFound < StandardError
|
|
end
|
|
|
|
# Each gem registers its `config/locales` directory so that `enable_i18n`
|
|
# loads the union of all translations (and an isolated gem still sees its own).
|
|
def self.locale_load_paths
|
|
@locale_load_paths ||= []
|
|
end
|
|
|
|
def self.register_locale_path(dir)
|
|
locale_load_paths << dir if locale_load_paths.exclude?(dir)
|
|
end
|
|
|
|
# Root of the `migrations-core` gem. Other gems expose their own root.
|
|
def self.root_path
|
|
@root_path ||= File.expand_path("..", __dir__)
|
|
end
|
|
|
|
# Root of the host Discourse application (the repository root). Used to lazily
|
|
# boot Rails and to discover private converters.
|
|
def self.host_app_root
|
|
@host_app_root ||= File.expand_path("../../..", __dir__)
|
|
end
|
|
|
|
def self.load_rails_environment(quiet: false)
|
|
message = "Loading Rails environment..."
|
|
print message if !quiet
|
|
|
|
rails_root = host_app_root
|
|
# rubocop:disable Discourse/NoChdir
|
|
Dir.chdir(rails_root) do
|
|
ENV["DISCOURSE_DEV_ALLOW_HTTPS"] = "1" # suppress warning
|
|
require File.join(rails_root, "config/environment")
|
|
rescue LoadError => e
|
|
$stderr.puts e.message
|
|
raise
|
|
end
|
|
# rubocop:enable Discourse/NoChdir
|
|
|
|
if !quiet
|
|
print "\r"
|
|
print " " * message.length
|
|
print "\r"
|
|
end
|
|
end
|
|
|
|
def self.loader
|
|
@loader ||=
|
|
begin
|
|
loader = Zeitwerk::Loader.new
|
|
loader.log! if ENV["DEBUG"]
|
|
configure_inflections(loader)
|
|
loader.push_dir(File.join(__dir__, "migrations"), namespace: Migrations)
|
|
configure_collapses(loader)
|
|
loader
|
|
end
|
|
end
|
|
|
|
def self.configure_inflections(loader)
|
|
loader.inflector.inflect(
|
|
{
|
|
"cli" => "CLI",
|
|
"id" => "ID",
|
|
"files_db" => "FilesDB",
|
|
"intermediate_db" => "IntermediateDB",
|
|
"mappings_db" => "MappingsDB",
|
|
},
|
|
)
|
|
end
|
|
|
|
# Collapse `common/` into the root `Migrations` namespace, matching the previous
|
|
# flat layout (e.g. `common/enum.rb` => `Migrations::Enum`). Nested directories
|
|
# such as `common/set_store/` keep contributing a namespace segment
|
|
# (`Migrations::SetStore::*`).
|
|
def self.configure_collapses(loader)
|
|
loader.collapse(File.join(__dir__, "migrations", "common"))
|
|
end
|
|
|
|
def self.setup_loader
|
|
loader.setup
|
|
end
|
|
|
|
def self.enable_i18n
|
|
require "i18n"
|
|
|
|
locale_load_paths.each { |dir| I18n.load_path += Dir[File.join(dir, "**", "migrations.*.yml")] }
|
|
I18n.backend.load_translations
|
|
|
|
# always use English for now
|
|
I18n.default_locale = :en
|
|
I18n.locale = :en
|
|
end
|
|
|
|
def self.apply_global_config
|
|
Regexp.timeout = 2
|
|
end
|
|
end
|
|
|
|
Migrations.register_locale_path(File.join(Migrations.root_path, "config", "locales"))
|
|
Migrations.setup_loader
|