discourse/migrations/importer/config/upload.yml.sample
Gerhard Schlager 06b32204c0
MT: Split the migrations tooling into separate gems (#40492)
Previously, the migrations tooling was a single flat `migrations/` tree,
autoloaded by one global Zeitwerk loader and driven by a Thor CLI, so each
planned next step had nowhere clean to land.

This change splits it into four `path:`-referenced gems — `migrations-core`,
`migrations-tooling`, `migrations-converters`, and `migrations-importer` —
served by a single Samovar-based `disco` binary, without rewriting any domain
logic.

### Why now

The DSL refactor that replaced the IntermediateDB YAML config just landed,
which is the cheapest moment to do this. Everything queued behind it — column
coverage verification, the `discourse-migrations` validation plugin, the
transformer framework, and private converter isolation — either has nowhere
clean to land in the flat tree or would have to be retrofitted into a gem
layout later. Doing the split now, while it's still a pure move (suite green,
no domain logic touched), is far cheaper than after another round of features
has built on the flat layout.

### What changes

- **Four gems under `migrations/`**, all `path:`-referenced from the root
  `Gemfile` (nothing is published to RubyGems): `core` (CLI framework, UI, DB
  infrastructure, IntermediateDB, and the conversion framework), `tooling`
  (schema DSL and `schema` commands), `converters` (implementations and source
  adapters), and `importer` (row and uploads import).
- **A single CLI binary:** `migrations/bin/cli` (Thor) becomes `disco`
  (Samovar), with each gem registering its own commands. Same surface —
  `convert`, `import`, `upload`, `schema generate|validate|…` — and Rails is
  still booted lazily.
- **Isolated test suites:** each gem runs its own no-Rails specs in a new CI
  job, while the existing job keeps running the Rails-integration specs.
2026-06-02 22:20:03 +02:00

50 lines
No EOL
1.8 KiB
Text
Vendored

source_db_path: "/path/to/your/db.sqlite3"
output_db_path: "/path/to/your/uploads.sqlite3"
root_paths:
- "/path/to/your/files"
- "/path/to/more/files"
# Files that are downloaded from URLs are cached in this directory.
download_cache_path: "/path/to/downloaded/files"
# The number of threads to use for processing uploads is calculated as:
# thread_count = [number of cores] * [thread_count_factor]
# The thread count will be doubled if uploads are stored on S3 because there's a higher latency.
thread_count_factor: 1.5
# Delete uploads from the output database that are not found in the source database.
delete_surplus_uploads: false
# Delete uploads from the output database that do not have a Discourse upload record.
delete_missing_uploads: false
# Check if files are missing in the upload store and update the database accordingly.
# Set to false and re-run the script afterwards if you want to create new uploads for missing files.
fix_missing: false
# Create optimized images for post uploads and avatars.
create_optimized_images: false
site_settings:
authorized_extensions: "*"
max_attachment_size_kb: 102_400
max_image_size_kb: 102_400
enable_s3_uploads: true
s3_upload_bucket: "your-bucket-name"
s3_region: "your-region"
s3_access_key_id: "your-access-key-id"
s3_secret_access_key: "your-secret-access-key"
s3_cdn_url: "https://your-cdn-url.com"
# Set this to true if the site is a multisite and configure the `multisite_db_name` accordingly
multisite: false
multisite_db_name: "default"
# Sometimes a file can be found at one of many locations. Here's a list of transformations that can
# be applied to the path to try and find the file. The first transformation that results in a file
# being found will be used.
path_replacements:
# - ["/foo/", "/bar"]
# - ["/foo/", "/bar/baz/"]