mirror of
https://github.com/discourse/discourse.git
synced 2026-08-07 13:19:19 +08:00
03fe4e4977
This is the next step in moving the upload pipeline over to files.db. Until now `disco upload` wrote its own uploads.db with a JSON blob per row (`uploads(id, upload JSON_TEXT, ...)`). files.db has the real Discourse upload columns instead, plus a separate `upload_results` table that records what happened to each source row. The tasks now write through the generated `FilesDB::*` models rather than hand-rolled INSERTs and JSON. What this buys us: - The ad-hoc SQL the tasks ran against `upload ->> 'sha1'` and friends turns into plain column reads, so joins and queries are actually useful. - `status` and `skip_reason` are string enums (`ok`/`skipped`/`error`, `file_not_found`/`download_error`/...), so they're greppable and easy to eyeball. Details: - The uploader writes one `upload_results` row per source row (id, status, skip_reason, skip_details, markdown, upload_id) and inserts into `files.uploads` only for staging upload ids it hasn't written yet. Several source rows can dedup onto one upload via sha1 (UploadCreator hands back the existing record), so the writer tracks seen staging ids and references them from later results. - I adopted the error taxonomy from #33546: `UploadSizeExceededError < DownloadFailedError`, unlinking the partial file on oversize, wrapping download failures in `DownloadFailedError`, and using `response.value` for HTTP >= 400. Oversize maps to `upload_size_exceeded`, other download failures to `download_error`. Behavior change worth calling out: a download failure now records an error result instead of silently dropping the row. To retry those, the existing delete-missing settings clear the null-upload results so they get reprocessed. - Fixed the last cross-thread hazard in the download cache: the `downloads` table is read into a Hash once in `before_run`, so worker threads no longer touch the DB connection to look up a cached filename. Fresh records still ride back on results and are written (and cached) by the single writer thread. - The optimizer writes one `optimized_images` row per generated image and reads its work from `upload_results JOIN uploads` real columns. I left its IntermediateDB queries (`posts.upload_ids`, `users.avatar_upload_id`) verbatim even though they're stale against the current IDB schema; that's a later PR. - The fixer reads uploads straight from the `uploads` columns and, on a missing store file, drops the upload, its optimized images, and the results pointing at it so the source rows get recreated on the next run. - Plumbing: `uploads.rb` migrates files.db and sets up `Database::FilesDB`; the importer config key `uploads_db` becomes `files_db` and the executor attaches it via the new schema path. Deleted the old `uploads_db_schema` and its `UPLOADS_DB_SCHEMA_PATH`. Known window: `disco import`'s `Steps::Uploads` still reads the old JSON shape (`up.upload`), so the import uploads step will fail against a files.db until the next PR in the series adapts it. I did not touch the step here. |
||
|---|---|---|
| .. | ||
| bin | ||
| config | ||
| db | ||
| lib | ||
| spec | ||
| .rspec | ||
| Gemfile | ||
| migrations-core.gemspec | ||
