0
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2026-08-09 21:45:25 +08:00
A platform for community discussion. Free, open, simple. https://www.discourse.org
  • Ruby 58.5%
  • JavaScript 35.5%
  • HTML 3.1%
  • SCSS 2.8%
Find a file
Gerhard Schlager d925a0da13 MT: Inline upload mode for disco import
This is for small migrations where running the separate `disco upload`
step first is more hassle than it's worth. Until now `disco import`'s
uploads step just skipped when no files.db was configured. Now, when
there's no files.db, it uploads the source files itself, straight into
the live target site, using the exact same upload-creation code that
`disco upload` uses.

To share that code without duplicating it, I pulled the upload creation
out of `Tasks::Uploader` into a few plain collaborators under
`uploads/`:

- `SourceFileLocator` — finds a file under the configured root_paths
  (with path_replacements), or writes a tempfile from an inline data
  blob.
- `FileDownloader` — the URL download, cache dir, and #33546 error
  taxonomy. The original-filename store is injected: files.db-backed in
  `disco upload`, a plain in-memory Hash inline. Either way the cache
  dir works the same.
- `UploadCreationService` — the seam. Given one `upload_sources` row it
  locates or downloads the bytes, runs core's `UploadCreator`, checks
  the file actually reached the store, retries the few things worth
  retrying, and returns a frozen result. No queues, no DB writes — each
  caller persists its own way.

`Tasks::Uploader` is now a thin consumer of that service; its written
rows and behavior don't change, so the existing uploader and pipeline
specs still pass unchanged.

For inline mode the results land in the mappings DB instead of files.db:
`mapped.ids` (type UPLOADS) for id resolution, and a new
`mapped.upload_markdown` table (migration 003) for the rendered
reference. That gives the posts placeholder resolver one uniform read
path — `files.upload_results.markdown` when files.db is attached, else
`mapped.upload_markdown`. I only create and populate the table here; the
maps object on the in-progress posts branch reads it, so that side needs
to land the read path.

On ownership: `disco upload` uploads everything as the system user and
the copy step reassigns the real user afterwards. Inline mode has no
later copy step, so I resolve the mapped importer user (system user as a
fallback) up front and pass it to `create_for`, so the upload is owned
correctly at creation time. Keeping `user_id` a plain argument to the
service is what lets both paths share it.

Image processing is CPU-heavy, so inline mode reuses the existing
`Pipeline` (with the adaptive worker gate) rather than a sequential
loop. The catch is that the step's IntermediateDB connection (with
`mapped` attached) isn't thread-safe. So I materialize the whole pending
work list into memory on the step thread first, then start the pipeline
where only its single writer thread touches that connection — the step
thread just waits for `pipeline.run` to return. This assumes the work
list fits in memory, which is fine because inline mode is for small
migrations (documented at the call site).

Settings: `disco import` runs on the live target site, whose upload
settings (extensions, size limits, S3 credentials) are already real, so
inline mode leaves them alone. The one thing it does set is
`clean_up_uploads = false`, otherwise the cleanup job would sweep these
freshly created uploads before the later post steps attach them. Inline
mode needs `root_paths` (plus optional `path_replacements` and
`download_cache_path`), which go in a new `uploads:` section under
`config:` in importer.yml. If there are uploads to import and neither a
files.db nor that section is set, the run stops with a message telling
you to configure one or the other.

`Steps::OptimizedImages` stays a no-op without a files.db (a rebake
regenerates them) — PR 8's skip already covers that, no change needed.
2026-07-11 01:16:44 +02:00
.agents DEV: symlink to skills (#38573) 2026-03-18 09:41:56 +11:00
.claude DEV: symlink to skills (#38573) 2026-03-18 09:41:56 +11:00
.cursor/rules DEV: update various ai agent configurations (#34192) 2025-08-11 10:08:41 +10:00
.devcontainer Bump devcontainer image to 20260707-0040 (#41503) 2026-07-07 21:40:22 +10:00
.github DEV: Enable Playwright soft reset on CI for all system tests (#41573) 2026-07-09 10:02:42 +08:00
.skills DEV: Improve RSpec agent testing guidance (#41611) 2026-07-10 12:06:53 +08:00
.vscode DEV: Replace JS build system with Rolldown (#35963) 2026-05-29 11:11:55 +01:00
.zed DEV: add Zed editor configuration (#36958) 2026-01-06 23:30:45 +01:00
app UX: Tweaks to default theme screenshots (#41592) 2026-07-10 09:05:41 -04:00
bin DEV: add typescript support for core, themes and plugins (#41478) 2026-07-08 12:57:18 +01:00
config FEATURE: Support bulk user search in the admin users list (#41572) 2026-07-09 10:34:52 -03:00
db FEATURE: Add category scope for AI translations (#41585) 2026-07-10 09:39:20 +08:00
docs DEV: Fix headings for theme setting docs (#41574) 2026-07-09 11:41:54 +10:00
frontend FIX: Print site setting descriptions and setting enums correctly (#41617) 2026-07-10 21:44:47 +08:00
images
lib DEV: Fix stylesheet watcher file matching (#41593) 2026-07-09 16:30:28 -04:00
log
migrations MT: Inline upload mode for disco import 2026-07-11 01:16:44 +02:00
patches DEV: Replace JS build system with Rolldown (#35963) 2026-05-29 11:11:55 +01:00
plugins FEATURE: Add AI agent suspend/silence tools with inline approval review (#41497) 2026-07-10 10:59:51 -03:00
public UX: add Klipy branding to gifs picker (#41501) 2026-07-07 10:31:30 +04:00
script FEATURE: Support AI agent workflow uploads and runner (#41042) 2026-06-19 18:06:17 +02:00
spec SECURITY: Private UserField value disclosure via directory_items order sort side-channel (#41598) 2026-07-09 21:52:51 -05:00
stylelint-rules DEV: Exploring a token system (#38700) 2026-04-23 13:47:45 -05:00
test DEV: Clean up smoke-test (#40066) 2026-05-15 12:28:41 +02:00
themes UX: Tweaks to default theme screenshots (#41592) 2026-07-10 09:05:41 -04:00
vendor DEV: move discourse gifs to core (#40162) 2026-06-10 11:35:44 +04:00
.annotaterb.yml
.editorconfig
.git-blame-ignore-revs DEV: Add nested module style commit to .git-blame-ignore-revs 2026-03-20 10:08:56 +01:00
.gitattributes
.gitignore DEV: Replace JS build system with Rolldown (#35963) 2026-05-29 11:11:55 +01:00
.ignore
.jsdoc
.licensed.yml DEPS: Bump oauth from 1.1.3 to 1.1.5 (#40263) 2026-06-04 01:06:17 +02:00
.licensee.json DEV: Bump vitest and vite (#40450) 2026-06-08 10:29:44 +01:00
.npmrc DEV: prevent pnpm to ask for update (#34808) 2025-09-15 20:11:12 +02:00
.pnpmfile.cjs DEV: Overhaul typechecking configuration (#35794) 2025-11-12 12:54:34 +00:00
.prettierignore DEV: add typescript support for core, themes and plugins (#41478) 2026-07-08 12:57:18 +01:00
.prettierrc.cjs
.rspec
.rspec_parallel
.rubocop.yml DEV: Enable Style/RedundantSelf rubocop rule (#40098) 2026-05-19 19:27:45 +02:00
.ruby-gemset.sample
.streerc
AGENTS.md DEV: update various ai agent configurations (#34192) 2025-08-11 10:08:41 +10:00
AI-AGENTS.md DEV: Optimise number of lint calls in AI-AGENTS.md (#40594) 2026-06-05 11:48:49 +08:00
Brewfile Add image optimization and utility tools to Brewfile (#35940) 2025-12-16 16:35:19 +11:00
CLAUDE.md DEV: update various ai agent configurations (#34192) 2025-08-11 10:08:41 +10:00
CODEOWNERS DEV: Add Block API for declarative, validated UI extension points (#36810) 2026-02-25 17:36:43 -03:00
config.ru
CONTRIBUTING.md
COPYRIGHT.md
d
discourse.sublime-project
eslint.config.mjs DEV: add typescript support for core, themes and plugins (#41478) 2026-07-08 12:57:18 +01:00
Gemfile FEATURE: Route all GitHub API requests through one rate-limited client (#40637) 2026-06-15 10:59:10 +02:00
Gemfile.lock MT: Run conversion steps concurrently and split the heavy ones across cores 2026-07-03 11:48:25 +02:00
GEMINI.md DEV: update various ai agent configurations (#34192) 2025-08-11 10:08:41 +10:00
lefthook.yml DEV: add typescript support for core, themes and plugins (#41478) 2026-07-08 12:57:18 +01:00
LICENSE.txt
package.json DEV: add typescript support for core, themes and plugins (#41478) 2026-07-08 12:57:18 +01:00
pnpm-lock.yaml DEV: add typescript support for core, themes and plugins (#41478) 2026-07-08 12:57:18 +01:00
pnpm-workspace.yaml DEV: Configure pnpm builtDependencies (#41332) 2026-07-01 15:26:00 +01:00
Rakefile
README.md DEV: Link to Docker CLI documentation (#40375) 2026-05-28 22:10:18 +02:00
stylelint.config.mjs DEV: Exploring a token system (#38700) 2026-04-23 13:47:45 -05:00
translator.yml DEV: Add discourse-footnote client.yml translator config (#37178) 2026-01-16 15:23:50 +00:00
tsconfig-base.json DEV: Add Block API for declarative, validated UI extension points (#36810) 2026-02-25 17:36:43 -03:00
tsconfig.json DEV: Add a script for generating external types in discourse-types (#37095) 2026-03-09 20:37:43 +01:00
versions.json DEV: Begin development of v2026.7.0-latest 2026-06-30 16:19:27 +02:00

The online home for your community.

github-readme

You can self-host Discourse on your own infrastructure. But if you'd rather skip the setup, maintenance, and server management, we offer official Discourse hosting.

👉 Learn more about Discourse hosting

Discourse is a 100% open-source community platform for those who want complete control over how and where their site is run.

Our platform has been battle-tested for over a decade and continues to evolve to meet users needs for a powerful community platform.

With Discourse, you can:

  • 💬 Create discussion topics to foster meaningful conversations.

  • Connect in real-time with built-in chat.

  • 🎨 Customize your experience with an ever-growing selection of official and community themes.

  • 🤖 Enhance your community with plugins, from chatbots powered by Discourse AI to advanced tools like SQL analysis with the Data Explorer plugin.

To learn more, visit discourse.org and join our support community at meta.discourse.org.

Here are just a few of the incredible communities using Discourse:

discourse-communities

👉 Discover more communities using Discourse

Development

To get your environment set up, follow one of the setup guides:

Before you get started, ensure you have the following minimum versions: Ruby 3.4+, PostgreSQL 15, Redis 7.

For more information, check out the Developer Documentation.

Setting up Discourse

If you want to set up a Discourse forum for production use, see our Discourse Install Guide.

If you're looking for official hosting, see discourse.org/pricing.

Requirements

Discourse supports the latest, stable releases of all major browsers and platforms:

Browsers Tablets Phones
Apple Safari iPadOS iOS
Google Chrome Android Android
Microsoft Edge
Mozilla Firefox

Additionally, we aim to support Safari on iOS 16.4+.

Built With

  • Ruby on Rails — Our back end API is a Rails app. It responds to requests RESTfully in JSON.
  • Ember.js — Our front end is an Ember.js app that communicates with the Rails API.
  • PostgreSQL — Our main data store is in Postgres.
  • Redis — We use Redis as a cache and for transient data.
  • BrowserStack — We use BrowserStack to test on real devices and browsers.

Plus lots of Ruby Gems, a complete list of which is at /main/Gemfile.

Contributing

Build Status

Discourse is 100% free and open source. We encourage and support an active, healthy community that accepts contributions from the public including you!

Before contributing to Discourse:

  1. Please read the complete mission statements on discourse.org. Yes we actually believe this stuff; you should too.
  2. Read and sign the Electronic Discourse Forums Contribution License Agreement.
  3. Dig into CONTRIBUTING.MD, which covers submitting bugs, requesting new features, preparing your code for a pull request, etc.
  4. Always strive to collaborate with mutual respect.
  5. Not sure what to work on? We've got some ideas.

We look forward to seeing your pull requests!

Security

We take security very seriously at Discourse; all our code is 100% open source and peer reviewed. Please read our security guide for an overview of security measures in Discourse, or if you wish to report a security issue.

Security fixes are listed in the release notes for each version.

The Discourse Team

The original Discourse code contributors can be found in AUTHORS.MD. For a complete list of the many individuals that contributed to the design and implementation of Discourse, please refer to the official Discourse blog and GitHub's list of contributors.

Copyright 2014 - 2025 Civilized Discourse Construction Kit, Inc.

Licensed under the GNU General Public License Version 2.0 (or later); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:

https://www.gnu.org/licenses/old-licenses/gpl-2.0.txt

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Discourse logo and “Discourse Forum” ®, Civilized Discourse Construction Kit, Inc.

Accessibility

To guide our ongoing effort to build accessible software we follow the W3Cs Web Content Accessibility Guidelines (WCAG). If you'd like to report an accessibility issue that makes it difficult for you to use Discourse, email accessibility@discourse.org. For more information visit discourse.org/accessibility.

Dedication

Discourse is built with love, Internet style.

For over a decade, our amazing community has helped shape Discourse into what it is today. Your support, feedback, and contributions have been invaluable in making Discourse a powerful and versatile platform.

Were deeply grateful for every feature request, bug report, and discussion that has driven Discourse forward. Thank you for being a part of this journey—we couldnt have done it without you!