Exact

Exact Union RegExp

Gerhard Schlager 12a0656627 MT: Run conversion steps concurrently and split the heavy ones across cores ... This is the big one I've been building up to across the step-concurrency series: the converter now uses every core instead of grinding through one step at a time on a single CPU. The win scales with the number of cores, so the figure depends on the machine. On my laptop, running all the currently implemented Discourse converter steps dropped from 102s to 18s, about 5.5x faster, with all cores busy the whole time. Two things were slow before: steps ran one after another, and the few genuinely large steps (topic_users above all) ran single-threaded even when nothing else was happening. This PR tackles both. Concurrent steps. A dependency-aware scheduler runs independent steps at the same time, each with its own source connection. It's event-driven: as soon as a step finishes, it frees its fork and the next ready step starts, so the cores don't sit idle between steps. How many run at once is bounded by the cores it can use, and `--max-parallel-steps` lowers it. A pull model for workers. Instead of the parent reading every row and streaming items down a pipe to the workers, each worker now opens its own source connection and reads its own slice directly. Only progress goes back over the pipe, and only once every thousand items, so it's never the bottleneck. The parent's single-threaded read was the real ceiling, and this removes it, along with a fair bit of machinery (the old worker pool, the item serialization, the handshake). Partitioning the heavy steps. A step opts in with `partition_by`, and the scheduler splits it across forks, each reading one chunk of the key range. I kept it general: a dense numeric key is divided into even chunks straight from a cheap MIN/MAX, while a sparse numeric key or a text/UUID or composite key falls back to a sorted-key scan so every chunk holds a similar number of rows whatever the key type. The dialect-specific SQL lives in the Postgres adapter, so other sources can fill in their own later. One caveat I documented on `partition_by`: only use it on steps whose processing is order-independent, since the forks run concurrently and their output is merged. Sharded writes. Each worker writes to its own SQLite shard, and a background consolidator folds finished shards back into the run database off the step's critical path, so a step no longer lingers at 100% while its merge runs. Reliability. A bad row or a worker that dies mid-step no longer hangs the run or takes down the other steps running at the same time: failures are caught and surfaced per step. Debugging. Forks make a debugger awkward, so there's a `--no-fork` flag: it runs each step inline in the main process, one at a time, so a breakpoint in a step's `process` stops where you can actually use it. The data path is the same (it still writes a shard the consolidator merges), only the fork is gone. Output order is no longer deterministic with concurrency, so to check correctness I added a small dev script under `migrations/tooling/scripts` that compares two IntermediateDBs order-insensitively and used it to confirm a parallel run produces the same data as a serial one. The gem suites are green.		2026-07-03 11:48:25 +02:00
..
benchmarks	MT: Add a TUI progress reporter and use it in the importer too	2026-06-25 17:46:04 +02:00
compare_intermediate_dbs.rb	MT: Run conversion steps concurrently and split the heavy ones across cores	2026-07-03 11:48:25 +02:00