mirror of
https://github.com/discourse/discourse.git
synced 2026-08-06 13:08:40 +08:00
ce89da6325
When a step runs across many forks, we split it into one slice per fork. Every fork gets the same number of rows, but not the same amount of work. Cooking a post can be quick or slow, so the forks finish at very different times, and the step waits for the slowest one while the rest sit idle. On my forum's 1.6M posts, the forks got the same row count but finished anywhere between 34 and 123 seconds. Three changes: Even chunks. The partitioner had a shortcut for a numeric key that cut the range into equal pieces by value. That only works with about one row per value. For a key like topic_id, where a busy topic has thousands of rows and a quiet one a handful, the pieces came out very uneven. Now it always samples the sorted key, so every chunk holds about the same number of rows. Work stealing. Instead of one slice per fork, we split the step into many small chunks and let each fork grab the next one as it finishes. A fork with cheap chunks keeps pulling more, so they all finish around the same time and the long tail goes away. They share a small queue to hand out the chunks (a pipe of chunk numbers they read one at a time), so it is a few reads per step and nothing per row. Each fork still writes its own shard. This replaces the slice-per-fork path, there is no switch to turn it off. Free up forks early. A partitioned step used to hold all its forks until it was done. Now each fork goes back to the scheduler the moment its worker finishes, so other steps can run on the freed cores instead of queueing behind the slow one. On my forum the forks now finish between 74 and 101 seconds, and the small single-fork steps run alongside the big one instead of after it. The fork count in the progress display counts down as workers finish, too. |
||
|---|---|---|
| .. | ||
| lib | ||
| support | ||
| spec_helper.rb | ||
| spec_setup.rb | ||
