Fetch terrain tiles concurrently with persistent connections
CI / check (push) Successful in 20s
CI / deploy (push) Successful in 13s

Serial fetches (no connection reuse) measured ~650ms/tile, almost all
TCP+TLS handshake — a real bbox at z0..15 is tens of thousands of
tiles, so this was hours for a modest region. Now: 16 worker threads
share a queue (I/O-bound, so the GVL doesn't get in the way), each
keeping its own persistent connection across every tile it fetches.
Measured live against the real Terrarium source: ~40ms/tile, ~16x
faster.

write_terrarium_mbtiles's SQLite writes and progress updates stay on
one thread (draining a results Queue the workers feed), so nothing
needed its own locking beyond the two Queues themselves.
This commit is contained in:
randy committed 2026-09-11 12:04:13 +02:00
1 parent bbe9f38bda
commit d041c21270
2 files changed
+73 -33

No files matched your search

+18 -11
View File
@@ -119,27 +119,34 @@ class PmtilesBuilderTest < Minitest::Test
assert_equal status[:tiles_total], status[:tiles_done]
end
# fetch_tile blocks after the very first call so the "running" progress
# can be observed mid-flight, deterministically, rather than only ever
# seeing the final done/total once the (very fast, faked) build finishes.
# Every fetch_tile call blocks on a gate until explicitly told not to —
# with CONCURRENT_TILE_FETCHES workers pulling from a shared queue, that
# holds the whole pool at "in flight, none finished yet" deterministically
# (a big enough bbox that z0..15 yields far more tiles than worker slots,
# so most of the job is still queued up behind the blocked pool).
def test_build_terrain_reports_progress_while_running
ENV["PMTILES_BIN"] = @pmtiles_ok
concurrency = Maptool::PmtilesBuilder::CONCURRENT_TILE_FETCHES
gate = Queue.new
first_call = true
unblocked = false
fetch_tile = lambda do |_z, _x, _y|
gate.pop if first_call
first_call = false
gate.pop unless unblocked
"bytes"
end
thread = Maptool::PmtilesBuilder.build_terrain([48.140, 17.100, 48.141, 17.101], data_dir: @data_dir, fetch_tile: fetch_tile)
thread = Maptool::PmtilesBuilder.build_terrain([47.5, 16.5, 48.5, 17.5], data_dir: @data_dir, fetch_tile: fetch_tile)
20.times { break if Maptool::PmtilesBuilder.terrain_status[:tiles_total]; sleep 0.01 }
# Let exactly one batch through (fills every worker's first call) then
# check before releasing the rest — done should be capped at the pool
# size, nowhere near the real total.
concurrency.times { gate.push(nil) }
sleep 0.05
status = Maptool::PmtilesBuilder.terrain_status
assert_equal "running", status[:state]
assert_operator status[:tiles_total], :>, 0
assert_equal 0, status[:tiles_done]
assert_operator status[:tiles_total], :>, concurrency
assert_operator status[:tiles_done], :<, status[:tiles_total]
ensure
gate.push(nil)
unblocked = true
concurrency.times { gate.push(nil) } # release whatever's still blocked in an in-progress gate.pop
thread&.join
end