Fetch terrain tiles concurrently with persistent connections
Serial fetches (no connection reuse) measured ~650ms/tile, almost all TCP+TLS handshake — a real bbox at z0..15 is tens of thousands of tiles, so this was hours for a modest region. Now: 16 worker threads share a queue (I/O-bound, so the GVL doesn't get in the way), each keeping its own persistent connection across every tile it fetches. Measured live against the real Terrarium source: ~40ms/tile, ~16x faster. write_terrarium_mbtiles's SQLite writes and progress updates stay on one thread (draining a results Queue the workers feed), so nothing needed its own locking beyond the two Queues themselves.
This commit is contained in:
1 parent
bbe9f38bda
commit
d041c21270
2 files changed
+73
-33
No files matched your search
@@ -119,27 +119,34 @@ class PmtilesBuilderTest < Minitest::Test
|
||||
assert_equal status[:tiles_total], status[:tiles_done]
|
||||
end
|
||||
|
||||
# fetch_tile blocks after the very first call so the "running" progress
|
||||
# can be observed mid-flight, deterministically, rather than only ever
|
||||
# seeing the final done/total once the (very fast, faked) build finishes.
|
||||
# Every fetch_tile call blocks on a gate until explicitly told not to —
|
||||
# with CONCURRENT_TILE_FETCHES workers pulling from a shared queue, that
|
||||
# holds the whole pool at "in flight, none finished yet" deterministically
|
||||
# (a big enough bbox that z0..15 yields far more tiles than worker slots,
|
||||
# so most of the job is still queued up behind the blocked pool).
|
||||
def test_build_terrain_reports_progress_while_running
|
||||
ENV["PMTILES_BIN"] = @pmtiles_ok
|
||||
concurrency = Maptool::PmtilesBuilder::CONCURRENT_TILE_FETCHES
|
||||
gate = Queue.new
|
||||
first_call = true
|
||||
unblocked = false
|
||||
fetch_tile = lambda do |_z, _x, _y|
|
||||
gate.pop if first_call
|
||||
first_call = false
|
||||
gate.pop unless unblocked
|
||||
"bytes"
|
||||
end
|
||||
thread = Maptool::PmtilesBuilder.build_terrain([48.140, 17.100, 48.141, 17.101], data_dir: @data_dir, fetch_tile: fetch_tile)
|
||||
thread = Maptool::PmtilesBuilder.build_terrain([47.5, 16.5, 48.5, 17.5], data_dir: @data_dir, fetch_tile: fetch_tile)
|
||||
|
||||
20.times { break if Maptool::PmtilesBuilder.terrain_status[:tiles_total]; sleep 0.01 }
|
||||
# Let exactly one batch through (fills every worker's first call) then
|
||||
# check before releasing the rest — done should be capped at the pool
|
||||
# size, nowhere near the real total.
|
||||
concurrency.times { gate.push(nil) }
|
||||
sleep 0.05
|
||||
status = Maptool::PmtilesBuilder.terrain_status
|
||||
assert_equal "running", status[:state]
|
||||
assert_operator status[:tiles_total], :>, 0
|
||||
assert_equal 0, status[:tiles_done]
|
||||
assert_operator status[:tiles_total], :>, concurrency
|
||||
assert_operator status[:tiles_done], :<, status[:tiles_total]
|
||||
ensure
|
||||
gate.push(nil)
|
||||
unblocked = true
|
||||
concurrency.times { gate.push(nil) } # release whatever's still blocked in an in-progress gate.pop
|
||||
thread&.join
|
||||
end
|
||||
|
||||
|
||||
Reference in new issue
Block a user