randy/maptool
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Build terrain.pmtiles from Mapterhorn instead of fetching tiles ourselves
CI / check (push) Successful in 18s
Details
CI / deploy (push) Successful in 1m52s
Details

Browse Source 
Mapterhorn (mapterhorn.com, Protomaps-affiliated) publishes pre-built
Terrarium-encoded terrain as PMTiles: a global planet.pmtiles (z0-12)
plus ~450 regional chunks (z13+). Same pmtiles extract --bbox=...
mechanism offline.pmtiles already uses, just against their files —
confirmed live against their real manifest and the documented
Interlaken example. Replaces fetching every individual tile ourselves
(even after this session's earlier concurrency fix, still tens of
minutes for a real region) with 1-3 extract calls plus a merge when
the bbox spans more than one chunk: seconds, not minutes.

Removes CONCURRENT_TILE_FETCHES, write_terrarium_mbtiles, tile_range,
lng/lat_to_tile_x/y, fetch_terrarium_tile, and the sqlite3 gem
entirely — none of it is needed once the tiles arrive pre-packaged.

dem.js's point-elevation lookup (shadow.js's terrain-shadow overlay)
hardcoded a 256px PNG tile assumption that Mapterhorn's 512px WebP
tiles would have silently broken — now reads the actually-decoded
tile's own size and mime type instead of assuming either.
This commit is contained in:
randy committed 2026-09-11 12:26:03 +02:00
1 parent d041c21270
commit f89dea5805
8 files changed
+224 -234
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

lib/maptool/pmtiles_builder.rb
+56 -126
View File
@@ -2,8 +2,8 @@ require "open3"
require "net/http"
require "uri"
require "json"
require "sqlite3"
require "tmpdir"
require "fileutils"
module Maptool
# Regenerates data/offline.pmtiles and data/terrain.pmtiles for a given
@@ -22,19 +22,20 @@ module Maptool
PROTOMAPS_BUILDS_INDEX = "https://build-metadata.protomaps.dev/builds.json"
PROTOMAPS_BUILD_BASE = "https://build.protomaps.com"
# Same public, no-API-key Terrarium tile source public/js/overlays.js's
# ONLINE_TERRAIN_TILES already fetches client-side — no shared constant
# between Ruby and JS, just the same literal URL template. Matches that
# source's own online maxzoom.
TERRARIUM_URL_TEMPLATE = "https://s3.amazonaws.com/elevation-tiles-prod/terrarium/%<z>d/%<x>d/%<y>d.png"
# Mapterhorn (a Protomaps-affiliated project) publishes Terrarium-
# encoded terrain as pre-built PMTiles archives: one global
# planet.pmtiles (z0-12) plus ~450 regional chunks (z13 up to a
# per-region max, 13-18) — same "extract only the bbox's byte ranges
# from a remote archive" story as offline.pmtiles above, so no more
# fetching individual tiles or building an MBTiles file ourselves the
# way an earlier version of this file did.
MAPTERHORN_MANIFEST_URL = "https://download.mapterhorn.com/download_urls.json"
MAPTERHORN_PLANET_NAME = "planet.pmtiles"
# Matches the online hillshade layer's own cap (public/js/overlays.js's
# ONLINE_TERRAIN_TILES source, maxzoom: 15) — applied uniformly even
# though some Mapterhorn chunks natively offer more, so the offline
# experience doesn't vary in detail from one region to the next.
TERRAIN_MAX_ZOOM = 15
# A real bbox at z0..15 is tens of thousands of tiles — serial fetches
# (the original ponytail'd version of this) measured ~0.65s/tile, almost
# all TCP+TLS handshake, no connection reuse: hours for a modest region.
# This many workers, each keeping its own persistent connection (see
# fetch_terrarium_tile), is a big enough win to matter without looking
# like abuse against a public bucket.
CONCURRENT_TILE_FETCHES = 16
OFFLINE_STATUS = { state: "idle" }
TERRAIN_STATUS = { state: "idle" }
@@ -71,16 +72,12 @@ module Maptool
Thread.new { run_job(OFFLINE_MUTEX, OFFLINE_STATUS) { extract_offline(bounds, data_dir, fetch_build_url) } }
end
# fetch_tile: same injectable-seam reasoning as fetch_build_url above,
# for the many individual Terrarium tile fetches this needs instead of
# one index lookup. This build can run for minutes with nothing else to
# show for it, so write_terrarium_mbtiles reports tiles_done/tiles_total
# into the status Hash as it goes — otherwise a poller sees "running"
# the whole time with no way to tell it apart from stuck.
def self.build_terrain(bounds, data_dir:, fetch_tile: method(:fetch_terrarium_tile))
# fetch_manifest: same injectable-seam reasoning as fetch_build_url
# above, for the one Mapterhorn manifest lookup this needs instead of
# the Protomaps build index.
def self.build_terrain(bounds, data_dir:, fetch_manifest: method(:fetch_mapterhorn_manifest))
begin_run!(TERRAIN_MUTEX, TERRAIN_STATUS)
progress = ->(done, total) { TERRAIN_MUTEX.synchronize { TERRAIN_STATUS.merge!(tiles_done: done, tiles_total: total) } }
Thread.new { run_job(TERRAIN_MUTEX, TERRAIN_STATUS) { build_terrain_pmtiles(bounds, data_dir, fetch_tile, progress) } }
Thread.new { run_job(TERRAIN_MUTEX, TERRAIN_STATUS) { build_terrain_pmtiles(bounds, data_dir, fetch_manifest) } }
end
def self.begin_run!(mutex, status)
@@ -128,122 +125,55 @@ module Maptool
# --- terrain.pmtiles ---
def self.build_terrain_pmtiles(bounds, data_dir, fetch_tile, progress)
# planet.pmtiles (z0-12, always) plus whichever regional chunks
# intersect the bbox (z13+, zero or more — most bboxes hit exactly one,
# a bbox straddling a chunk boundary hits two) are each `extract`ed on
# their own, then combined with `pmtiles merge` if there was more than
# one (planet + a chunk are disjoint by zoom level, so this is always a
# valid merge; go-pmtiles requires clustered, matching-tile-type inputs,
# which every Mapterhorn PMTiles archive already satisfies).
def self.build_terrain_pmtiles(bounds, data_dir, fetch_manifest)
south, west, north, east = bounds
tmp_mbtiles = File.join(Dir.tmpdir, "maptool-terrain-#{Process.pid}-#{Time.now.to_i}.mbtiles")
tmp_path = File.join(data_dir, "terrain.pmtiles.tmp")
items = fetch_manifest.call
planet = items.find { |item| item["name"] == MAPTERHORN_PLANET_NAME }
raise BuildError, "mapterhorn manifest is missing #{MAPTERHORN_PLANET_NAME}" unless planet
write_terrarium_mbtiles(tmp_mbtiles, south, west, north, east, fetch_tile, progress)
run_pmtiles!("convert", tmp_mbtiles, tmp_path)
chunks = items.select { |item| item["name"] != MAPTERHORN_PLANET_NAME && bbox_intersects?(item, south, west, north, east) }
scratch_dir = Dir.mktmpdir("maptool-terrain")
tmp_path = File.join(data_dir, "terrain.pmtiles.tmp")
extracts = [planet, *chunks].each_with_index.map do |item, i|
extract_path = File.join(scratch_dir, "extract-#{i}.pmtiles")
run_pmtiles!("extract", item["url"], extract_path, "--bbox=#{west},#{south},#{east},#{north}", "--maxzoom=#{TERRAIN_MAX_ZOOM}")
extract_path
end
if extracts.size == 1
File.rename(extracts.first, tmp_path)
else
run_pmtiles!("merge", *extracts, tmp_path)
end
File.rename(tmp_path, File.join(data_dir, "terrain.pmtiles"))
ensure
File.delete(tmp_mbtiles) if tmp_mbtiles && File.exist?(tmp_mbtiles)
FileUtils.remove_entry(scratch_dir) if scratch_dir && Dir.exist?(scratch_dir)
File.delete(tmp_path) if tmp_path && File.exist?(tmp_path)
end
private_class_method :build_terrain_pmtiles
# Terrarium PNG tiles for z0..TERRAIN_MAX_ZOOM over the bbox, packed
# into a minimal MBTiles (SQLite) archive — go-pmtiles' own `convert`
# only accepts MBTiles/PMTiles as input, no raw tile directory, and no
# public Terrarium-encoded PMTiles archive is known to exist to
# `extract` from the way offline.pmtiles does above.
#
# fetch_tile.call happens on CONCURRENT_TILE_FETCHES worker threads at
# once (I/O-bound — Ruby's GVL is released while a thread blocks on
# network I/O, so this genuinely overlaps); every SQLite write and
# progress update stays on this thread, draining a results Queue the
# workers feed, so nothing here needs its own locking.
def self.write_terrarium_mbtiles(path, south, west, north, east, fetch_tile, progress)
db = SQLite3::Database.new(path)
db.execute_batch(<<~SQL)
CREATE TABLE metadata (name TEXT, value TEXT);
CREATE TABLE tiles (zoom_level INTEGER, tile_column INTEGER, tile_row INTEGER, tile_data BLOB);
CREATE UNIQUE INDEX tile_index ON tiles (zoom_level, tile_column, tile_row);
SQL
{ "format" => "png", "name" => "terrain", "type" => "baselayer", "minzoom" => "0", "maxzoom" => TERRAIN_MAX_ZOOM.to_s }.each do |name, value|
db.execute("INSERT INTO metadata (name, value) VALUES (?, ?)", [name, value])
end
# tile_range per zoom computed once upfront (cheap — no fetching, just
# tile math), so progress can report a real total from the very first
# call instead of only once the run finishes.
work = (0..TERRAIN_MAX_ZOOM).flat_map { |z| tile_range(z, south, west, north, east).map { |x, y| [z, x, y] } }
total = work.size
done = 0
progress.call(done, total)
queue = Queue.new
work.each { |item| queue << item }
CONCURRENT_TILE_FETCHES.times { queue << nil } # one stop signal per worker
results = Queue.new
workers = CONCURRENT_TILE_FETCHES.times.map do
Thread.new do
while (item = queue.pop)
z, x, y = item
results << [z, x, y, fetch_tile.call(z, x, y)]
end
ensure
# Only set when fetch_tile is fetch_terrarium_tile (a test fake
# has no reason to touch this) — closes this worker's persistent
# connection rather than leaving the socket for GC to eventually
# notice.
Thread.current[:maptool_terrarium_http]&.finish
end
end
total.times do
z, x, y, bytes = results.pop
done += 1
progress.call(done, total)
next unless bytes
# MBTiles uses TMS row numbering (0 at the south) — the opposite of
# the XYZ scheme both the Terrarium source and this app's own online
# raster-dem layer use (0 at the north) — flipped here so go-pmtiles
# ends up serving the same tile at the same z/x/y this app already
# requests online.
tms_y = (2**z) - 1 - y
db.execute("INSERT OR REPLACE INTO tiles (zoom_level, tile_column, tile_row, tile_data) VALUES (?, ?, ?, ?)", [z, x, tms_y, bytes.b])
end
workers.each(&:join)
ensure
db&.close
def self.bbox_intersects?(item, south, west, north, east)
item["min_lon"] < east && item["max_lon"] > west && item["min_lat"] < north && item["max_lat"] > south
end
private_class_method :write_terrarium_mbtiles
private_class_method :bbox_intersects?
def self.tile_range(z, south, west, north, east)
n = 2**z
x_min, x_max = [lng_to_tile_x(west, n), lng_to_tile_x(east, n)].minmax
y_min, y_max = [lat_to_tile_y(north, n), lat_to_tile_y(south, n)].minmax
x_min.upto(x_max).flat_map { |x| y_min.upto(y_max).map { |y| [x, y] } }
end
private_class_method :tile_range
def self.fetch_mapterhorn_manifest
response = Net::HTTP.get_response(URI(MAPTERHORN_MANIFEST_URL))
raise BuildError, "could not reach #{MAPTERHORN_MANIFEST_URL}" unless response.is_a?(Net::HTTPSuccess)
def self.lng_to_tile_x(lng, n)
(((lng + 180) / 360) * n).floor.clamp(0, n - 1)
JSON.parse(response.body).fetch("items")
rescue JSON::ParserError
raise BuildError, "mapterhorn manifest was not valid JSON"
end
private_class_method :lng_to_tile_x
def self.lat_to_tile_y(lat, n)
lat_rad = lat * Math::PI / 180
(((1 - (Math.log(Math.tan(lat_rad) + (1 / Math.cos(lat_rad))) / Math::PI)) / 2) * n).floor.clamp(0, n - 1)
end
private_class_method :lat_to_tile_y
# One persistent connection per calling Thread (each of
# write_terrarium_mbtiles' worker threads calls this many times) instead
# of Net::HTTP.get_response's usual "new TCP+TLS handshake per call" —
# that handshake was most of the ~0.65s/tile this used to cost.
def self.fetch_terrarium_tile(z, x, y)
uri = URI(format(TERRARIUM_URL_TEMPLATE, z: z, x: x, y: y))
http = (Thread.current[:maptool_terrarium_http] ||= Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https"))
response = http.get(uri.request_uri)
response.is_a?(Net::HTTPSuccess) ? response.body : nil
rescue StandardError
nil
end
private_class_method :fetch_terrarium_tile
private_class_method :fetch_mapterhorn_manifest
# --- shared ---