Add a country-based region picker for offline data
Replaces "regenerate for the current viewport" with what Locus/OsmAnd- style apps give you: tick which countries you want offline (a Natural Earth-derived checklist, public domain, vendored), Apply rebuilds offline.pmtiles/terrain.pmtiles to cover the whole selection at once. Multiple regions can be offline simultaneously (e.g. Slovakia + Austria for a trip that crosses the border) — the selection itself is the source of truth (data/offline_regions.json), not a set of per-country files, so "remove a country" and "refresh a country" are both just "change the selection, then Apply." Confirmed against go-pmtiles' own source (not just its docs) before committing to this shape: `extract --region=<geojson>` treats a FeatureCollection's features as their union already (no dissolve/union math needed on our side — pmtiles/region.go's UnmarshalRegion just concatenates every feature's rings), and `merge` hard-errors on any overlapping tile id between inputs (merge.go's prepareInputs). That second fact rules out "extract each country separately, merge them all" — adjacent countries' region-extracts would share border tiles, since extraction copies whole tiles rather than clipping vector content to the polygon. The actual fix: extract per *source archive* (the Protomaps build, each intersecting Mapterhorn chunk), each call covering the whole current selection via one --region= file — the terrain pipeline already extracted-then-merged per source this way, it just needed a multi-country region instead of a single bbox.
This commit is contained in:
randy committed
1 parent
f89dea5805
commit
53c9a82af2
10 files changed
+442
-170
No files matched your search
@@ -0,0 +1 @@
|
||||
{"type":"FeatureCollection","features":[{"type":"Feature","properties":{"name":"Albania","iso_a2":"AL","iso_a3":"ALB"},"geometry":{"type":"Polygon","coordinates":[[[21.02004,40.842727],[20.99999,40.580004],[20.674997,40.435],[20.615,40.110007],[20.150016,39.624998],[19.98,39.694993],[19.960002,39.915006],[19.406082,40.250773],[19.319059,40.72723],[19.40355,41.409566],[19.540027,41.719986],[19.371769,41.877548],[19.371768,41.877551],[19.304486,42.195745],[19.738051,42.688247],[19.801613,42.500093],[20.0707,42.58863],[20.283755,42.32026],[20.52295,42.21787],[20.590247,41.855409],[20.590247,41.855404],[20.463175,41.515089],[20.605182,41.086226],[21.02004,40.842727]]]}},{"type":"Feature","properties":{"name":"Austria","iso_a2":"AT","iso_a3":"AUT"},"geometry":{"type":"Polygon","coordinates":[[[16.979667,48.123497],[16.903754,47.714866],[16.340584,47.712902],[16.534268,47.496171],[16.202298,46.852386],[16.011664,46.683611],[15.137092,46.658703],[14.632472,46.431817],[13.806475,46.509306],[12.376485,46.767559],[12.153088,47.115393],[11.164828,46.941579],[11.048556,46.751359],[10.442701,46.893546],[9.932448,46.920728],[9.47997,47.10281],[9.632932,47.347601],[9.594226,47.525058],[9.896068,47.580197],[10.402084,47.302488],[10.544504,47.566399],[11.426414,47.523766],[12.141357,47.703083],[12.62076,47.672388],[12.932627,47.467646],[13.025851,47.637584],[12.884103,48.289146],[13.243357,48.416115],[13.595946,48.877172],[14.338898,48.555305],[14.901447,48.964402],[15.253416,49.039074],[16.029647,48.733899],[16.499283,48.785808],[16.960288,48.596982],[16.879983,48.470013],[16.979667,48.123497]]]}},{"type":"Feature","properties":{"name":"Belarus","iso_a2":"BY","iso_a3":"BLR"},"geometry":{"type":"Polygon","coordinates":[[[28.176709,56.16913],[29.229513,55.918344],[29.371572,55.670091],[29.896294,55.789463],[30.873909,55.550976],[30.971836,55.081548],[30.757534,54.811771],[31.384472,54.157056],[31.791424,53.974639],[31.731273,53.794029],[32.405599,53.618045],[32.693643,53.351421],[32.304519,53.132726],[31.49764,53.16743],[31.305201,53.073996],[31.540018,52.742052],[31.78597,52.10168],[31.785992,52.101678],[30.927549,52.042353],[30.619454,51.822806],[30.555117,51.319503],[30.157364,51.416138],[29.254938,51.368234],[28.992835,51.602044],[28.617613,51.427714],[28.241615,51.572227],[27.454066,51.592303],[26.337959,51.832289],[25.327788,51.910656],[24.553106,51.888461],[24.005078,51.617444],[23.527071,51.578454],[23.508002,52.023647],[23.199494,52.486977],[23.799199,52.691099],[23.804935,53.089731],[23.527536,53.470122],[23.484128,53.912498],[24.450684,53.905702],[25.536354,54.282423],[25.768433,54.846963],[26.588279,55.167176],[26.494331,55.615107],[27.10246,55.783314],[28.176709,56.16913]]]}},{"type":"Feature","properties":{"name":"Belgium","iso_a2":"BE","iso_a3":"BEL"},"geometry":{"type":"Polygon","coordinates":[[[6.156658,50.803721],[6.043073,50.128052],[5.782417,50.090328],[5.674052,49.529484],[4.799222,49.985373],[4.286023,49.907497],[3.588184,50.378992],[3.123252,50.780363],[2.658422,50.796848],[2.513573,51.148506],[3.314971,51.345781],[3.315011,51.345777],[3.314971,51.345755],[4.047071,51.267259],[4.973991,51.475024],[5.606976,51.037298],[6.156658,50.803721]]]}},{"type":"Feature","properties":{"name":"Bosnia and Herz.","iso_a2":"BA","iso_a3":"BIH"},"geometry":{"type":"Polygon","coordinates":[[[18.56,42.65],[17.674922,43.028563],[17.297373,43.446341],[16.916156,43.667722],[16.456443,44.04124],[16.23966,44.351143],[15.750026,44.818712],[15.959367,45.233777],[16.318157,45.004127],[16.534939,45.211608],[17.002146,45.233777],[17.861783,45.06774],[18.553214,45.08159],[19.005485,44.860234],[19.00548,44.86023],[19.36803,44.863],[19.11761,44.42307],[19.59976,44.03847],[19.454,43.5681],[19.21852,43.52384],[19.03165,43.43253],[18.70648,43.20011],[18.56,42.65]]]}},{"type":"Feature","properties":{"name":"Bulgaria","iso_a2":"BG","iso_a3":"BGR"},"geometry":{"type":"Polygon","coordinates":[[[22.65715,44.234923],[22.944832,43.823785],[23.332302,43.897011],[24.100679,43.741051],[25.569272,43.688445],[26.065159,43.943494],[27.2424,44.175986],[27.970107,43.812468],[28.558081,43.707462],[28.039095,43.293172],[27.673898,42.577892],[27.99672,42.007359],[27.135739,42.141485],[26.117042,41.826905],[26.106138,41.328899],[25.197201,41.234486],[24.492645,41.583896],[23.692074,41.309081],[22.952377,41.337994],[22.881374,41.999297],[22.380526,42.32026],[22.545012,42.461362],[22.436595,42.580321],[22.604801,42.898519],[22.986019,43.211161],[22.500157,43.642814],[22.410446,44.008063],[22.65715,44.234923]]]}},{"type":"Feature","properties":{"name":"Croatia","iso_a2":"HR","iso_a3":"HRV"},"geometry":{"type":"Polygon","coordinates":[[[16.564808,46.503751],[16.882515,46.380632],[17.630066,45.951769],[18.456062,45.759481],[18.829825,45.908872],[19.072769,45.521511],[19.390476,45.236516],[19.005485,44.860234],[18.553214,45.08159],[17.861783,45.06774],[17.002146,45.233777],[16.534939,45.211608],[16.318157,45.004127],[15.959367,45.233777],[15.750026,44.818712],[Line truncated
|
||||
+122
-28
@@ -6,17 +6,28 @@ require "tmpdir"
|
||||
require "fileutils"
|
||||
|
||||
module Maptool
|
||||
# Regenerates data/offline.pmtiles and data/terrain.pmtiles for a given
|
||||
# bbox by shelling out to go-pmtiles — the same tool docs/development.adoc
|
||||
# already documents running by hand. No job queue: each build runs on its
|
||||
# own background Thread and reports into a Mutex-guarded status Hash a
|
||||
# route can poll, the same shape app.rb's own GNSS poller (GNSS_LATEST/
|
||||
# GNSS_LATEST_MUTEX) already uses.
|
||||
# Regenerates data/offline.pmtiles and data/terrain.pmtiles for the
|
||||
# currently *selected* regions (data/offline_regions.json, an array of
|
||||
# Natural Earth iso_a2 country codes — see available_regions/
|
||||
# selected_regions/set_selected_regions below) by shelling out to
|
||||
# go-pmtiles — the same tool docs/development.adoc already documents
|
||||
# running by hand. No job queue: each build runs on its own background
|
||||
# Thread and reports into a Mutex-guarded status Hash a route can poll,
|
||||
# the same shape app.rb's own GNSS poller (GNSS_LATEST/GNSS_LATEST_MUTEX)
|
||||
# already uses.
|
||||
module PmtilesBuilder
|
||||
class BuildError < StandardError; end
|
||||
|
||||
# Natural Earth (public domain, no attribution required) 110m admin-0
|
||||
# countries, filtered to Europe and trimmed to name/iso_a2/iso_a3 —
|
||||
# see the plan this was built from for the exact filter. 110m is
|
||||
# deliberately coarse (a km or so of border precision, not
|
||||
# survey-grade) — fine for "which tiles to grab," a same-schema swap
|
||||
# to Natural Earth's 50m/10m cut if that ever needs to be tighter.
|
||||
REGIONS_FILE = File.join(__dir__, "data", "europe_countries.geojson")
|
||||
|
||||
# Protomaps' own daily full-planet basemap build index — extract_offline
|
||||
# below only pulls the HTTP byte ranges covering the requested bbox
|
||||
# below only pulls the HTTP byte ranges covering the requested region
|
||||
# (go-pmtiles's `extract` supports a remote source over range requests),
|
||||
# never the whole ~120GB file.
|
||||
PROTOMAPS_BUILDS_INDEX = "https://build-metadata.protomaps.dev/builds.json"
|
||||
@@ -25,7 +36,7 @@ module Maptool
|
||||
# Mapterhorn (a Protomaps-affiliated project) publishes Terrarium-
|
||||
# encoded terrain as pre-built PMTiles archives: one global
|
||||
# planet.pmtiles (z0-12) plus ~450 regional chunks (z13 up to a
|
||||
# per-region max, 13-18) — same "extract only the bbox's byte ranges
|
||||
# per-region max, 13-18) — same "extract only the region's byte ranges
|
||||
# from a remote archive" story as offline.pmtiles above, so no more
|
||||
# fetching individual tiles or building an MBTiles file ourselves the
|
||||
# way an earlier version of this file did.
|
||||
@@ -59,25 +70,97 @@ module Maptool
|
||||
TERRAIN_MUTEX.synchronize { TERRAIN_STATUS.replace(state: "idle") }
|
||||
end
|
||||
|
||||
# bounds: [south, west, north, east]. Returns once the background Thread
|
||||
# has started, not once the build finishes — raises BuildError (the
|
||||
# caller turns this into a 409) if one's already running.
|
||||
# --- region selection ---
|
||||
|
||||
def self.available_regions
|
||||
region_features.map { |f| { code: f["properties"]["iso_a2"], name: f["properties"]["name"] } }.sort_by { |r| r[:name] }
|
||||
end
|
||||
|
||||
def self.selected_regions(data_dir)
|
||||
path = selected_regions_path(data_dir)
|
||||
return [] unless File.exist?(path)
|
||||
|
||||
JSON.parse(File.read(path))
|
||||
rescue JSON::ParserError
|
||||
[]
|
||||
end
|
||||
|
||||
def self.set_selected_regions(data_dir, codes)
|
||||
unknown = codes - available_regions.map { |r| r[:code] }
|
||||
raise BuildError, "unknown region code(s): #{unknown.join(", ")}" unless unknown.empty?
|
||||
|
||||
File.write(selected_regions_path(data_dir), JSON.generate(codes))
|
||||
end
|
||||
|
||||
def self.selected_regions_path(data_dir)
|
||||
File.join(data_dir, "offline_regions.json")
|
||||
end
|
||||
private_class_method :selected_regions_path
|
||||
|
||||
def self.region_features
|
||||
@region_features ||= JSON.parse(File.read(REGIONS_FILE))["features"]
|
||||
end
|
||||
private_class_method :region_features
|
||||
|
||||
# Writes the FeatureCollection of just the selected countries into
|
||||
# `dir` — this *is* the file go-pmtiles' `--region=` reads. No
|
||||
# union/dissolve needed: go-pmtiles treats a FeatureCollection's
|
||||
# features as one combined MultiPolygon on its own (confirmed against
|
||||
# its own source, pmtiles/region.go's UnmarshalRegion). Also returns a
|
||||
# plain aggregate [south,west,north,east] (min/max over every ring
|
||||
# coordinate, no geometry library) used only for the Mapterhorn
|
||||
# chunk-intersection check below — approximate is fine there, the
|
||||
# `--region=` file is what does the real precise selection.
|
||||
def self.region_geojson_and_bbox(codes, dir)
|
||||
selected = region_features.select { |f| codes.include?(f["properties"]["iso_a2"]) }
|
||||
raise BuildError, "no matching regions for #{codes.join(", ")}" if selected.empty?
|
||||
|
||||
path = File.join(dir, "region.geojson")
|
||||
File.write(path, JSON.generate(type: "FeatureCollection", features: selected))
|
||||
|
||||
coords = selected.flat_map { |f| ring_coordinates(f["geometry"]) }
|
||||
lons = coords.map { |lng, _lat| lng }
|
||||
lats = coords.map { |_lng, lat| lat }
|
||||
[path, [lats.min, lons.min, lats.max, lons.max]]
|
||||
end
|
||||
private_class_method :region_geojson_and_bbox
|
||||
|
||||
def self.ring_coordinates(geometry)
|
||||
case geometry["type"]
|
||||
when "Polygon" then geometry["coordinates"].flatten(1)
|
||||
when "MultiPolygon" then geometry["coordinates"].flatten(2)
|
||||
else []
|
||||
end
|
||||
end
|
||||
private_class_method :ring_coordinates
|
||||
|
||||
# --- builds ---
|
||||
|
||||
# Returns once the background Thread has started, not once the build
|
||||
# finishes — raises BuildError (the caller turns this into an error
|
||||
# response) if nothing's selected yet, or one's already running.
|
||||
#
|
||||
# fetch_build_url exists as an injectable seam so tests can exercise the
|
||||
# rest of this (temp path, go-pmtiles invocation, atomic rename) without
|
||||
# a real network call — same reasoning test_helper.rb already gives for
|
||||
# keeping /api/plugins route tests off the real Overpass API.
|
||||
def self.build_offline(bounds, data_dir:, fetch_build_url: method(:latest_protomaps_build_url))
|
||||
def self.build_offline(data_dir:, fetch_build_url: method(:latest_protomaps_build_url))
|
||||
codes = selected_regions(data_dir)
|
||||
raise BuildError, "select at least one region first" if codes.empty?
|
||||
|
||||
begin_run!(OFFLINE_MUTEX, OFFLINE_STATUS)
|
||||
Thread.new { run_job(OFFLINE_MUTEX, OFFLINE_STATUS) { extract_offline(bounds, data_dir, fetch_build_url) } }
|
||||
Thread.new { run_job(OFFLINE_MUTEX, OFFLINE_STATUS) { extract_offline(codes, data_dir, fetch_build_url) } }
|
||||
end
|
||||
|
||||
# fetch_manifest: same injectable-seam reasoning as fetch_build_url
|
||||
# above, for the one Mapterhorn manifest lookup this needs instead of
|
||||
# the Protomaps build index.
|
||||
def self.build_terrain(bounds, data_dir:, fetch_manifest: method(:fetch_mapterhorn_manifest))
|
||||
def self.build_terrain(data_dir:, fetch_manifest: method(:fetch_mapterhorn_manifest))
|
||||
codes = selected_regions(data_dir)
|
||||
raise BuildError, "select at least one region first" if codes.empty?
|
||||
|
||||
begin_run!(TERRAIN_MUTEX, TERRAIN_STATUS)
|
||||
Thread.new { run_job(TERRAIN_MUTEX, TERRAIN_STATUS) { build_terrain_pmtiles(bounds, data_dir, fetch_manifest) } }
|
||||
Thread.new { run_job(TERRAIN_MUTEX, TERRAIN_STATUS) { build_terrain_pmtiles(codes, data_dir, fetch_manifest) } }
|
||||
end
|
||||
|
||||
def self.begin_run!(mutex, status)
|
||||
@@ -98,14 +181,16 @@ module Maptool
|
||||
|
||||
# --- offline.pmtiles ---
|
||||
|
||||
def self.extract_offline(bounds, data_dir, fetch_build_url)
|
||||
def self.extract_offline(codes, data_dir, fetch_build_url)
|
||||
source = fetch_build_url.call
|
||||
south, west, north, east = bounds
|
||||
scratch_dir = Dir.mktmpdir("maptool-offline")
|
||||
region_path, = region_geojson_and_bbox(codes, scratch_dir)
|
||||
tmp_path = File.join(data_dir, "offline.pmtiles.tmp")
|
||||
|
||||
run_pmtiles!("extract", source, tmp_path, "--bbox=#{west},#{south},#{east},#{north}", "--maxzoom=15")
|
||||
run_pmtiles!("extract", source, tmp_path, "--region=#{region_path}", "--maxzoom=15")
|
||||
File.rename(tmp_path, File.join(data_dir, "offline.pmtiles"))
|
||||
ensure
|
||||
FileUtils.remove_entry(scratch_dir) if scratch_dir && Dir.exist?(scratch_dir)
|
||||
File.delete(tmp_path) if tmp_path && File.exist?(tmp_path)
|
||||
end
|
||||
private_class_method :extract_offline
|
||||
@@ -126,25 +211,34 @@ module Maptool
|
||||
# --- terrain.pmtiles ---
|
||||
|
||||
# planet.pmtiles (z0-12, always) plus whichever regional chunks
|
||||
# intersect the bbox (z13+, zero or more — most bboxes hit exactly one,
|
||||
# a bbox straddling a chunk boundary hits two) are each `extract`ed on
|
||||
# their own, then combined with `pmtiles merge` if there was more than
|
||||
# one (planet + a chunk are disjoint by zoom level, so this is always a
|
||||
# valid merge; go-pmtiles requires clustered, matching-tile-type inputs,
|
||||
# which every Mapterhorn PMTiles archive already satisfies).
|
||||
def self.build_terrain_pmtiles(bounds, data_dir, fetch_manifest)
|
||||
south, west, north, east = bounds
|
||||
# intersect the selection's aggregate bbox (z13+, zero or more) are
|
||||
# each `extract`ed on their own using the *same* --region= file, then
|
||||
# combined with `pmtiles merge` if there was more than one. Always
|
||||
# safe to merge regardless of how many countries are selected or
|
||||
# whether they're adjacent: each Mapterhorn chunk is its own
|
||||
# pre-partitioned, non-overlapping source grid cell, so two different
|
||||
# chunks' extracts can never share a tile id — the only thing `merge`
|
||||
# refuses (see pmtiles/merge.go's own disjointness check). What would
|
||||
# NOT be safe is extracting the *same* source once per country and
|
||||
# merging those — adjacent countries' region-extracts would share
|
||||
# border tiles (extraction copies whole tiles, it doesn't clip vector
|
||||
# content to the polygon) — so this always does one extract per
|
||||
# *source archive*, never one per country.
|
||||
def self.build_terrain_pmtiles(codes, data_dir, fetch_manifest)
|
||||
items = fetch_manifest.call
|
||||
planet = items.find { |item| item["name"] == MAPTERHORN_PLANET_NAME }
|
||||
raise BuildError, "mapterhorn manifest is missing #{MAPTERHORN_PLANET_NAME}" unless planet
|
||||
|
||||
scratch_dir = Dir.mktmpdir("maptool-terrain")
|
||||
region_path, bbox = region_geojson_and_bbox(codes, scratch_dir)
|
||||
south, west, north, east = bbox
|
||||
|
||||
chunks = items.select { |item| item["name"] != MAPTERHORN_PLANET_NAME && bbox_intersects?(item, south, west, north, east) }
|
||||
|
||||
scratch_dir = Dir.mktmpdir("maptool-terrain")
|
||||
tmp_path = File.join(data_dir, "terrain.pmtiles.tmp")
|
||||
extracts = [planet, *chunks].each_with_index.map do |item, i|
|
||||
extract_path = File.join(scratch_dir, "extract-#{i}.pmtiles")
|
||||
run_pmtiles!("extract", item["url"], extract_path, "--bbox=#{west},#{south},#{east},#{north}", "--maxzoom=#{TERRAIN_MAX_ZOOM}")
|
||||
run_pmtiles!("extract", item["url"], extract_path, "--region=#{region_path}", "--maxzoom=#{TERRAIN_MAX_ZOOM}")
|
||||
extract_path
|
||||
end
|
||||
|
||||
|
||||
Reference in new issue
Block a user