Add in-app regeneration of offline.pmtiles and terrain.pmtiles

A logged-in user can now trigger, from a new "Offline data" sidebar
panel, what was previously a fully manual go-pmtiles workflow
(docs/development.adoc): regenerate either file for the current map
viewport, no area-picker (same "visible area is the area" convention
the Overpass POI plugin already uses).

- lib/maptool/pmtiles_builder.rb: two independent background jobs
  (Thread + Mutex-guarded status Hash, same shape as app.rb's own GNSS
  poller), each writing to a .tmp path and atomically renaming into
  place so a build never serves a partially-written file mid-run.
  offline.pmtiles is a single `pmtiles extract` against Protomaps'
  public daily basemap build (byte-range extraction, not a full
  ~120GB download). terrain.pmtiles has no equivalent public
  Terrarium-encoded archive to extract from, so it fetches individual
  Terrarium tiles itself, packs them into an intermediate MBTiles
  (new sqlite3 gem), then the same `pmtiles convert`.
- 4 new app.rb routes (POST .../build, GET .../build-status) plus
  size/mtime added to the existing availability endpoints.
- public/js/offline_data.js: new sidebar panel, polling build-status
  the same self-rescheduling-setTimeout shape livetrack.js already
  uses for its own refresh loop.
- Both network-heavy pieces (the Protomaps build index, per-tile
  fetches) are injectable, so test/test_pmtiles_builder.rb exercises
  the actual build pipeline against a fake `pmtiles` binary without
  hitting a real network, matching this suite's existing
  keep-it-offline convention for external services.

lib/maptool/pmtiles_builder.rb

@@ -0,0 +1,217 @@
+require "open3"
+require "net/http"
+require "uri"
+require "json"
+require "sqlite3"
+require "tmpdir"
+
+module Maptool
+  # Regenerates data/offline.pmtiles and data/terrain.pmtiles for a given
+  # bbox by shelling out to go-pmtiles — the same tool docs/development.adoc
+  # already documents running by hand. No job queue: each build runs on its
+  # own background Thread and reports into a Mutex-guarded status Hash a
+  # route can poll, the same shape app.rb's own GNSS poller (GNSS_LATEST/
+  # GNSS_LATEST_MUTEX) already uses.
+  module PmtilesBuilder
+    class BuildError < StandardError; end
+
+    # Protomaps' own daily full-planet basemap build index — extract_offline
+    # below only pulls the HTTP byte ranges covering the requested bbox
+    # (go-pmtiles's `extract` supports a remote source over range requests),
+    # never the whole ~120GB file.
+    PROTOMAPS_BUILDS_INDEX = "https://build-metadata.protomaps.dev/builds.json"
+    PROTOMAPS_BUILD_BASE = "https://build.protomaps.com"
+
+    # Same public, no-API-key Terrarium tile source public/js/overlays.js's
+    # ONLINE_TERRAIN_TILES already fetches client-side — no shared constant
+    # between Ruby and JS, just the same literal URL template. Matches that
+    # source's own online maxzoom.
+    TERRARIUM_URL_TEMPLATE = "https://s3.amazonaws.com/elevation-tiles-prod/terrarium/%<z>d/%<x>d/%<y>d.png"
+    TERRAIN_MAX_ZOOM = 15
+
+    OFFLINE_STATUS = { state: "idle" }
+    TERRAIN_STATUS = { state: "idle" }
+    OFFLINE_MUTEX = Mutex.new
+    TERRAIN_MUTEX = Mutex.new
+
+    def self.offline_status
+      OFFLINE_MUTEX.synchronize { OFFLINE_STATUS.dup }
+    end
+
+    def self.terrain_status
+      TERRAIN_MUTEX.synchronize { TERRAIN_STATUS.dup }
+    end
+
+    # Test-only: the status hashes are process-wide state (same caveat as
+    # GPX_STORE/LIVETRACK_STORE in app.rb), so the suite needs a way to
+    # reset them between tests the way AppTestHelper#teardown already does
+    # for those.
+    def self.reset!
+      OFFLINE_MUTEX.synchronize { OFFLINE_STATUS.replace(state: "idle") }
+      TERRAIN_MUTEX.synchronize { TERRAIN_STATUS.replace(state: "idle") }
+    end
+
+    # bounds: [south, west, north, east]. Returns once the background Thread
+    # has started, not once the build finishes — raises BuildError (the
+    # caller turns this into a 409) if one's already running.
+    #
+    # fetch_build_url exists as an injectable seam so tests can exercise the
+    # rest of this (temp path, go-pmtiles invocation, atomic rename) without
+    # a real network call — same reasoning test_helper.rb already gives for
+    # keeping /api/plugins route tests off the real Overpass API.
+    def self.build_offline(bounds, data_dir:, fetch_build_url: method(:latest_protomaps_build_url))
+      begin_run!(OFFLINE_MUTEX, OFFLINE_STATUS)
+      Thread.new { run_job(OFFLINE_MUTEX, OFFLINE_STATUS) { extract_offline(bounds, data_dir, fetch_build_url) } }
+    end
+
+    # fetch_tile: same injectable-seam reasoning as fetch_build_url above,
+    # for the many individual Terrarium tile fetches this needs instead of
+    # one index lookup.
+    def self.build_terrain(bounds, data_dir:, fetch_tile: method(:fetch_terrarium_tile))
+      begin_run!(TERRAIN_MUTEX, TERRAIN_STATUS)
+      Thread.new { run_job(TERRAIN_MUTEX, TERRAIN_STATUS) { build_terrain_pmtiles(bounds, data_dir, fetch_tile) } }
+    end
+
+    def self.begin_run!(mutex, status)
+      mutex.synchronize do
+        raise BuildError, "a build is already running" if status[:state] == "running"
+        status.replace(state: "running", started_at: Time.now.utc.iso8601)
+      end
+    end
+    private_class_method :begin_run!
+
+    def self.run_job(mutex, status)
+      yield
+      mutex.synchronize { status.merge!(state: "done", finished_at: Time.now.utc.iso8601).delete(:message) }
+    rescue => e
+      mutex.synchronize { status.merge!(state: "error", finished_at: Time.now.utc.iso8601, message: e.message) }
+    end
+    private_class_method :run_job
+
+    # --- offline.pmtiles ---
+
+    def self.extract_offline(bounds, data_dir, fetch_build_url)
+      source = fetch_build_url.call
+      south, west, north, east = bounds
+      tmp_path = File.join(data_dir, "offline.pmtiles.tmp")
+
+      run_pmtiles!("extract", source, tmp_path, "--bbox=#{west},#{south},#{east},#{north}", "--maxzoom=15")
+      File.rename(tmp_path, File.join(data_dir, "offline.pmtiles"))
+    ensure
+      File.delete(tmp_path) if tmp_path && File.exist?(tmp_path)
+    end
+    private_class_method :extract_offline
+
+    def self.latest_protomaps_build_url
+      response = Net::HTTP.get_response(URI(PROTOMAPS_BUILDS_INDEX))
+      raise BuildError, "could not reach #{PROTOMAPS_BUILDS_INDEX}" unless response.is_a?(Net::HTTPSuccess)
+
+      builds = JSON.parse(response.body)
+      raise BuildError, "protomaps build index was empty" if builds.empty?
+
+      "#{PROTOMAPS_BUILD_BASE}/#{builds.last.fetch("key")}"
+    rescue JSON::ParserError
+      raise BuildError, "protomaps build index was not valid JSON"
+    end
+    private_class_method :latest_protomaps_build_url
+
+    # --- terrain.pmtiles ---
+
+    def self.build_terrain_pmtiles(bounds, data_dir, fetch_tile)
+      south, west, north, east = bounds
+      tmp_mbtiles = File.join(Dir.tmpdir, "maptool-terrain-#{Process.pid}-#{Time.now.to_i}.mbtiles")
+      tmp_path = File.join(data_dir, "terrain.pmtiles.tmp")
+
+      write_terrarium_mbtiles(tmp_mbtiles, south, west, north, east, fetch_tile)
+      run_pmtiles!("convert", tmp_mbtiles, tmp_path)
+      File.rename(tmp_path, File.join(data_dir, "terrain.pmtiles"))
+    ensure
+      File.delete(tmp_mbtiles) if tmp_mbtiles && File.exist?(tmp_mbtiles)
+      File.delete(tmp_path) if tmp_path && File.exist?(tmp_path)
+    end
+    private_class_method :build_terrain_pmtiles
+
+    # Terrarium PNG tiles for z0..TERRAIN_MAX_ZOOM over the bbox, packed
+    # into a minimal MBTiles (SQLite) archive — go-pmtiles' own `convert`
+    # only accepts MBTiles/PMTiles as input, no raw tile directory, and no
+    # public Terrarium-encoded PMTiles archive is known to exist to
+    # `extract` from the way offline.pmtiles does above.
+    #
+    # ponytail: fetches tiles one at a time, serially — fine for a
+    # park-sized region, slow for a big one. Add a small bounded-concurrency
+    # fetch pool if regenerating a large area turns out to matter in
+    # practice.
+    def self.write_terrarium_mbtiles(path, south, west, north, east, fetch_tile)
+      db = SQLite3::Database.new(path)
+      db.execute_batch(<<~SQL)
+        CREATE TABLE metadata (name TEXT, value TEXT);
+        CREATE TABLE tiles (zoom_level INTEGER, tile_column INTEGER, tile_row INTEGER, tile_data BLOB);
+        CREATE UNIQUE INDEX tile_index ON tiles (zoom_level, tile_column, tile_row);
+      SQL
+      { "format" => "png", "name" => "terrain", "type" => "baselayer", "minzoom" => "0", "maxzoom" => TERRAIN_MAX_ZOOM.to_s }.each do |name, value|
+        db.execute("INSERT INTO metadata (name, value) VALUES (?, ?)", [name, value])
+      end
+
+      (0..TERRAIN_MAX_ZOOM).each do |z|
+        tile_range(z, south, west, north, east).each do |x, y|
+          bytes = fetch_tile.call(z, x, y)
+          next unless bytes
+
+          # MBTiles uses TMS row numbering (0 at the south) — the opposite
+          # of the XYZ scheme both the Terrarium source and this app's own
+          # online raster-dem layer use (0 at the north) — flipped here so
+          # go-pmtiles ends up serving the same tile at the same z/x/y this
+          # app already requests online.
+          tms_y = (2**z) - 1 - y
+          db.execute("INSERT OR REPLACE INTO tiles (zoom_level, tile_column, tile_row, tile_data) VALUES (?, ?, ?, ?)", [z, x, tms_y, bytes.b])
+        end
+      end
+    ensure
+      db&.close
+    end
+    private_class_method :write_terrarium_mbtiles
+
+    def self.tile_range(z, south, west, north, east)
+      n = 2**z
+      x_min, x_max = [lng_to_tile_x(west, n), lng_to_tile_x(east, n)].minmax
+      y_min, y_max = [lat_to_tile_y(north, n), lat_to_tile_y(south, n)].minmax
+      x_min.upto(x_max).flat_map { |x| y_min.upto(y_max).map { |y| [x, y] } }
+    end
+    private_class_method :tile_range
+
+    def self.lng_to_tile_x(lng, n)
+      (((lng + 180) / 360) * n).floor.clamp(0, n - 1)
+    end
+    private_class_method :lng_to_tile_x
+
+    def self.lat_to_tile_y(lat, n)
+      lat_rad = lat * Math::PI / 180
+      (((1 - (Math.log(Math.tan(lat_rad) + (1 / Math.cos(lat_rad))) / Math::PI)) / 2) * n).floor.clamp(0, n - 1)
+    end
+    private_class_method :lat_to_tile_y
+
+    def self.fetch_terrarium_tile(z, x, y)
+      response = Net::HTTP.get_response(URI(format(TERRARIUM_URL_TEMPLATE, z: z, x: x, y: y)))
+      response.is_a?(Net::HTTPSuccess) ? response.body : nil
+    rescue StandardError
+      nil
+    end
+    private_class_method :fetch_terrarium_tile
+
+    # --- shared ---
+
+    # ENV read lazily (not a load-time constant) so tests can point this at
+    # a fake script without a require-order dance.
+    def self.pmtiles_bin
+      ENV.fetch("PMTILES_BIN", "pmtiles")
+    end
+
+    def self.run_pmtiles!(*args)
+      _stdout, stderr, status = Open3.capture3(pmtiles_bin, *args)
+      raise BuildError, "pmtiles exited with status #{status.exitstatus}: #{stderr.strip}" unless status.success?
+    rescue Errno::ENOENT, Errno::EACCES => e
+      raise BuildError, "could not run '#{pmtiles_bin}' (#{e.message}) — is go-pmtiles installed and on PATH? See docs/development.adoc."
+    end
+    private_class_method :run_pmtiles!
+  end
+end