Add in-app regeneration of offline.pmtiles and terrain.pmtiles
CI / check (push) Successful in 20s
CI / deploy (push) Successful in 1m52s

A logged-in user can now trigger, from a new "Offline data" sidebar
panel, what was previously a fully manual go-pmtiles workflow
(docs/development.adoc): regenerate either file for the current map
viewport, no area-picker (same "visible area is the area" convention
the Overpass POI plugin already uses).

- lib/maptool/pmtiles_builder.rb: two independent background jobs
  (Thread + Mutex-guarded status Hash, same shape as app.rb's own GNSS
  poller), each writing to a .tmp path and atomically renaming into
  place so a build never serves a partially-written file mid-run.
  offline.pmtiles is a single `pmtiles extract` against Protomaps'
  public daily basemap build (byte-range extraction, not a full
  ~120GB download). terrain.pmtiles has no equivalent public
  Terrarium-encoded archive to extract from, so it fetches individual
  Terrarium tiles itself, packs them into an intermediate MBTiles
  (new sqlite3 gem), then the same `pmtiles convert`.
- 4 new app.rb routes (POST .../build, GET .../build-status) plus
  size/mtime added to the existing availability endpoints.
- public/js/offline_data.js: new sidebar panel, polling build-status
  the same self-rescheduling-setTimeout shape livetrack.js already
  uses for its own refresh loop.
- Both network-heavy pieces (the Protomaps build index, per-tile
  fetches) are injectable, so test/test_pmtiles_builder.rb exercises
  the actual build pipeline against a fake `pmtiles` binary without
  hitting a real network, matching this suite's existing
  keep-it-offline convention for external services.
This commit is contained in:
randy committed 2026-09-11 01:25:08 +02:00
1 parent fba8ca10a2
commit cd56ebe0a1
14 files changed
+740 -15

No files matched your search

+43 -13
View File
@@ -27,18 +27,25 @@ reload (e.g. Ctrl/Cmd+Shift+R) if that happens.
To enable the offline style, place a PMTiles file at
`data/offline.pmtiles` (see `docs/architecture.adoc` for how it is
served and styled). It is not part of the repository — obtain or
build one locally:
served and styled). Logged-in users can generate this from the app
itself — see "In-app regeneration" below — or it can be built by hand:
* The simplest source is a pre-built extract from
https://app.protomaps.com[protomaps.com's extract builder] — pick a
region and download it as `.pmtiles`. Its schema is what
`protomaps-themes-base` (the styling library used for the offline
style) expects.
* Extract a region from Protomaps' own daily full-planet basemap build
(https://build-metadata.protomaps.dev/builds.json[build index],
files at `https://build.protomaps.com/<key>`, ~120GB, ODbL/OSM
attribution) with the
https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI's `extract`
subcommand, which pulls only the byte ranges covering a bbox over
HTTP range requests rather than the whole file:
`pmtiles extract https://build.protomaps.com/<key> data/offline.pmtiles --bbox=west,south,east,north --maxzoom=15`.
(The old `app.protomaps.com` extract-builder UI this section used to
point at has since redirected elsewhere — this is the same
underlying mechanism, just the CLI instead of a web UI.)
* To convert some other vector tile source (e.g. an `.mbtiles` file)
into PMTiles, use the
https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI:
`pmtiles convert in.mbtiles data/offline.pmtiles`.
into PMTiles instead, use the same CLI's `convert` subcommand:
`pmtiles convert in.mbtiles data/offline.pmtiles`. Its schema is
what `protomaps-themes-base` (the styling library used for the
offline style) expects.
Once the file is in place, `GET /api/offline-tiles` reports
`available: true` and the map defaults to the offline style.
@@ -50,15 +57,38 @@ The sidebar's "Terrain shading" toggle works online out of the box
no API key). For it to work offline too, place a PMTiles archive of
Terrarium-encoded `raster-dem` tiles at `data/terrain.pmtiles`
(`GET /api/terrain-tiles` reports `available: true` once it's there,
same pattern as offline map data above). It is not part of the
repository — build one locally from a DEM source (e.g. Copernicus
GLO-30 or SRTM) with a tool like
same pattern as offline map data above). Also generatable in-app (see
below) or by hand — build one locally from a DEM source (e.g.
Copernicus GLO-30 or SRTM) with a tool like
https://github.com/mapbox/rio-rgbify[rio-rgbify] (GeoTIFF ->
Terrarium-encoded PNG tiles in an `.mbtiles`), then
`pmtiles convert out.mbtiles data/terrain.pmtiles` with
https://github.com/protomaps/go-pmtiles[go-pmtiles], same as
`data/offline.pmtiles`.
== In-app regeneration
A logged-in user can also generate both files above from the app
itself, no local tooling needed — the sidebar's "Offline data" icon
(Offline hub group) shows each file's status (size, last updated) and
a "Regenerate for current view" button. The region is always the
current map viewport, same "no area-picker, the visible area is the
area" convention the Overpass POI plugin already uses below. This
wraps exactly the manual `go-pmtiles` workflows documented above
(`lib/maptool/pmtiles_builder.rb`) — the server needs the
https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI installed and
on `PATH` (override with the `PMTILES_BIN` env var if it lives
somewhere else); regenerating terrain.pmtiles this way fetches
individual Terrarium tiles itself and packs them into an MBTiles file
(the `sqlite3` gem) before the same `pmtiles convert` step, since no
public Terrarium-encoded PMTiles archive exists to `extract` from the
way offline.pmtiles above can.
A build runs in the background (`POST /api/offline-tiles/build` /
`/api/terrain-tiles/build`, polled via the matching `*/build-status`
route) — only one of each kind at a time; starting a second while one's
running gets a 409.
== Camper-relevant POIs (Overpass)
`rake overpass:pois[south,west,north,east,out.geojson]` queries