Add in-app regeneration of offline.pmtiles and terrain.pmtiles
A logged-in user can now trigger, from a new "Offline data" sidebar panel, what was previously a fully manual go-pmtiles workflow (docs/development.adoc): regenerate either file for the current map viewport, no area-picker (same "visible area is the area" convention the Overpass POI plugin already uses). - lib/maptool/pmtiles_builder.rb: two independent background jobs (Thread + Mutex-guarded status Hash, same shape as app.rb's own GNSS poller), each writing to a .tmp path and atomically renaming into place so a build never serves a partially-written file mid-run. offline.pmtiles is a single `pmtiles extract` against Protomaps' public daily basemap build (byte-range extraction, not a full ~120GB download). terrain.pmtiles has no equivalent public Terrarium-encoded archive to extract from, so it fetches individual Terrarium tiles itself, packs them into an intermediate MBTiles (new sqlite3 gem), then the same `pmtiles convert`. - 4 new app.rb routes (POST .../build, GET .../build-status) plus size/mtime added to the existing availability endpoints. - public/js/offline_data.js: new sidebar panel, polling build-status the same self-rescheduling-setTimeout shape livetrack.js already uses for its own refresh loop. - Both network-heavy pieces (the Protomaps build index, per-tile fetches) are injectable, so test/test_pmtiles_builder.rb exercises the actual build pipeline against a fake `pmtiles` binary without hitting a real network, matching this suite's existing keep-it-offline convention for external services.
This commit is contained in:
1 parent
fba8ca10a2
commit
cd56ebe0a1
14 files changed
+740
-15
No files matched your search
+43
-13
@@ -27,18 +27,25 @@ reload (e.g. Ctrl/Cmd+Shift+R) if that happens.
|
||||
|
||||
To enable the offline style, place a PMTiles file at
|
||||
`data/offline.pmtiles` (see `docs/architecture.adoc` for how it is
|
||||
served and styled). It is not part of the repository — obtain or
|
||||
build one locally:
|
||||
served and styled). Logged-in users can generate this from the app
|
||||
itself — see "In-app regeneration" below — or it can be built by hand:
|
||||
|
||||
* The simplest source is a pre-built extract from
|
||||
https://app.protomaps.com[protomaps.com's extract builder] — pick a
|
||||
region and download it as `.pmtiles`. Its schema is what
|
||||
`protomaps-themes-base` (the styling library used for the offline
|
||||
style) expects.
|
||||
* Extract a region from Protomaps' own daily full-planet basemap build
|
||||
(https://build-metadata.protomaps.dev/builds.json[build index],
|
||||
files at `https://build.protomaps.com/<key>`, ~120GB, ODbL/OSM
|
||||
attribution) with the
|
||||
https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI's `extract`
|
||||
subcommand, which pulls only the byte ranges covering a bbox over
|
||||
HTTP range requests rather than the whole file:
|
||||
`pmtiles extract https://build.protomaps.com/<key> data/offline.pmtiles --bbox=west,south,east,north --maxzoom=15`.
|
||||
(The old `app.protomaps.com` extract-builder UI this section used to
|
||||
point at has since redirected elsewhere — this is the same
|
||||
underlying mechanism, just the CLI instead of a web UI.)
|
||||
* To convert some other vector tile source (e.g. an `.mbtiles` file)
|
||||
into PMTiles, use the
|
||||
https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI:
|
||||
`pmtiles convert in.mbtiles data/offline.pmtiles`.
|
||||
into PMTiles instead, use the same CLI's `convert` subcommand:
|
||||
`pmtiles convert in.mbtiles data/offline.pmtiles`. Its schema is
|
||||
what `protomaps-themes-base` (the styling library used for the
|
||||
offline style) expects.
|
||||
|
||||
Once the file is in place, `GET /api/offline-tiles` reports
|
||||
`available: true` and the map defaults to the offline style.
|
||||
@@ -50,15 +57,38 @@ The sidebar's "Terrain shading" toggle works online out of the box
|
||||
no API key). For it to work offline too, place a PMTiles archive of
|
||||
Terrarium-encoded `raster-dem` tiles at `data/terrain.pmtiles`
|
||||
(`GET /api/terrain-tiles` reports `available: true` once it's there,
|
||||
same pattern as offline map data above). It is not part of the
|
||||
repository — build one locally from a DEM source (e.g. Copernicus
|
||||
GLO-30 or SRTM) with a tool like
|
||||
same pattern as offline map data above). Also generatable in-app (see
|
||||
below) or by hand — build one locally from a DEM source (e.g.
|
||||
Copernicus GLO-30 or SRTM) with a tool like
|
||||
https://github.com/mapbox/rio-rgbify[rio-rgbify] (GeoTIFF ->
|
||||
Terrarium-encoded PNG tiles in an `.mbtiles`), then
|
||||
`pmtiles convert out.mbtiles data/terrain.pmtiles` with
|
||||
https://github.com/protomaps/go-pmtiles[go-pmtiles], same as
|
||||
`data/offline.pmtiles`.
|
||||
|
||||
== In-app regeneration
|
||||
|
||||
A logged-in user can also generate both files above from the app
|
||||
itself, no local tooling needed — the sidebar's "Offline data" icon
|
||||
(Offline hub group) shows each file's status (size, last updated) and
|
||||
a "Regenerate for current view" button. The region is always the
|
||||
current map viewport, same "no area-picker, the visible area is the
|
||||
area" convention the Overpass POI plugin already uses below. This
|
||||
wraps exactly the manual `go-pmtiles` workflows documented above
|
||||
(`lib/maptool/pmtiles_builder.rb`) — the server needs the
|
||||
https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI installed and
|
||||
on `PATH` (override with the `PMTILES_BIN` env var if it lives
|
||||
somewhere else); regenerating terrain.pmtiles this way fetches
|
||||
individual Terrarium tiles itself and packs them into an MBTiles file
|
||||
(the `sqlite3` gem) before the same `pmtiles convert` step, since no
|
||||
public Terrarium-encoded PMTiles archive exists to `extract` from the
|
||||
way offline.pmtiles above can.
|
||||
|
||||
A build runs in the background (`POST /api/offline-tiles/build` /
|
||||
`/api/terrain-tiles/build`, polled via the matching `*/build-status`
|
||||
route) — only one of each kind at a time; starting a second while one's
|
||||
running gets a 409.
|
||||
|
||||
== Camper-relevant POIs (Overpass)
|
||||
|
||||
`rake overpass:pois[south,west,north,east,out.geojson]` queries
|
||||
|
||||
Reference in new issue
Block a user