The only visibility into downloaded data was the whole offline.pmtiles
row, and the only delete was "everything" (whole file + whole cache).
GET /api/offline-tiles/cache lists what's really in
data/geofabrik_cache/ right now — distinct from offline.pmtiles' own
meta.sources, which only reflects the last successful build and can
disagree with both the current selection and the cache (ticked-but-
not-Applied, or built-from-but-since-unticked).
The region picker now shows each cached country's size next to its
checkbox, with its own delete button (DELETE
/api/offline-tiles/cache/:code, already added but unused by the UI
until now) — removing just that entry rather than clearing the whole
cache.
Root cause of "planetiler exited with status 1: ... Unsupported format
tmp" (and likely the earlier unexplained "pmtiles exited with status
1"): Planetiler/go-pmtiles pick their output archive format from the
file extension, and the atomic-rename tmp files were named
"offline.pmtiles.tmp"/"terrain.pmtiles.tmp" — ".tmp" as the extension,
not ".pmtiles". Renamed to "offline.tmp.pmtiles"/"terrain.tmp.pmtiles".
Also, since build-status only ever said "running since X" with no way
to tell what it was actually doing:
- GET .../build-status now includes a `step` string ("fetching PL
(3/4) — done: SK, CZ — pending: HU", "tiling with planetiler", ...)
while a build is in progress.
- DELETE /api/offline-tiles / /api/terrain-tiles clear a built file,
its sidecar meta, and (for offline) the whole per-country download
cache — a manual "reclaim space" / "start over" action the region
picker's Apply never did on its own. DELETE
/api/offline-tiles/cache/:code clears just one country's cached
extract. All three are 409 while that kind's build is running, and
surfaced as a small "Delete" button on each row in the Offline data
panel.
Every region-selection change re-downloaded every selected country's
.osm.pbf from scratch, even ones already fetched by a prior build —
painful for multi-GB countries. Extracts now persist in
data/geofabrik_cache/, keyed by country, and are only re-fetched when
Geofabrik's Last-Modified for that country actually moved on (checked
with a cheap HEAD) or the file isn't cached yet.
Offline tiles were built from Protomaps' own basemap schema, needing a
completely separate hand-tuned theme (protomaps-themes-base + OFFLINE_THEME)
from the online Liberty style's ONLINE_STYLE_PATCH — the two could never be
made to match exactly no matter how much either side got tuned (today's
session found and fixed several mismatches: building opacity, park
boundaries, road casings, road widths).
Offline.pmtiles is now built with Planetiler's default OpenMapTiles profile
against Geofabrik per-country OSM extracts (osmium-merged when more than one
country is selected), the same vector schema Liberty already expects.
offlineStyle() now clones the same Liberty JSON (vendored as a local
snapshot for reliability with no internet) and applies the identical
ONLINE_STYLE_PATCH, just repointed at local PMTiles/glyphs/sprite — one
palette instead of two, correct on both sides by construction.
Also vendors Liberty's icon sprite for the first time (offline had none
before), adds Java/osmium-tool/planetiler.jar to the Docker image, and
removes the now-unused protomaps-themes-base dependency and the
theme-comparison debug page that found today's bugs.
The .pmtiles files themselves carry no record of what upstream
snapshot they came from — only a local mtime. Each successful build
now writes a small sidecar (offline.pmtiles.meta.json: the Protomaps
build date used; terrain.pmtiles.meta.json: the Mapterhorn source
names/md5sums) recording that, surfaced in the panel next to the
existing size/updated line.
New GET .../check-update routes compare the recorded build against
the live upstream index/manifest — true/false/null, null meaning
"don't know" (no local build yet, or the check itself failed) rather
than a false "no". Kept separate from the plain status endpoints
(which map.js's own topbar dot hits on every page load) since this
one reaches an external service; only called once when the Offline
data panel is actually opened.
Also: flag emoji next to each country in the picker (derived from the
ISO code via Unicode regional indicators, no image assets needed).
Replaces "regenerate for the current viewport" with what Locus/OsmAnd-
style apps give you: tick which countries you want offline (a Natural
Earth-derived checklist, public domain, vendored), Apply rebuilds
offline.pmtiles/terrain.pmtiles to cover the whole selection at once.
Multiple regions can be offline simultaneously (e.g. Slovakia + Austria
for a trip that crosses the border) — the selection itself is the
source of truth (data/offline_regions.json), not a set of per-country
files, so "remove a country" and "refresh a country" are both just
"change the selection, then Apply."
Confirmed against go-pmtiles' own source (not just its docs) before
committing to this shape: `extract --region=<geojson>` treats a
FeatureCollection's features as their union already (no dissolve/union
math needed on our side — pmtiles/region.go's UnmarshalRegion just
concatenates every feature's rings), and `merge` hard-errors on any
overlapping tile id between inputs (merge.go's prepareInputs). That
second fact rules out "extract each country separately, merge them
all" — adjacent countries' region-extracts would share border tiles,
since extraction copies whole tiles rather than clipping vector
content to the polygon. The actual fix: extract per *source archive*
(the Protomaps build, each intersecting Mapterhorn chunk), each call
covering the whole current selection via one --region= file — the
terrain pipeline already extracted-then-merged per source this way, it
just needed a multi-country region instead of a single bbox.
Mapterhorn (mapterhorn.com, Protomaps-affiliated) publishes pre-built
Terrarium-encoded terrain as PMTiles: a global planet.pmtiles (z0-12)
plus ~450 regional chunks (z13+). Same pmtiles extract --bbox=...
mechanism offline.pmtiles already uses, just against their files —
confirmed live against their real manifest and the documented
Interlaken example. Replaces fetching every individual tile ourselves
(even after this session's earlier concurrency fix, still tens of
minutes for a real region) with 1-3 extract calls plus a merge when
the bbox spans more than one chunk: seconds, not minutes.
Removes CONCURRENT_TILE_FETCHES, write_terrarium_mbtiles, tile_range,
lng/lat_to_tile_x/y, fetch_terrarium_tile, and the sqlite3 gem
entirely — none of it is needed once the tiles arrive pre-packaged.
dem.js's point-elevation lookup (shadow.js's terrain-shadow overlay)
hardcoded a 256px PNG tile assumption that Mapterhorn's 512px WebP
tiles would have silently broken — now reads the actually-decoded
tile's own size and mime type instead of assuming either.
Serial fetches (no connection reuse) measured ~650ms/tile, almost all
TCP+TLS handshake — a real bbox at z0..15 is tens of thousands of
tiles, so this was hours for a modest region. Now: 16 worker threads
share a queue (I/O-bound, so the GVL doesn't get in the way), each
keeping its own persistent connection across every tile it fetches.
Measured live against the real Terrarium source: ~40ms/tile, ~16x
faster.
write_terrarium_mbtiles's SQLite writes and progress updates stay on
one thread (draining a results Queue the workers feed), so nothing
needed its own locking beyond the two Queues themselves.
terrain.pmtiles can take minutes (tiles fetched one at a time) with no
way to tell a slow build from a stuck one. Now reports tiles_done/
tiles_total into the same status Hash the frontend already polls, and
the panel shows elapsed time (both files) plus tile counts (terrain
only) so the numbers visibly move every ~3s poll instead of sitting on
one static string.
A logged-in user can now trigger, from a new "Offline data" sidebar
panel, what was previously a fully manual go-pmtiles workflow
(docs/development.adoc): regenerate either file for the current map
viewport, no area-picker (same "visible area is the area" convention
the Overpass POI plugin already uses).
- lib/maptool/pmtiles_builder.rb: two independent background jobs
(Thread + Mutex-guarded status Hash, same shape as app.rb's own GNSS
poller), each writing to a .tmp path and atomically renaming into
place so a build never serves a partially-written file mid-run.
offline.pmtiles is a single `pmtiles extract` against Protomaps'
public daily basemap build (byte-range extraction, not a full
~120GB download). terrain.pmtiles has no equivalent public
Terrarium-encoded archive to extract from, so it fetches individual
Terrarium tiles itself, packs them into an intermediate MBTiles
(new sqlite3 gem), then the same `pmtiles convert`.
- 4 new app.rb routes (POST .../build, GET .../build-status) plus
size/mtime added to the existing availability endpoints.
- public/js/offline_data.js: new sidebar panel, polling build-status
the same self-rescheduling-setTimeout shape livetrack.js already
uses for its own refresh loop.
- Both network-heavy pieces (the Protomaps build index, per-tile
fetches) are injectable, so test/test_pmtiles_builder.rb exercises
the actual build pipeline against a fake `pmtiles` binary without
hitting a real network, matching this suite's existing
keep-it-offline convention for external services.
randy