data/hiking_trails.geojson now generates automatically as the last
step of every offline.pmtiles build (PmtilesBuilder.extract_hiking_trails,
best-effort — a failure here doesn't fail the whole build), merged
across all selected countries with cross-border dedup by OSM relation
id (HikingTrails.extract_merged). Served at GET /tiles/hiking_trails.geojson
and GET /api/hiking-trails, fetched client-side only once the overlay
is actually toggled on.
hiking_trails.js now prefers this local file whenever it exists,
regardless of which base map (online/offline) is active — same
"prefer local when available" story as HillshadeControl already uses
for terrain. Offline click hit-tests the real vector layer directly
(queryRenderedFeatures) instead of querying Overpass — instant, no
network. Coordinates are rounded to 5dp to keep the combined file
smaller; colour tag values are normalized against a known-safe set
before ever reaching a style paint property.
Verified extract_merged (new: combines multiple countries' extracts,
dedupes overlapping relations) against two real cached country PBFs
on brick directly, not just fixtures.
Verified live on brick against real cached extracts: a plain Geofabrik
PBF's ways only carry bare node references, not coordinates, so
locations_on_ways on the final `cat -f opl` had nothing to embed —
every way came out as e.g. n123xy, empty x/y, and every relation got
dropped for having no usable geometry (0 routes for Macedonia).
osmium add-locations-to-ways between the tags-filter and cat steps
resolves those references first. Re-verified after the fix: 45 routes
for Macedonia, 3009 for Slovakia, with real colour/ref/operator/
distance (326 red, 782 yellow, 510 blue, 902 green KST-style routes).
No custom Planetiler profile needed after all — Maptool::HikingTrails
runs osmium tags-filter r/route=hiking against the same .osm.pbf
PmtilesBuilder already downloads per country for offline.pmtiles, then
osmium cat -f opl (locations_on_ways=true) to get member-way geometry
back as plain text, no separate node-id lookup table to build.
No general OSM tool stitches a route relation into one line (routes
are a semantic grouping, not a geometry type) — parse_opl does that
walk itself, same as the Overpass-based click handler already does
client-side. Kept pure/testable apart from the two osmium subprocess
calls, mirroring OverpassPois.fetch vs .to_geojson.
rake hiking:trails[pbf,out] to try it by hand against a real cached
extract. Not wired into the actual offline build or rendering yet —
still just a prototype, per TODO.adoc.
The single shared overpass-api.de instance varies wildly in load —
Promise.any against three independent public mirrors (also CORS-open,
verified) means only the slowest one determines wait time, not
whichever happens to be hit.
Also show the description tag (falls back to it for the title when a
route has no name, e.g. many KST route relations), the ref number, and
a "View on OpenStreetMap" link — the popup was skipping data that was
already in the response for free.
The Waymarked Trails raster overlay has no queryable features (just
pixels), so this answers "what trail is this" with a one-off Overpass
query on click instead: way(around:40m) finds candidate ways, then
rel(bw) walks backward to their parent route=hiking relations. Overpass
has permissive CORS, so this runs straight from the browser, no server
proxy needed — verified against a real KST route in the Tatras.
Route length isn't tagged directly, so it's summed from each member
way's own geometry (measure.js's existing haversine lineLength, now
exported and reused rather than duplicated). The colour tag is only
ever used as a CSS background after matching a strict hex/word regex —
untrusted free text otherwise.
New Overlays panel toggle, same plain on/off shape as Contour lines —
raster tiles from waymarkedtrails.org, which already color each route
by its OSM colour tag (the KST/KČT paint-blaze colors used in
Slovakia/Czechia, plain red elsewhere), so no styling work needed here.
Online only for now; offline (baked into offline.pmtiles) needs a
custom Planetiler profile for route relations, deferred until there's
a route planner/router to pair it with.
brick-build's job container (opensuse/leap:16.0) has no Node.js, and
every step in check/renovate (checkout, setup-ruby, setup-node) is a
JS-based Action that needs it just to start — both runs since the
brick-build switch failed immediately, and deploy never ran because it
depends on check passing. ubuntu-latest is shared with "Gitea Runner on
Cube" (not pinned to this specific machine), but that's fine: only
deploy (a unique label held by just brick's native runner) actually
needs to land on a specific box.
The 12-country selection's terrain.pmtiles was headed well past the
181GB an earlier 5-country build already produced at z15. Raster DEM
tiles don't compress away like vector ones (every tile is a real
image, even over empty terrain), so each extra zoom level roughly
quadruples storage. z13 is each Mapterhorn regional chunk's own native
minzoom — one raster level per chunk instead of a 15-13=2-level
pyramid on top — still enough for the hillshade toggle and the planned
slope readout. Online hillshade keeps its own z15 cap; this only
affects the offline copy.
The web UI was showing a raw ~500k-character stdout/stderr dump on a
failed build — go-pmtiles' extract logs one progress line per chunk,
which is fine on a terminal (it overwrites itself) but not captured
non-interactively. command_output now keeps just the tail, and the
status line renders it with preserved line breaks in a scrollable box
instead of one unreadable blob.
Root cause of the actual failure (terrain build dying ~11 minutes in
with an HTTP/2 stream reset): go-pmtiles' default --overfetch=0.05
merged this multi-country planet.pmtiles extract into a single ~60GB
request with no way to resume after a dropped stream. --overfetch=0
splits it into many smaller requests instead.
Renovate no longer needs Docker access for a one-shot docker-run job:
switched to setup-node + npm install -g renovate (matching labelprint's
workflow), so it fits the same brick-build runner as check instead of
the stale "deploy" label. Also note down the build-log readability/API
gaps flagged from a real failed terrain build.
Its own autodetection was leaving most of a 16-core host idle
(observed in production) — likely a container cgroup/JVM detection
quirk. Etc.nprocessors asks the OS directly instead.
App crashed on startup on brick (Permission denied @ dir_s_mkdir -
/app/data/gpx) — brick has SELinux enforcing, unlike cube, and Podman
denies a container write access to a bind-mounted host directory
without a relabel regardless of Unix permissions. Applied to both
maptool's ./data mount and kiwix's ZIM mount (same underlying cause,
would have hit this too once a real ZIM is placed there).
The only visibility into downloaded data was the whole offline.pmtiles
row, and the only delete was "everything" (whole file + whole cache).
GET /api/offline-tiles/cache lists what's really in
data/geofabrik_cache/ right now — distinct from offline.pmtiles' own
meta.sources, which only reflects the last successful build and can
disagree with both the current selection and the cache (ticked-but-
not-Applied, or built-from-but-since-unticked).
The region picker now shows each cached country's size next to its
checkbox, with its own delete button (DELETE
/api/offline-tiles/cache/:code, already added but unused by the UI
until now) — removing just that entry rather than clearing the whole
cache.
Real build failure once the tmp-filename fix got past the format-
detection error: "data/sources/lake_centerline.shp.zip does not exist.
Run with --download to fetch it". The OpenMapTiles profile needs a few
small non-OSM sources (water polygons, Natural Earth, lake
centerlines) alongside the .osm.pbf itself, and Planetiler only
fetches them when told to. They land under its own default
data/sources/, gitignored alongside the rest of data/ — fetched once,
reused by every later build.
Root cause of "planetiler exited with status 1: ... Unsupported format
tmp" (and likely the earlier unexplained "pmtiles exited with status
1"): Planetiler/go-pmtiles pick their output archive format from the
file extension, and the atomic-rename tmp files were named
"offline.pmtiles.tmp"/"terrain.pmtiles.tmp" — ".tmp" as the extension,
not ".pmtiles". Renamed to "offline.tmp.pmtiles"/"terrain.tmp.pmtiles".
Also, since build-status only ever said "running since X" with no way
to tell what it was actually doing:
- GET .../build-status now includes a `step` string ("fetching PL
(3/4) — done: SK, CZ — pending: HU", "tiling with planetiler", ...)
while a build is in progress.
- DELETE /api/offline-tiles / /api/terrain-tiles clear a built file,
its sidecar meta, and (for offline) the whole per-country download
cache — a manual "reclaim space" / "start over" action the region
picker's Apply never did on its own. DELETE
/api/offline-tiles/cache/:code clears just one country's cached
extract. All three are 409 while that kind's build is running, and
surfaced as a small "Delete" button on each row in the Offline data
panel.
Real cause of "pmtiles exited with status 1:" with nothing after the
colon: go-pmtiles (like some other CLIs) can write its actual error to
stdout, which was being silently discarded. Applies the same fix to
osmium/planetiler for consistency.
Every region-selection change re-downloaded every selected country's
.osm.pbf from scratch, even ones already fetched by a prior build —
painful for multi-GB countries. Extracts now persist in
data/geofabrik_cache/, keyed by country, and are only re-fetched when
Geofabrik's Last-Modified for that country actually moved on (checked
with a cheap HEAD) or the file isn't cached yet.
Planetiler 0.10.2 needs Java 21+; bookworm's default-jre-headless (and
its backports) only goes up to 17, causing a real build to fail with
UnsupportedClassVersionError. trixie's default-jre-headless is Java 21.
Geofabrik redirects europe/<slug>-latest.osm.pbf to a dated filename
(confirmed live: .../europe/austria-latest.osm.pbf -> .../europe/austria-
260910.osm.pbf) — a real build hit this immediately. Net::HTTP doesn't
follow redirects on its own; both the streaming download and the HEAD-based
update-check now do, up to a small limit.
Net::HTTP's plain #get loads the whole response body before returning —
fine for the small JSON manifests elsewhere in this file, but a real risk
for a multi-GB country .osm.pbf (Germany, Russia, ...). Streams to disk
via read_body instead.
Offline tiles were built from Protomaps' own basemap schema, needing a
completely separate hand-tuned theme (protomaps-themes-base + OFFLINE_THEME)
from the online Liberty style's ONLINE_STYLE_PATCH — the two could never be
made to match exactly no matter how much either side got tuned (today's
session found and fixed several mismatches: building opacity, park
boundaries, road casings, road widths).
Offline.pmtiles is now built with Planetiler's default OpenMapTiles profile
against Geofabrik per-country OSM extracts (osmium-merged when more than one
country is selected), the same vector schema Liberty already expects.
offlineStyle() now clones the same Liberty JSON (vendored as a local
snapshot for reliability with no internet) and applies the identical
ONLINE_STYLE_PATCH, just repointed at local PMTiles/glyphs/sprite — one
palette instead of two, correct on both sides by construction.
Also vendors Liberty's icon sprite for the first time (offline had none
before), adds Java/osmium-tool/planetiler.jar to the Docker image, and
removes the now-unused protomaps-themes-base dependency and the
theme-comparison debug page that found today's bugs.
Renders the bare protomaps-themes-base themes next to the app's actual
offline/online styles (each real app instance, embedded), with pan/zoom
kept in sync via the existing URL-hash mechanism, to make it easy to see
where the hand-tuned offline palette diverges from the online one.
Emoji flags are font glyphs, not shapes — no way to resize, border, or
tone them down to fit this app's controlled, hand-drawn-icon look
(same reasoning the topbar readout already settled earlier this
session). Vendored flag-icons' pre-cropped square SVGs (MIT) instead,
framed in a small rounded-square badge — the same convention
icons.js's MAP_MARKER_ICON already uses for framing a glyph, not a new
visual language.
The .pmtiles files themselves carry no record of what upstream
snapshot they came from — only a local mtime. Each successful build
now writes a small sidecar (offline.pmtiles.meta.json: the Protomaps
build date used; terrain.pmtiles.meta.json: the Mapterhorn source
names/md5sums) recording that, surfaced in the panel next to the
existing size/updated line.
New GET .../check-update routes compare the recorded build against
the live upstream index/manifest — true/false/null, null meaning
"don't know" (no local build yet, or the check itself failed) rather
than a false "no". Kept separate from the plain status endpoints
(which map.js's own topbar dot hits on every page load) since this
one reaches an external service; only called once when the Offline
data panel is actually opened.
Also: flag emoji next to each country in the picker (derived from the
ISO code via Unicode regional indicators, no image assets needed).
Replaces "regenerate for the current viewport" with what Locus/OsmAnd-
style apps give you: tick which countries you want offline (a Natural
Earth-derived checklist, public domain, vendored), Apply rebuilds
offline.pmtiles/terrain.pmtiles to cover the whole selection at once.
Multiple regions can be offline simultaneously (e.g. Slovakia + Austria
for a trip that crosses the border) — the selection itself is the
source of truth (data/offline_regions.json), not a set of per-country
files, so "remove a country" and "refresh a country" are both just
"change the selection, then Apply."
Confirmed against go-pmtiles' own source (not just its docs) before
committing to this shape: `extract --region=<geojson>` treats a
FeatureCollection's features as their union already (no dissolve/union
math needed on our side — pmtiles/region.go's UnmarshalRegion just
concatenates every feature's rings), and `merge` hard-errors on any
overlapping tile id between inputs (merge.go's prepareInputs). That
second fact rules out "extract each country separately, merge them
all" — adjacent countries' region-extracts would share border tiles,
since extraction copies whole tiles rather than clipping vector
content to the polygon. The actual fix: extract per *source archive*
(the Protomaps build, each intersecting Mapterhorn chunk), each call
covering the whole current selection via one --region= file — the
terrain pipeline already extracted-then-merged per source this way, it
just needed a multi-country region instead of a single bbox.
Mapterhorn (mapterhorn.com, Protomaps-affiliated) publishes pre-built
Terrarium-encoded terrain as PMTiles: a global planet.pmtiles (z0-12)
plus ~450 regional chunks (z13+). Same pmtiles extract --bbox=...
mechanism offline.pmtiles already uses, just against their files —
confirmed live against their real manifest and the documented
Interlaken example. Replaces fetching every individual tile ourselves
(even after this session's earlier concurrency fix, still tens of
minutes for a real region) with 1-3 extract calls plus a merge when
the bbox spans more than one chunk: seconds, not minutes.
Removes CONCURRENT_TILE_FETCHES, write_terrarium_mbtiles, tile_range,
lng/lat_to_tile_x/y, fetch_terrarium_tile, and the sqlite3 gem
entirely — none of it is needed once the tiles arrive pre-packaged.
dem.js's point-elevation lookup (shadow.js's terrain-shadow overlay)
hardcoded a 256px PNG tile assumption that Mapterhorn's 512px WebP
tiles would have silently broken — now reads the actually-decoded
tile's own size and mime type instead of assuming either.
Serial fetches (no connection reuse) measured ~650ms/tile, almost all
TCP+TLS handshake — a real bbox at z0..15 is tens of thousands of
tiles, so this was hours for a modest region. Now: 16 worker threads
share a queue (I/O-bound, so the GVL doesn't get in the way), each
keeping its own persistent connection across every tile it fetches.
Measured live against the real Terrarium source: ~40ms/tile, ~16x
faster.
write_terrarium_mbtiles's SQLite writes and progress updates stay on
one thread (draining a results Queue the workers feed), so nothing
needed its own locking beyond the two Queues themselves.
terrain.pmtiles can take minutes (tiles fetched one at a time) with no
way to tell a slow build from a stuck one. Now reports tiles_done/
tiles_total into the same status Hash the frontend already polls, and
the panel shows elapsed time (both files) plus tile counts (terrain
only) so the numbers visibly move every ~3s poll instead of sitting on
one static string.
The "Regenerate for current view" build was failing in production with
"could not run 'pmtiles'" — go-pmtiles was never actually installed
anywhere, just documented as something a developer runs by hand
locally (docs/development.adoc). Installs the static Linux binary from
go-pmtiles' own GitHub releases, architecture-matched via dpkg rather
than hardcoded amd64.
Verified the exact download/extract/exec sequence end-to-end outside
Docker (no daemon available in this sandbox to run a real build).
A logged-in user can now trigger, from a new "Offline data" sidebar
panel, what was previously a fully manual go-pmtiles workflow
(docs/development.adoc): regenerate either file for the current map
viewport, no area-picker (same "visible area is the area" convention
the Overpass POI plugin already uses).
- lib/maptool/pmtiles_builder.rb: two independent background jobs
(Thread + Mutex-guarded status Hash, same shape as app.rb's own GNSS
poller), each writing to a .tmp path and atomically renaming into
place so a build never serves a partially-written file mid-run.
offline.pmtiles is a single `pmtiles extract` against Protomaps'
public daily basemap build (byte-range extraction, not a full
~120GB download). terrain.pmtiles has no equivalent public
Terrarium-encoded archive to extract from, so it fetches individual
Terrarium tiles itself, packs them into an intermediate MBTiles
(new sqlite3 gem), then the same `pmtiles convert`.
- 4 new app.rb routes (POST .../build, GET .../build-status) plus
size/mtime added to the existing availability endpoints.
- public/js/offline_data.js: new sidebar panel, polling build-status
the same self-rescheduling-setTimeout shape livetrack.js already
uses for its own refresh loop.
- Both network-heavy pieces (the Protomaps build index, per-tile
fetches) are injectable, so test/test_pmtiles_builder.rb exercises
the actual build pipeline against a fake `pmtiles` binary without
hitting a real network, matching this suite's existing
keep-it-offline convention for external services.
5th Overlays item, generated client-side from the same online Terrarium
DEM tiles Terrain shading already fetches (no offline fallback — the
plugin's DemSource has no public hook to route through the pmtiles://
protocol the offline basemap uses, so this quietly renders nothing
without a network, same as Radar/the Wikipedia layer).
A plain vector source/layer under MapLibre's own tile lifecycle, no
manual refetch-on-move code needed. Thresholds are the plugin's own
generic starting point, not yet tuned against real terrain. Labels
skipped for now (font-name matching across the online/offline styles'
own glyph sets adds real risk for a nice-to-have, not the actual ask).
Top-left map control (topmost, above Home/zoom) with a text input:
"lat, lng" is parsed and jumped to directly (same plain-text shape the
topbar's coordinate readout already copies to the clipboard), anything
else is geocoded via Nominatim — OSM's own geocoder, free, no API key,
matching the online basemap's own OSM-derived style. Online only, same
story as this app's other non-basemap online features (Radar, the
Wikipedia layer) — no offline geocoding index to fall back to.
Self-hosted Bold woff2 only (the one weight .topbar__brand renders at)
under public/vendor/national-park-typeface/ — a router-carved-signage
display face for the "MAPTOOL" heading, matching the uppercase/
letter-spaced treatment it already had. Everything else stays on the
system font stack; this is a wordmark swap, not a body-text one.
docs/nps-resources.adoc/third-party-notices.adoc updated to record it.
WAYPOINT_ICON (coords) and PENDING_ICON (countdown) reused as-is; two new
hand-drawn SUNRISE_ICON/SUNSET_ICON since NPS's symbol set has no match —
same currentColor stroke style as the rest of icons.js, so they render
monochrome in the dark topbar rather than colored emoji.
sun.js's formatSunInfo now returns {sunrise, sunset, countdown} instead of
one joined string, so map.js can place its own icon in front of each
piece instead of the module owning any markup.
Same Open-Meteo call and cache as the daily outlook, just with an hourly
field added — a rolling window from now rather than "rest of today", so
it doesn't shrink to nothing right before midnight.
Shades the visible map by whether each point is currently blocked from
the sun by surrounding terrain (shademap.app-style), computed fully
offline via the same Terrarium DEM tiles the Terrain shading layer
already renders, plus this session's sun-position math and Radar's
checkbox+expandable UI shape.
dem.js pulls elevation-at-a-point out on its own since TODO.adoc's
slope readout and solar calculator will need the same lookup.
Both used to be their own separate sidebar icons; neither has
anything to manage beyond on/off, same as Wikipedia already got its
own Overlays entry for. Terrain shading is a plain checkbox item now.
Radar keeps its scrub/forecast controls, but behind an explicit
expand (clicking its name), separate from the checkbox — checking it
on shouldn't force the details open, and expanding to scrub/check the
forecast shouldn't also flip the layer on.
Both controls now build their own <li> (RadarControl.mount(li),
HillshadeControl.mount(li) — the latter moved into overlays.js
entirely, it has no reason to live in map.js once it's not building
its own sidebar icon) rather than a whole sidebar-item + flyout the
way they used to.
Also gave each Overlays item a leading icon: Wikipedia and Terrain
shading reuse existing NPS pictograms (WIKI_ICON, and WILDERNESS_ICON
newly exported from icons.js's forest entry — NPS has no dedicated
mountain/terrain glyph, wilderness-22.svg is the closest honest
match). Radar's is hand-drawn (a rain cloud) since NPS's set has
nothing weather-related to reuse.
The topbar's coordinate display was plain decimal degrees ("48.14611,
17.17722") — switched to DMS (48°08'46"N 17°10'38"E), the more
GPS-unit-like format Locus Pro itself uses, and made it click-to-copy
(plain decimal, since that's what pastes cleanly into another map/GPS
app — not the DMS text actually shown).
Added sunrise/sunset + a countdown to whichever's next, for the same
map-center point the coordinates already describe. Vendored SunCalc
(BSD-2-Clause, github.com/mourner/suncalc) rather than hand-rolling
the astronomical math — small, dependency-free, and correctness here
has real pitfalls (equation of time, refraction, polar day/night) a
first-pass reimplementation would risk getting subtly wrong.
Also added a test asserting every public/js/*.js module has a
versioned importmap entry in layout.erb — caught that overlays.js
(from the previous commit) was missing one, which would have quietly
skipped ASSET_VERSION's cache-busting for that file after every
deploy.
Wikipedia doesn't belong inside "Tracks & Points" — that panel is for
files you add/remove/reorder, while Wikipedia is a live layer with
nothing to manage, just on/off. Gave it its own sidebar panel
(#overlays-panel, same shell as LiveTrackPanelControl) instead, sized
to hold more such checklist-only layers later without a redesign.
Moved all the wiki layer logic out of GpxPanelControl into a new
OverlaysControl (public/js/overlays.js). svgToImageData/badgeImageId/
escapeHtml are now exported from gpx.js so it doesn't have to
reimplement them — the same three helpers a second overlay would also
need.
The old link only ever surfaced one nearby article at a time, buried
in whatever POI's popup you happened to open, and only for uploaded
files with a category. Replaced it with a standalone "Wikipedia"
toggle under the renamed "Tracks & Points" panel: enabling it adds a
live map layer that refetches GET /api/kiwix/nearby on every pan/zoom
and plots every nearby article as its own point.
kiwix-serve's own geo search never echoes back a result's coordinates
(confirmed live against the real production ZIM, both its HTML and
?format=xml output) — it only filters by distance from a point. So
Maptool::Kiwix.nearby fetches each matching article's own page too and
reads its <meta name="geo.position"> tag, which every geotagged
Wikipedia article carries. Tried parallelizing those per-article
fetches with threads first; measured no improvement against the real
ZIM (kiwix-serve appears to serialize requests regardless), so it
stayed sequential — ~1.6s for a 30-result page, fine for a
refetch-on-pan-end layer.
Clicking a point opens a small popup linking to the article (proxied
through the existing /kiwix/* passthrough). New WIKI_ICON (NPS's
bookstore-22.svg) marks these, kept out of POI_ICONS since that set
doubles as notes.js's icon-picker option list and this fixed-glyph
layer has no per-feature category to key off of.
randy