data/hiking_trails.geojson is now data/hiking_trails/<CODE>.geojson —
one file per country instead of a merge, generated per-source
(PmtilesBuilder.extract_hiking_trails), cached the same way
geofabrik_cache is (skipped once a country's own extract is already
newer than its source .osm.pbf, so growing the selection by one
country only re-extracts that one). GET /api/hiking-trails now lists
available country codes instead of a single file's status.
The Overlays panel's "Hiking trails" item becomes an expandable
per-country checklist once any offline extract exists (same shape as
Radar's own expand), replacing the plain on/off checkbox — nobody
covering a dozen countries' worth of offline map data needs all of
their hiking trails loaded and parsed at once, so each ticked country
fetches its own GeoJSON independently instead.
HikingTrails.extract_merged (server-side multi-country merge, no
longer needed now that the client picks per country) removed again.
Verified the new extract_hiking_trails + caching skip live on brick
against a real cached country PBF.
data/hiking_trails.geojson now generates automatically as the last
step of every offline.pmtiles build (PmtilesBuilder.extract_hiking_trails,
best-effort — a failure here doesn't fail the whole build), merged
across all selected countries with cross-border dedup by OSM relation
id (HikingTrails.extract_merged). Served at GET /tiles/hiking_trails.geojson
and GET /api/hiking-trails, fetched client-side only once the overlay
is actually toggled on.
hiking_trails.js now prefers this local file whenever it exists,
regardless of which base map (online/offline) is active — same
"prefer local when available" story as HillshadeControl already uses
for terrain. Offline click hit-tests the real vector layer directly
(queryRenderedFeatures) instead of querying Overpass — instant, no
network. Coordinates are rounded to 5dp to keep the combined file
smaller; colour tag values are normalized against a known-safe set
before ever reaching a style paint property.
Verified extract_merged (new: combines multiple countries' extracts,
dedupes overlapping relations) against two real cached country PBFs
on brick directly, not just fixtures.
The 12-country selection's terrain.pmtiles was headed well past the
181GB an earlier 5-country build already produced at z15. Raster DEM
tiles don't compress away like vector ones (every tile is a real
image, even over empty terrain), so each extra zoom level roughly
quadruples storage. z13 is each Mapterhorn regional chunk's own native
minzoom — one raster level per chunk instead of a 15-13=2-level
pyramid on top — still enough for the hillshade toggle and the planned
slope readout. Online hillshade keeps its own z15 cap; this only
affects the offline copy.
The web UI was showing a raw ~500k-character stdout/stderr dump on a
failed build — go-pmtiles' extract logs one progress line per chunk,
which is fine on a terminal (it overwrites itself) but not captured
non-interactively. command_output now keeps just the tail, and the
status line renders it with preserved line breaks in a scrollable box
instead of one unreadable blob.
Root cause of the actual failure (terrain build dying ~11 minutes in
with an HTTP/2 stream reset): go-pmtiles' default --overfetch=0.05
merged this multi-country planet.pmtiles extract into a single ~60GB
request with no way to resume after a dropped stream. --overfetch=0
splits it into many smaller requests instead.
The only visibility into downloaded data was the whole offline.pmtiles
row, and the only delete was "everything" (whole file + whole cache).
GET /api/offline-tiles/cache lists what's really in
data/geofabrik_cache/ right now — distinct from offline.pmtiles' own
meta.sources, which only reflects the last successful build and can
disagree with both the current selection and the cache (ticked-but-
not-Applied, or built-from-but-since-unticked).
The region picker now shows each cached country's size next to its
checkbox, with its own delete button (DELETE
/api/offline-tiles/cache/:code, already added but unused by the UI
until now) — removing just that entry rather than clearing the whole
cache.
Root cause of "planetiler exited with status 1: ... Unsupported format
tmp" (and likely the earlier unexplained "pmtiles exited with status
1"): Planetiler/go-pmtiles pick their output archive format from the
file extension, and the atomic-rename tmp files were named
"offline.pmtiles.tmp"/"terrain.pmtiles.tmp" — ".tmp" as the extension,
not ".pmtiles". Renamed to "offline.tmp.pmtiles"/"terrain.tmp.pmtiles".
Also, since build-status only ever said "running since X" with no way
to tell what it was actually doing:
- GET .../build-status now includes a `step` string ("fetching PL
(3/4) — done: SK, CZ — pending: HU", "tiling with planetiler", ...)
while a build is in progress.
- DELETE /api/offline-tiles / /api/terrain-tiles clear a built file,
its sidecar meta, and (for offline) the whole per-country download
cache — a manual "reclaim space" / "start over" action the region
picker's Apply never did on its own. DELETE
/api/offline-tiles/cache/:code clears just one country's cached
extract. All three are 409 while that kind's build is running, and
surfaced as a small "Delete" button on each row in the Offline data
panel.
Every region-selection change re-downloaded every selected country's
.osm.pbf from scratch, even ones already fetched by a prior build —
painful for multi-GB countries. Extracts now persist in
data/geofabrik_cache/, keyed by country, and are only re-fetched when
Geofabrik's Last-Modified for that country actually moved on (checked
with a cheap HEAD) or the file isn't cached yet.
Offline tiles were built from Protomaps' own basemap schema, needing a
completely separate hand-tuned theme (protomaps-themes-base + OFFLINE_THEME)
from the online Liberty style's ONLINE_STYLE_PATCH — the two could never be
made to match exactly no matter how much either side got tuned (today's
session found and fixed several mismatches: building opacity, park
boundaries, road casings, road widths).
Offline.pmtiles is now built with Planetiler's default OpenMapTiles profile
against Geofabrik per-country OSM extracts (osmium-merged when more than one
country is selected), the same vector schema Liberty already expects.
offlineStyle() now clones the same Liberty JSON (vendored as a local
snapshot for reliability with no internet) and applies the identical
ONLINE_STYLE_PATCH, just repointed at local PMTiles/glyphs/sprite — one
palette instead of two, correct on both sides by construction.
Also vendors Liberty's icon sprite for the first time (offline had none
before), adds Java/osmium-tool/planetiler.jar to the Docker image, and
removes the now-unused protomaps-themes-base dependency and the
theme-comparison debug page that found today's bugs.
Renders the bare protomaps-themes-base themes next to the app's actual
offline/online styles (each real app instance, embedded), with pan/zoom
kept in sync via the existing URL-hash mechanism, to make it easy to see
where the hand-tuned offline palette diverges from the online one.
The .pmtiles files themselves carry no record of what upstream
snapshot they came from — only a local mtime. Each successful build
now writes a small sidecar (offline.pmtiles.meta.json: the Protomaps
build date used; terrain.pmtiles.meta.json: the Mapterhorn source
names/md5sums) recording that, surfaced in the panel next to the
existing size/updated line.
New GET .../check-update routes compare the recorded build against
the live upstream index/manifest — true/false/null, null meaning
"don't know" (no local build yet, or the check itself failed) rather
than a false "no". Kept separate from the plain status endpoints
(which map.js's own topbar dot hits on every page load) since this
one reaches an external service; only called once when the Offline
data panel is actually opened.
Also: flag emoji next to each country in the picker (derived from the
ISO code via Unicode regional indicators, no image assets needed).
Replaces "regenerate for the current viewport" with what Locus/OsmAnd-
style apps give you: tick which countries you want offline (a Natural
Earth-derived checklist, public domain, vendored), Apply rebuilds
offline.pmtiles/terrain.pmtiles to cover the whole selection at once.
Multiple regions can be offline simultaneously (e.g. Slovakia + Austria
for a trip that crosses the border) — the selection itself is the
source of truth (data/offline_regions.json), not a set of per-country
files, so "remove a country" and "refresh a country" are both just
"change the selection, then Apply."
Confirmed against go-pmtiles' own source (not just its docs) before
committing to this shape: `extract --region=<geojson>` treats a
FeatureCollection's features as their union already (no dissolve/union
math needed on our side — pmtiles/region.go's UnmarshalRegion just
concatenates every feature's rings), and `merge` hard-errors on any
overlapping tile id between inputs (merge.go's prepareInputs). That
second fact rules out "extract each country separately, merge them
all" — adjacent countries' region-extracts would share border tiles,
since extraction copies whole tiles rather than clipping vector
content to the polygon. The actual fix: extract per *source archive*
(the Protomaps build, each intersecting Mapterhorn chunk), each call
covering the whole current selection via one --region= file — the
terrain pipeline already extracted-then-merged per source this way, it
just needed a multi-country region instead of a single bbox.
Mapterhorn (mapterhorn.com, Protomaps-affiliated) publishes pre-built
Terrarium-encoded terrain as PMTiles: a global planet.pmtiles (z0-12)
plus ~450 regional chunks (z13+). Same pmtiles extract --bbox=...
mechanism offline.pmtiles already uses, just against their files —
confirmed live against their real manifest and the documented
Interlaken example. Replaces fetching every individual tile ourselves
(even after this session's earlier concurrency fix, still tens of
minutes for a real region) with 1-3 extract calls plus a merge when
the bbox spans more than one chunk: seconds, not minutes.
Removes CONCURRENT_TILE_FETCHES, write_terrarium_mbtiles, tile_range,
lng/lat_to_tile_x/y, fetch_terrarium_tile, and the sqlite3 gem
entirely — none of it is needed once the tiles arrive pre-packaged.
dem.js's point-elevation lookup (shadow.js's terrain-shadow overlay)
hardcoded a 256px PNG tile assumption that Mapterhorn's 512px WebP
tiles would have silently broken — now reads the actually-decoded
tile's own size and mime type instead of assuming either.
A logged-in user can now trigger, from a new "Offline data" sidebar
panel, what was previously a fully manual go-pmtiles workflow
(docs/development.adoc): regenerate either file for the current map
viewport, no area-picker (same "visible area is the area" convention
the Overpass POI plugin already uses).
- lib/maptool/pmtiles_builder.rb: two independent background jobs
(Thread + Mutex-guarded status Hash, same shape as app.rb's own GNSS
poller), each writing to a .tmp path and atomically renaming into
place so a build never serves a partially-written file mid-run.
offline.pmtiles is a single `pmtiles extract` against Protomaps'
public daily basemap build (byte-range extraction, not a full
~120GB download). terrain.pmtiles has no equivalent public
Terrarium-encoded archive to extract from, so it fetches individual
Terrarium tiles itself, packs them into an intermediate MBTiles
(new sqlite3 gem), then the same `pmtiles convert`.
- 4 new app.rb routes (POST .../build, GET .../build-status) plus
size/mtime added to the existing availability endpoints.
- public/js/offline_data.js: new sidebar panel, polling build-status
the same self-rescheduling-setTimeout shape livetrack.js already
uses for its own refresh loop.
- Both network-heavy pieces (the Protomaps build index, per-tile
fetches) are injectable, so test/test_pmtiles_builder.rb exercises
the actual build pipeline against a fake `pmtiles` binary without
hitting a real network, matching this suite's existing
keep-it-offline convention for external services.
A plugin is a directory under plugins/ with a plugin.json manifest
({"label": "...", "command": "./run.sh"}) plus a script that reads
--south/--west/--north/--east and prints a GeoJSON FeatureCollection
to stdout (lib/maptool/plugins.rb) — that's the entire interface a
plugin needs to know about this app. The sidebar's new import icon
(logged in only, GET /api/plugins + POST /api/plugins/:name/run) lists
whatever's found and runs one for the current map viewport, saving its
output as a new .geojson file the same way an upload would be.
The point: a scraper for a site whose ToS forbids that (Park4Night,
iOverlander, Freecampsites) doesn't need to live in this public repo
to work here — its own directory just goes under plugins/ locally and
is never git add'ed, no special-casing needed anywhere.
plugins/overpass-pois wraps the existing rake overpass:pois query as
the first, in-repo, reference example of the contract.
Also fills in four modules (auth.js, locate.js, measure.js, notes.js)
missing from layout.erb's cache-busting importmap alongside adding
plugins.js's own entry — a pre-existing gap that meant they could keep
serving a stale cached copy across a deploy despite the importmap
existing specifically to prevent that.
map.priesol.net sits behind a reverse proxy that only forwards to
maptool itself, so the previous approach (publish kiwix-serve's port,
link the browser straight at it) had no address a real browser could
ever reach — not just a Docker-internal hostname as first assumed, but
no host/port combination at all works from outside.
GET /kiwix/* now proxies to kiwix-serve (Kiwix.proxy in
lib/maptool/kiwix.rb, a plain Net::HTTP passthrough), and the kiwix
service runs with --urlRootLocation=/kiwix so every link/asset it
emits already carries that same prefix and needs no rewriting
(confirmed against libkiwix/internalServer.cpp, which strips this
exact prefix off incoming requests). Drops the port-publishing
approach from the previous commit entirely.
- "Nearby on Wikipedia" linked to KIWIX_URL, a Docker-internal hostname
(http://kiwix:8080) no real browser can resolve. Split it: KIWIX_URL
stays server-side-only for the reachability check, and the link
itself is built from KIWIX_PUBLIC_PORT (kiwix-serve's newly
published host port) plus whatever host the visitor already used to
reach maptool — works regardless of which hostname/IP that is.
- The Save button went white on hover in GTK-themed Chrome: the
browser's native button skin was painting over the custom
background because appearance wasn't reset.
- The map preview marker for a pending note always showed a plain red
pin no matter which icon was picked in the form. NoteSaveForm now
reports icon changes via onIconChange so NotesControl can update the
marker's glyph live.
- Saving a note (or a Measure "Save as note") left the new/grown
.geojson file unchecked and invisible until manually toggled on.
notifyGpxStoreChanged now carries the affected path so
GpxPanelControl auto-shows it.
Bind-mounting data/kiwix/ (the directory) meant a symlink placed
inside it broke: the container has to resolve that symlink itself,
in its own filesystem, where the host path it points to doesn't
exist. Binding the exact data/kiwix/wikipedia.zim file path instead
resolves it on the host at mount time (plain mount(2) semantics), so
the real ZIM can live anywhere on the host without duplicating a
multi-GB file into this repo's data/ directory.
lib/maptool/kiwix.rb: a small reachability check for an external
kiwix-serve instance. GET /api/kiwix reports it (same optional-
integration pattern as offline-tiles/terrain-tiles); the frontend
adds a "Nearby on Wikipedia" link to a point's popup pointing straight
at kiwix-serve's own geo search (GET /search?latitude=&longitude=
&distance=, confirmed from libkiwix's actual source rather than
guessed) — linked to directly, not proxied/embedded, since it already
has a full search UI of its own.
docker-compose.yaml: new `kiwix` service (ghcr.io/kiwix/kiwix-serve)
on the default network, KIWIX_URL wired to reach it by service name.
The ZIM itself isn't part of the repo (large, self-contained archive)
— data/kiwix/ gitignored same as the *.pmtiles files, setup steps in
docs/development.adoc.
No kiwix-serve/ZIM available here to test end-to-end — verified the
app side against a stand-in local HTTP server (GET /api/kiwix flips
available correctly) and confirmed the generated popup link matches
the real geo-search query shape. docker-compose.yaml validated with
`docker-compose config`. Full test suite: 118 runs, 0 failures.
lib/maptool/overpass_pois.rb: queries Overpass for drinking water,
dump stations, LPG refill, toilets, laundromats, supermarkets/
convenience stores, and springs within a bbox, writes GeoJSON. A
dev-time tool (like the offline.pmtiles/terrain.pmtiles generation
steps already documented) — not called by the running app, drop the
output into data/gpx/ or upload it through the Tracks panel like any
other .geojson layer.
Verified against the real Overpass API (88 POIs for central
Bratislava) in addition to the new unit tests (pure to_geojson/
build_query logic, no network in the suite itself). Full suite: 103
runs, 0 failures.
Sidebar toggle (new HillshadeControl) adds a raster-dem hillshade
layer: public Terrarium tiles when online, or a local
data/terrain.pmtiles when present (mirrors offline.pmtiles' optional-
local-file pattern, new /api/terrain-tiles + /tiles/terrain.pmtiles
endpoints). No offline terrain data is bundled, so the toggle is
online-only until one is generated — see docs/development.adoc for
how.
bin/gnss_simulator.rb speaks gpsd's own JSON wire protocol directly
on its default port -- GpsdClient connects to it exactly as it would
to real gpsd, no app changes needed, and no gpsd install/daemon
required either (unlike the existing fake-NMEA-source approach in
docs/development.adoc, which still needs gpsd itself in front of it).
Drives a loop around a center point at a driving-ish pace rather than
sitting still, since a fixed position isn't much of a "GNSS position"
to look at.
Verified end to end: started it, pointed a dev server at it, and
confirmed /api/gnss/position reports a moving fix across two polls.
`bundle exec rake test` (minitest, a default gem but pinned in
Gemfile for reproducibility) now covers what ad-hoc scripts kept
re-verifying by hand during development: Gpx/Fit/TrackStats parsing,
GpxStore/LiveTrackStore file/TTL logic, LiveTrackClient's payload
parsing, and app.rb's routes end to end via Rack::MockRequest.
MAPTOOL_DATA_DIR (new, app.rb) lets the suite point GPX_STORE/
LIVETRACK_STORE at a throwaway directory instead of real data/ —
without it every run would leave uploaded test files behind. Each
lib file's old `if __FILE__ == $0` self-check is now redundant with
real tests covering the same ground, so removed rather than kept as
a second copy of the same assertions. Wired into ci.yml so it isn't
just a local convenience.
Runs renovate/renovate against this Gitea instance on the native
deploy runner, opening PRs for outdated Bundler dependencies instead
of relying on manually noticing (as happened with puma/mustermann
earlier).
Ruby syntax check, a require_relative "app" smoke load, and JS syntax
check on every push/PR. No test suite exists yet, so these are cheap
syntax/load checks rather than behavioral tests.
Restructure the map controls from MapLibre corner buttons into a left
sidebar of icons with flyouts/panels (radar, GPX, GNSS), matching
OsmAnd's layout more closely:
- Base map and GPX selection are now full-height side panels docked
next to the sidebar (not small dropdown flyouts or a right-docked
panel), opening/closing exclusively via a small shared event bus
(panels.js) so only one is ever open at a time.
- The GPX panel's upload/new-folder controls are now icon buttons in
the header instead of an always-visible dropzone and form; dragging
a GPX file onto the tree, a specific folder, or straight onto the
map itself all upload it.
- New: Maptool::GpsdClient (lib/maptool/gpsd_client.rb) streams live
GNSS fixes from a local gpsd daemon over Server-Sent Events
(/api/gnss/stream) to a marker on the map (public/js/gnss.js) — one
background gpsd connection shared by every browser tab.
- The online map style (public/js/osmand-style.js) is now styled to
resemble OsmAnd's own rendering, applied to both the online and
offline (PMTiles) styles.
Also adds a GeolocateControl (browser geolocation) next to the zoom
controls.
Replace Leaflet/raster tiles with MapLibre GL JS: an online OpenFreeMap
style and an offline style built from a local PMTiles archive via
pmtiles.js and protomaps-themes-base (no glyphs/labels offline by
design, see docs/architecture.adoc). Frontend libraries are vendored
locally rather than loaded from a CDN so the app keeps working with no
internet access.
Also add an animated RainViewer precipitation radar overlay (toggle,
play/pause, scrub slider), matching the default time window of the
weather-radar-card Home Assistant integration.
randy