Root cause of "planetiler exited with status 1: ... Unsupported format
tmp" (and likely the earlier unexplained "pmtiles exited with status
1"): Planetiler/go-pmtiles pick their output archive format from the
file extension, and the atomic-rename tmp files were named
"offline.pmtiles.tmp"/"terrain.pmtiles.tmp" — ".tmp" as the extension,
not ".pmtiles". Renamed to "offline.tmp.pmtiles"/"terrain.tmp.pmtiles".
Also, since build-status only ever said "running since X" with no way
to tell what it was actually doing:
- GET .../build-status now includes a `step` string ("fetching PL
(3/4) — done: SK, CZ — pending: HU", "tiling with planetiler", ...)
while a build is in progress.
- DELETE /api/offline-tiles / /api/terrain-tiles clear a built file,
its sidecar meta, and (for offline) the whole per-country download
cache — a manual "reclaim space" / "start over" action the region
picker's Apply never did on its own. DELETE
/api/offline-tiles/cache/:code clears just one country's cached
extract. All three are 409 while that kind's build is running, and
surfaced as a small "Delete" button on each row in the Offline data
panel.
Offline tiles were built from Protomaps' own basemap schema, needing a
completely separate hand-tuned theme (protomaps-themes-base + OFFLINE_THEME)
from the online Liberty style's ONLINE_STYLE_PATCH — the two could never be
made to match exactly no matter how much either side got tuned (today's
session found and fixed several mismatches: building opacity, park
boundaries, road casings, road widths).
Offline.pmtiles is now built with Planetiler's default OpenMapTiles profile
against Geofabrik per-country OSM extracts (osmium-merged when more than one
country is selected), the same vector schema Liberty already expects.
offlineStyle() now clones the same Liberty JSON (vendored as a local
snapshot for reliability with no internet) and applies the identical
ONLINE_STYLE_PATCH, just repointed at local PMTiles/glyphs/sprite — one
palette instead of two, correct on both sides by construction.
Also vendors Liberty's icon sprite for the first time (offline had none
before), adds Java/osmium-tool/planetiler.jar to the Docker image, and
removes the now-unused protomaps-themes-base dependency and the
theme-comparison debug page that found today's bugs.
The .pmtiles files themselves carry no record of what upstream
snapshot they came from — only a local mtime. Each successful build
now writes a small sidecar (offline.pmtiles.meta.json: the Protomaps
build date used; terrain.pmtiles.meta.json: the Mapterhorn source
names/md5sums) recording that, surfaced in the panel next to the
existing size/updated line.
New GET .../check-update routes compare the recorded build against
the live upstream index/manifest — true/false/null, null meaning
"don't know" (no local build yet, or the check itself failed) rather
than a false "no". Kept separate from the plain status endpoints
(which map.js's own topbar dot hits on every page load) since this
one reaches an external service; only called once when the Offline
data panel is actually opened.
Also: flag emoji next to each country in the picker (derived from the
ISO code via Unicode regional indicators, no image assets needed).
Replaces "regenerate for the current viewport" with what Locus/OsmAnd-
style apps give you: tick which countries you want offline (a Natural
Earth-derived checklist, public domain, vendored), Apply rebuilds
offline.pmtiles/terrain.pmtiles to cover the whole selection at once.
Multiple regions can be offline simultaneously (e.g. Slovakia + Austria
for a trip that crosses the border) — the selection itself is the
source of truth (data/offline_regions.json), not a set of per-country
files, so "remove a country" and "refresh a country" are both just
"change the selection, then Apply."
Confirmed against go-pmtiles' own source (not just its docs) before
committing to this shape: `extract --region=<geojson>` treats a
FeatureCollection's features as their union already (no dissolve/union
math needed on our side — pmtiles/region.go's UnmarshalRegion just
concatenates every feature's rings), and `merge` hard-errors on any
overlapping tile id between inputs (merge.go's prepareInputs). That
second fact rules out "extract each country separately, merge them
all" — adjacent countries' region-extracts would share border tiles,
since extraction copies whole tiles rather than clipping vector
content to the polygon. The actual fix: extract per *source archive*
(the Protomaps build, each intersecting Mapterhorn chunk), each call
covering the whole current selection via one --region= file — the
terrain pipeline already extracted-then-merged per source this way, it
just needed a multi-country region instead of a single bbox.
A logged-in user can now trigger, from a new "Offline data" sidebar
panel, what was previously a fully manual go-pmtiles workflow
(docs/development.adoc): regenerate either file for the current map
viewport, no area-picker (same "visible area is the area" convention
the Overpass POI plugin already uses).
- lib/maptool/pmtiles_builder.rb: two independent background jobs
(Thread + Mutex-guarded status Hash, same shape as app.rb's own GNSS
poller), each writing to a .tmp path and atomically renaming into
place so a build never serves a partially-written file mid-run.
offline.pmtiles is a single `pmtiles extract` against Protomaps'
public daily basemap build (byte-range extraction, not a full
~120GB download). terrain.pmtiles has no equivalent public
Terrarium-encoded archive to extract from, so it fetches individual
Terrarium tiles itself, packs them into an intermediate MBTiles
(new sqlite3 gem), then the same `pmtiles convert`.
- 4 new app.rb routes (POST .../build, GET .../build-status) plus
size/mtime added to the existing availability endpoints.
- public/js/offline_data.js: new sidebar panel, polling build-status
the same self-rescheduling-setTimeout shape livetrack.js already
uses for its own refresh loop.
- Both network-heavy pieces (the Protomaps build index, per-tile
fetches) are injectable, so test/test_pmtiles_builder.rb exercises
the actual build pipeline against a fake `pmtiles` binary without
hitting a real network, matching this suite's existing
keep-it-offline convention for external services.
The old link only ever surfaced one nearby article at a time, buried
in whatever POI's popup you happened to open, and only for uploaded
files with a category. Replaced it with a standalone "Wikipedia"
toggle under the renamed "Tracks & Points" panel: enabling it adds a
live map layer that refetches GET /api/kiwix/nearby on every pan/zoom
and plots every nearby article as its own point.
kiwix-serve's own geo search never echoes back a result's coordinates
(confirmed live against the real production ZIM, both its HTML and
?format=xml output) — it only filters by distance from a point. So
Maptool::Kiwix.nearby fetches each matching article's own page too and
reads its <meta name="geo.position"> tag, which every geotagged
Wikipedia article carries. Tried parallelizing those per-article
fetches with threads first; measured no improvement against the real
ZIM (kiwix-serve appears to serialize requests regardless), so it
stayed sequential — ~1.6s for a 30-result page, fine for a
refetch-on-pan-end layer.
Clicking a point opens a small popup linking to the article (proxied
through the existing /kiwix/* passthrough). New WIKI_ICON (NPS's
bookstore-22.svg) marks these, kept out of POI_ICONS since that set
doubles as notes.js's icon-picker option list and this fixed-glyph
layer has no per-feature category to key off of.
plugins/ ships baked into the image and gets wiped/replaced by the
checkout sync on every deploy (docs/deployment.adoc), so a private
plugin dropped there — the exact use case the plugin contract was
designed for — didn't survive a redeploy the way uploaded GPX/kiwix
data under data/ does.
Plugins.list/run now accept either one directory or an array, and
app.rb checks the built-in plugins/ plus data/plugins/ (already
covered by the existing ./data bind mount, no new volume needed).
Overpass categories now carry the same icon+color styling a p4n2gpx
waypoint gets, instead of falling back to gpx.js's plain red marker,
plus a popup description built from opening_hours/operator/website
tags. Supermarket/convenience store reuse NPS's generic store glyph,
the only honest match in that icon set.
Each plugin run's saved filename now bakes in the area's center and a
timestamp, so repeated downloads for different areas land as separate,
identifiable files instead of overpass-pois.geojson/-1/-2/...
A plugin is a directory under plugins/ with a plugin.json manifest
({"label": "...", "command": "./run.sh"}) plus a script that reads
--south/--west/--north/--east and prints a GeoJSON FeatureCollection
to stdout (lib/maptool/plugins.rb) — that's the entire interface a
plugin needs to know about this app. The sidebar's new import icon
(logged in only, GET /api/plugins + POST /api/plugins/:name/run) lists
whatever's found and runs one for the current map viewport, saving its
output as a new .geojson file the same way an upload would be.
The point: a scraper for a site whose ToS forbids that (Park4Night,
iOverlander, Freecampsites) doesn't need to live in this public repo
to work here — its own directory just goes under plugins/ locally and
is never git add'ed, no special-casing needed anywhere.
plugins/overpass-pois wraps the existing rake overpass:pois query as
the first, in-repo, reference example of the contract.
Also fills in four modules (auth.js, locate.js, measure.js, notes.js)
missing from layout.erb's cache-busting importmap alongside adding
plugins.js's own entry — a pre-existing gap that meant they could keep
serving a stale cached copy across a deploy despite the importmap
existing specifically to prevent that.
map.priesol.net sits behind a reverse proxy that only forwards to
maptool itself, so the previous approach (publish kiwix-serve's port,
link the browser straight at it) had no address a real browser could
ever reach — not just a Docker-internal hostname as first assumed, but
no host/port combination at all works from outside.
GET /kiwix/* now proxies to kiwix-serve (Kiwix.proxy in
lib/maptool/kiwix.rb, a plain Net::HTTP passthrough), and the kiwix
service runs with --urlRootLocation=/kiwix so every link/asset it
emits already carries that same prefix and needs no rewriting
(confirmed against libkiwix/internalServer.cpp, which strips this
exact prefix off incoming requests). Drops the port-publishing
approach from the previous commit entirely.
- "Nearby on Wikipedia" linked to KIWIX_URL, a Docker-internal hostname
(http://kiwix:8080) no real browser can resolve. Split it: KIWIX_URL
stays server-side-only for the reachability check, and the link
itself is built from KIWIX_PUBLIC_PORT (kiwix-serve's newly
published host port) plus whatever host the visitor already used to
reach maptool — works regardless of which hostname/IP that is.
- The Save button went white on hover in GTK-themed Chrome: the
browser's native button skin was painting over the custom
background because appearance wasn't reset.
- The map preview marker for a pending note always showed a plain red
pin no matter which icon was picked in the form. NoteSaveForm now
reports icon changes via onIconChange so NotesControl can update the
marker's glyph live.
- Saving a note (or a Measure "Save as note") left the new/grown
.geojson file unchecked and invisible until manually toggled on.
notifyGpxStoreChanged now carries the affected path so
GpxPanelControl auto-shows it.
lib/maptool/kiwix.rb: a small reachability check for an external
kiwix-serve instance. GET /api/kiwix reports it (same optional-
integration pattern as offline-tiles/terrain-tiles); the frontend
adds a "Nearby on Wikipedia" link to a point's popup pointing straight
at kiwix-serve's own geo search (GET /search?latitude=&longitude=
&distance=, confirmed from libkiwix's actual source rather than
guessed) — linked to directly, not proxied/embedded, since it already
has a full search UI of its own.
docker-compose.yaml: new `kiwix` service (ghcr.io/kiwix/kiwix-serve)
on the default network, KIWIX_URL wired to reach it by service name.
The ZIM itself isn't part of the repo (large, self-contained archive)
— data/kiwix/ gitignored same as the *.pmtiles files, setup steps in
docs/development.adoc.
No kiwix-serve/ZIM available here to test end-to-end — verified the
app side against a stand-in local HTTP server (GET /api/kiwix flips
available correctly) and confirmed the generated popup link matches
the real geo-search query shape. docker-compose.yaml validated with
`docker-compose config`. Full test suite: 118 runs, 0 failures.
Measure already draws distance lines and area polygons by hand — a
second click-to-draw UI in the notes tool (Point/Area mode, from the
previous commit) duplicated that. Replaced it with a "Save as note"
button on the Measure panel that saves whatever shape is currently
drawn there; NotesControl goes back to being just a one-click point
pin. Both share a single NoteSaveForm (notes.js) for the note-text +
target-layer-file UI instead of two copies of it.
gpx.js: the "lines" source now also renders a note-flagged LineString
(dashed, popup-on-click) alongside the Polygon fill/outline from
before — the plain track-line layer excludes category:"Note" features
so a saved distance note doesn't also get the track hover-summary/
profile-panel treatment.
Notes are now individually editable/deletable from their own popup
(logged-in only): Edit swaps the popup content for an inline
textarea, Delete confirms and removes. Both target the feature by a
new client-generated properties.id rather than array position (new
GpxStore#update_feature/#delete_feature, PUT/DELETE /api/gpx/notes),
since neither array index nor the note's own text (the thing being
edited) is a stable identifier.
Verified live: Measure's Save button (disabled/enabled state, saves
the exact on-screen shape), the popup's Edit/Delete buttons and inline
edit form. Full test suite: 115 runs, 0 failures (new coverage for
update_feature/delete_feature and the PUT/DELETE routes).
New map-corner "Add note" control (public/js/notes.js), mirroring
measure.js's click-to-place UX. Click the map, write a short note,
pick an existing .geojson layer file or name a new one — saved via a
new GpxStore#append_feature (grows the target file in place, unlike
save()'s always-disambiguate-the-filename behavior) through a new
POST /api/gpx/notes route.
No new rendering path: a note is just another point in a .geojson
file the Tracks panel's tree already lists, checkboxes, and pops up —
visibility toggling ("layers") comes for free from that existing
machinery, per-file, with no changes needed there.
Points only for now — a polygon mode would need gpx.js's point/line
split taught a third Polygon bucket first, or a saved polygon
wouldn't render back on reload (see TODO.adoc).
Covered by new tests: GpxStore#append_feature (create/append/missing-
path/filename-safety) and the /api/gpx/notes route (auth gate, create,
append, malformed/missing-geometry rejection, 404 on a missing
target). Full suite: 97 runs, 0 failures.
Sidebar toggle (new HillshadeControl) adds a raster-dem hillshade
layer: public Terrarium tiles when online, or a local
data/terrain.pmtiles when present (mirrors offline.pmtiles' optional-
local-file pattern, new /api/terrain-tiles + /tiles/terrain.pmtiles
endpoints). No offline terrain data is bundled, so the toggle is
online-only until one is generated — see docs/development.adoc for
how.
/api/health now reports REVISION alongside status, and the CI wait
step checks for this deploy's own short SHA specifically - a stray
old container that never got torn down (or a build that silently
kept serving stale code) would still 200 on a plain health check but
now won't match.
Ended sessions past the ~24h Garmin appears to keep a live-track page
around for get verified once at app startup and, once confirmed dead,
flagged so the panel greys them out instead of only failing after a
click. Already-flagged sessions are skipped on later startups so this
doesn't keep re-hitting Garmin for the same dead session.
GET /api/gnss/position now requires login (app.rb) -- it reveals
where the receiver physically is, not something to leave open to
anyone with the URL. GnssControl's sidebar icon is hidden entirely
when logged out (same pattern as the tree UI's write controls) rather
than shown and failing to ever get a fix.
LocateControl needs no changes: it only ever reaches GNSS through
GnssControl#isEnabled(), which can't turn true for a logged-out
visitor if the toggle is never on the page to click. Its
browser-geolocation fallback stays open to everyone -- a visitor's
own device asking for their own location isn't something to gate.
Extracted isLoggedIn() out of gpx.js into a shared public/js/auth.js
(gnss.js needed the same check) rather than duplicating it.
Verified: /api/gnss/position returns 401 logged out; the sidebar icon
is actually hidden (computed display: none, not just present-but-
unstyled -- checked properly this time after almost repeating the
same present-vs-visible test mistake the .gpx-panel__action bug
already taught).
ASSET_VERSION only proves *a* deploy happened recently, not *which*
commit -- not enough to answer "did my push actually deploy, or am I
looking at a stale browser cache" (today's actual answer: neither --
deploy was current, a GeoJSON cache predating a code change was just
never invalidated, since it only checks the source .gpx's mtime).
The CI deploy job now writes REVISION (short SHA) before dropping
.git; app.rb reads it at boot, falling back to "dev" locally/in CI's
own check job where the file never exists.
sinatra/json defaults its :json_encoder to MultiJson and calls
.encode on it, which newer multi_json versions deprecate in favor of
a constant this app has no other use for (sinatra-contrib 4.2.1, the
current release, still calls the old one -- not something fixable by
bumping a version). Every use of the json helper here is a plain
hash/array, so pointing :json_encoder at stdlib JSON instead sidesteps
multi_json entirely.
?download=... on GET /api/gpx/file now requires login, same as
upload/move/delete/create-folder -- plain viewing (no ?download)
stays public. Added ?download=original alongside the existing
GeoJSON export, serving the untouched, as-uploaded file directly via
send_file -- the file name link still downloads GeoJSON, a new
button next to delete downloads the original.
Since both are real <a href> links (so right-click/middle-click still
work), a logged-out click can't just be left as a plain navigation --
it would land on a raw JSON error page instead of being sent to log
in. _download (gpx.js) fetches instead, reuses the existing
redirectIfUnauthorized 401 check, and only then turns the response
into a blob and clicks a throwaway <a download> to trigger the save.
Verified in a real browser: an anonymous click on either download
correctly redirects through /login to the real OAuth login page
instead of showing raw JSON.
Once a .gpx/.fit is cached, the tree shows it under its converted name
("ride.gpx" -> "ride.gpx.geojson", matching the real cache file next
to it on disk) instead of still reading as un-converted -- the
GeoJSON is what's actually on the map from that point on, and the
original is never deleted, just not re-shown as itself. The name is
now a real link (GET /api/gpx/file/<path>?download=1) instead of
plain text, the only way to get that converted form out of the app.
Verified in a real browser that an <a> nested in a <label> claims its
own click rather than the label forwarding it to the checkbox, so
downloading and toggling visibility don't fight over the same click.
A .geojson file needs no GPX/FIT conversion — it's already the shape
gpx.js expects — so GpxStore#geojson now just reads and returns it
as-is, no cache sidecar. save/tree/the upload dropzone and its zip
handling all pick it up through the existing TRACK_EXTENSIONS list.
Also fixes a bug the new "cached" tree icon would have shipped with:
a GPX/FIT's own .geojson cache sidecar (e.g. "ride.gpx.geojson") also
matches the new .geojson branch in list_children, which would have
listed it as a second, phantom tree entry — excluded explicitly, with
a regression test.
A several-thousand-waypoint park4night area export re-parses the
whole file with REXML on every toggle — ~2.4s for a 5k-waypoint/3.6MB
sample, so tens of seconds for a 50k-waypoint one, likely the actual
cause behind a "Could not load" report on a 41.5MB export. MapLibre
already clusters the waypoint layer client-side, so the bottleneck was
never rendering — cache the GeoJSON conversion alongside the source
file (invalidated by mtime) instead of a bigger bbox/tiling rework.
`bundle exec rake test` (minitest, a default gem but pinned in
Gemfile for reproducibility) now covers what ad-hoc scripts kept
re-verifying by hand during development: Gpx/Fit/TrackStats parsing,
GpxStore/LiveTrackStore file/TTL logic, LiveTrackClient's payload
parsing, and app.rb's routes end to end via Rack::MockRequest.
MAPTOOL_DATA_DIR (new, app.rb) lets the suite point GPX_STORE/
LIVETRACK_STORE at a throwaway directory instead of real data/ —
without it every run would leave uploaded test files behind. Each
lib file's old `if __FILE__ == $0` self-check is now redundant with
real tests covering the same ground, so removed rather than kept as
a second copy of the same assertions. Wired into ci.yml so it isn't
just a local convenience.
Maptool::Fit reads Garmin's binary FIT format directly rather than
adding a gem — the one Ruby option depends on activesupport, unmaintained
since 2022, for a single-purpose binary parser. Only "record" messages
are read (position + altitude; everything else is skipped by byte
length, never interpreted); field numbers and scale/offset are
Garmin's own (fit-python-sdk's profile.py), and the framing plus all
extracted values were checked byte-for-byte against 5 real device
files from python-fitparse's test suite before this shipped.
A .zip upload (Garmin's bulk/FIT export format) is expanded into its
.gpx/.fit entries via rubyzip — that one's an actual dependency, since
ZIP's own format isn't the kind of narrow, single-purpose parsing FIT
above is.
The track endpoint now merges the session summary into the marker's
GeoJSON properties (already computed for storage, just wasn't passed
through), and the marker gets the same hover-popup treatment GPX
points already have.
Fetching a session's track now also grabs its summary (name, activity
type, distance, duration) from the same page and stores it — sessions
the panel has actually shown replace their raw UUID with that summary,
and are kept indefinitely regardless of age. A session nobody ever
opened (no summary) is more likely test/dead data than something worth
keeping, so it's dropped automatically after 48h.
Clicking a LiveTrack session now fetches and draws its track (line +
latest position). Garmin's own API 403s any plain server-side request
(Cloudflare bot protection), so LiveTrackClient fetches the session
page instead and pulls the track out of the server-rendered data
embedded in it. The webhook now captures the session's token too,
since the id alone isn't enough to fetch anything.
Browsers (and a caching proxy in front of production) were holding
onto /js and /css for hours, so a deploy could go unnoticed by an
already-open tab. A boot-time version query string on the entry
script/stylesheet, plus an import map remapping map.js's internal
module imports to the same versioned URLs, fixes that without editing
every JS file's import statements.
A Gmail-watching Apps Script (documented in docs/deployment.adoc) posts
the session ID from each LiveTrack email to a new webhook, which stores
it in data/livetrack.json; a new sidebar panel lists what's been
received.
MAPTOOL_ALLOWED_GITEA_USER now accepts a comma-separated list of
Gitea usernames instead of just one, for when more than one person
needs upload/delete/move access.
Upload/delete/move/create-folder now require logging in via Gitea
OAuth2 (only MAPTOOL_ALLOWED_GITEA_USER is granted write access);
reading stays public. Deploy secrets (session secret, OAuth2 client
credentials, allowed user) are written to .env from Gitea Actions
repo secrets on every deploy instead of living on the server by hand.
Also switch the container's bind address to "[::]" (dual-stack) so
the app is reachable over IPv6, not just IPv4.
Restructure the map controls from MapLibre corner buttons into a left
sidebar of icons with flyouts/panels (radar, GPX, GNSS), matching
OsmAnd's layout more closely:
- Base map and GPX selection are now full-height side panels docked
next to the sidebar (not small dropdown flyouts or a right-docked
panel), opening/closing exclusively via a small shared event bus
(panels.js) so only one is ever open at a time.
- The GPX panel's upload/new-folder controls are now icon buttons in
the header instead of an always-visible dropzone and form; dragging
a GPX file onto the tree, a specific folder, or straight onto the
map itself all upload it.
- New: Maptool::GpsdClient (lib/maptool/gpsd_client.rb) streams live
GNSS fixes from a local gpsd daemon over Server-Sent Events
(/api/gnss/stream) to a marker on the map (public/js/gnss.js) — one
background gpsd connection shared by every browser tab.
- The online map style (public/js/osmand-style.js) is now styled to
resemble OsmAnd's own rendering, applied to both the online and
offline (PMTiles) styles.
Also adds a GeolocateControl (browser geolocation) next to the zoom
controls.
Uploading is now drag & drop (or click to browse) instead of a form;
folders can be created any time, not just during upload; files can be
dragged onto a folder to move them, or dropped elsewhere in the tree
to move back to the top level; files and folders can be deleted. Both
drag sources (an OS file drag and dragging an existing tree entry)
land on the same drop targets, told apart by a custom drag payload
type versus the browser's own "Files" type.
Also sort folders before files in the tree (previously interleaved
alphabetically).
Upload GPX files into a folder hierarchy (traversal-safe path
resolution against the store root), browse them in a tree panel, and
toggle each one's waypoints/tracks on the map. Waypoints render as
clickable points showing name, description and time; bare URLs in the
description are turned into safe links. GPX layers survive switching
between the Online/Offline base map, same as the radar overlay.
Adds rexml as an explicit dependency — it moved to a bundled gem in
this Ruby version and isn't picked up by Bundler otherwise.
Replace Leaflet/raster tiles with MapLibre GL JS: an online OpenFreeMap
style and an offline style built from a local PMTiles archive via
pmtiles.js and protomaps-themes-base (no glyphs/labels offline by
design, see docs/architecture.adoc). Frontend libraries are vendored
locally rather than loaded from a CDN so the app keeps working with no
internet access.
Also add an animated RainViewer precipitation radar overlay (toggle,
play/pause, scrub slider), matching the default time window of the
weather-radar-card Home Assistant integration.
randy