maptool/docs/development.adoc

Development guide

Table of Contents

• Web app only
• Offline map data
• Offline terrain data (hillshade)
• Offline hiking trails
• In-app regeneration
• Camper-relevant POIs (Overpass)
• Import plugins
• Offline Wikipedia (kiwix-serve)
• GNSS position (gpsd)
• Tests
• Continuous integration

Web app only

bundle install
bundle exec puma -C config/puma.rb

Open http://127.0.0.1:4567. For auto-reload during development, run via the rerun gem instead:

bundle exec rerun -- bundle exec puma -C config/puma.rb

Tip	Browsers apply heuristic caching to static files even without explicit cache headers, so an edit to public/ (or views/) can appear not to have taken effect on the next reload. Force a real reload (e.g. Ctrl/Cmd+Shift+R) if that happens.

Offline map data

To enable the offline style, place a PMTiles file at data/offline.pmtiles (see docs/architecture.adoc for how it is served and styled — the same OpenMapTiles schema/style the online map uses, just from local data). Logged-in users can generate this from the app itself — see "In-app regeneration" below — or it can be built by hand:

• Download one or more country extracts from Geofabrik (.osm.pbf, ODbL/OSM attribution) — e.g. curl -O https://download.geofabrik.de/europe/slovakia-latest.osm.pbf.

• If more than one country, merge them with osmium-tool: osmium merge slovakia-latest.osm.pbf austria-latest.osm.pbf -o merged.osm.pbf (each Geofabrik file is already clipped to that country’s real border, so no separate bbox/polygon clip step is needed either before or after — skip this step entirely for a single country).

• Tile it with Planetiler's default profile, which is OpenMapTiles — no --profile flag needed — writing PMTiles directly: java -jar planetiler.jar --osm-path=merged.osm.pbf --output=data/offline.pmtiles (needs Java 21+; planetiler.jar from the latest release).

Once the file is in place, GET /api/offline-tiles reports available: true and the map defaults to the offline style.

Offline terrain data (hillshade)

The sidebar’s "Terrain shading" toggle works online out of the box (public Terrarium elevation tiles, no API key). For it to work offline too, place a PMTiles archive of Terrarium-encoded raster-dem tiles at data/terrain.pmtiles (GET /api/terrain-tiles reports available: true once it’s there, same pattern as offline map data above). Also generatable in-app (see below) or by hand, the same go-pmtiles extract mechanism as data/offline.pmtiles above — but from Mapterhorn, a Protomaps-affiliated project publishing pre-built Terrarium-encoded terrain as PMTiles (a global planet.pmtiles, z0-12, plus ~450 regional chunks covering z13 and up — see their own data-access page for the full manifest and per-source license/attribution details, CC-BY-4.0 and similar, varying by region):

pmtiles extract https://download.mapterhorn.com/planet.pmtiles planet-extract.pmtiles --bbox=west,south,east,north --maxzoom=13 --overfetch=0
# only if a regional chunk covers the bbox too (see download_urls.json) —
# most bboxes hit exactly one, a bbox straddling a chunk boundary hits two:
pmtiles extract https://download.mapterhorn.com/<chunk>.pmtiles chunk-extract.pmtiles --bbox=west,south,east,north --maxzoom=13 --overfetch=0
pmtiles merge planet-extract.pmtiles chunk-extract.pmtiles data/terrain.pmtiles

--overfetch=0: the default (0.05) merges nearby byte ranges into fewer, bigger requests to save round-trips — for planet.pmtiles (global z0-12, tens of GB) over a bbox spanning several countries this can collapse the whole extract into a single multi-GB request with no way to resume after a mid-transfer HTTP/2 stream reset. --overfetch=0 trades that one fragile mega-request for many small ones instead (spread over --download-threads, default 4) — slower end to end, but a dropped connection only loses one small request’s worth of progress, not the whole extract.

--maxzoom=13: raster DEM tiles don’t compress away like vector ones do (every tile is a real image, even over flat/empty terrain), so each extra zoom level roughly quadruples storage — the app’s own build (PmtilesBuilder::TERRAIN_MAX_ZOOM) caps at 13, each regional chunk’s own native minzoom, rather than the online hillshade layer’s z15. A 12-country selection at z15 was headed well past 181GB; z13 is still plenty for the hillshade toggle and the planned slope readout.

Tiles are 512px WebP, not the 256px PNG this app’s online Terrarium source uses — `public/js/dem.js’s own point-elevation lookups (`shadow.js’s terrain-shadow overlay) read the actual decoded tile size rather than assuming one, so this isn’t a problem either way.

Offline hiking trails

The sidebar’s "Hiking trails" overlay works online out of the box (waymarkedtrails.org raster tiles, public/js/hiking_trails.js) — clicking a trail fires a one-off Overpass query for its name/colour/ network/distance. For it to work offline too, one data/hiking_trails/ <CODE>.geojson per country is generated automatically as the last step of every offline.pmtiles build (Maptool::HikingTrails, lib/maptool/hiking_trails.rb), straight from the same per-country .osm.pbf files already downloaded for offline.pmtiles itself — no separate download, no third-party service:

osmium tags-filter country.osm.pbf r/route=hiking -o hiking.osm.pbf
osmium add-locations-to-ways hiking.osm.pbf -o hiking-located.osm.pbf
osmium cat hiking-located.osm.pbf -f opl,locations_on_ways=true,add_metadata=false -o hiking.opl

add-locations-to-ways matters: a plain Geofabrik PBF’s ways only carry bare node references, not coordinates, so skipping this step silently produces ways with empty x/y in the OPL output and every route gets dropped for having no usable geometry — not an error, just wrong output (verified against a real extract before this was caught). No general OSM tool stitches a route relation’s member ways into one line for you (routes are a semantic grouping, not a geometry type), so HikingTrails.parse_opl does that walk itself — the same thing waymarkedtrails.org and this app’s own online Overpass click handler already do.

Best-effort per country: an extraction problem on one country’s PBF logs a warning and is skipped rather than failing the whole offline.pmtiles build, or even the other countries in the same selection, over what’s a bonus overlay, not the core map (PmtilesBuilder.extract_hiking_trails). Cached the same way geofabrik_cache is (skipped once a country’s own extract is already newer than its source .osm.pbf), so growing the region selection by one country only extracts that one. Coordinates are rounded to 5 decimal places (~1m, plenty for a rendered line) — Slovakia alone is ~3000 routes, ~30MB unrounded.

One file per country rather than one combined file: GET /api/hiking-trails lists which country codes have an extract (countries: […​]), and the Overlays panel’s "Hiking trails" item turns that into a checklist — each ticked country fetches its own GET /tiles/hiking_trails/<CODE>.geojson only then, not on page load and not for countries never ticked. Offline click hit-tests the loaded vector layer(s) directly (queryRenderedFeatures) instead of querying Overpass, so it resolves instantly with no network at all. rake hiking:trails[pbf,out] runs the extraction by hand against one .osm.pbf for testing.

DELETE /api/offline-tiles removes the whole data/hiking_trails/ directory along with offline.pmtiles and the Geofabrik cache — there’s no separate delete/rebuild route for it, since it’s not an independent build.

In-app regeneration

A logged-in user can also generate both files above from the app itself, no local tooling needed — the sidebar’s "Offline data" icon (Offline hub group) shows a checklist of European countries (lib/maptool/data/europe_countries.geojson, Natural Earth 110m admin-0, public domain) plus each file’s status (size, last updated). Ticking countries and clicking "Apply changes" saves the selection (data/offline_regions.json, an array of ISO A2 codes) and rebuilds both files to cover it.

There’s no standalone per-country built file — offline.pmtiles/ terrain.pmtiles are always one combined extract for the whole current selection. "Remove a country" is just "change the selection, then Apply" (unticking a country and applying rebuilds without it). terrain.pmtiles re-extracts its selection from scratch on every Apply — fine given that extraction is byte-range requests against a remote archive, not full downloads. offline.pmtiles, by contrast, downloads a whole country’s .osm.pbf from Geofabrik (can be multi-GB), so it keeps a per-country cache under data/geofabrik_cache/ instead of re-fetching on every Apply — a country already cached is only re-downloaded if Geofabrik’s own Last-Modified for it has moved on since (checked with a cheap HEAD, see PmtilesBuilder.cached_extract), so growing or shrinking the selection only touches whatever’s actually new to it.

This wraps exactly the manual workflows documented above (lib/maptool/pmtiles_builder.rb). offline.pmtiles downloads each selected country’s Geofabrik extract, osmium merge`s them if there’s more than one, then tiles the result with Planetiler — the server needs `osmium and java + planetiler.jar installed and on PATH (override with the OSMIUM_BIN/JAVA_BIN/PLANETILER_JAR env vars if they live somewhere else; the Docker image installs all three, see Dockerfile). terrain.pmtiles still uses go-pmtiles (PMTILES_BIN) against Mapterhorn, unrelated to this — it extracts planet.pmtiles (always) plus whichever Mapterhorn regional chunks intersect the selection, merge`s them if there was more than one — always safe regardless of how many/which countries are selected, since each chunk is Mapterhorn’s own pre-partitioned, non-overlapping source file (extracting the same source once per country and merging those, by contrast, would fail — adjacent countries' region-extracts share border tiles, since extraction copies whole tiles rather than clipping vector content to the polygon, and `pmtiles merge refuses any overlapping tile id between inputs).

A build runs in the background (GET /api/offline-regions to list/see the current selection, POST to change it, then POST /api/offline-tiles/build / /api/terrain-tiles/build, polled via the matching */build-status route) — only one of each kind at a time; starting a second while one’s running gets a 409, as does starting either with nothing selected yet. While running, build-status also includes a step string (e.g. "fetching PL (3/4) — done: SK, CZ — pending: HU", or "tiling with planetiler") — see PmtilesBuilder.run_job’s `report_step.

DELETE /api/offline-tiles removes offline.pmtiles, its sidecar meta, and the whole per-country download cache (data/geofabrik_cache/) — DELETE /api/terrain-tiles does the same for terrain.pmtiles. Both are a manual "reclaim the space" / "start over" action (409 while that kind’s build is running), separate from the region picker’s Apply, which only changes what a future build covers and never frees anything already on disk. DELETE /api/offline-tiles/cache/:code removes just one country’s cached extract, e.g. to force a clean re-download of it without touching the rest of the cache. GET /api/offline-tiles/cache lists what’s actually in that cache right now (code/size/Last-Modified per file) — not the same thing as offline.pmtiles' own meta.sources, which only reflects the last successful build: a country can be cached without ever having been built (ticked, not yet Applied) or built from without still being selected (unticked since). The region picker shows this next to each country’s checkbox, with its own delete button, rather than only offering "delete everything" on the whole offline.pmtiles row.

Each successful build also writes a small sidecar (offline.pmtiles. meta.json — each source country’s code/Geofabrik slug/Last-Modified; terrain.pmtiles.meta.json — the Mapterhorn source names/md5sums) recording what was actually built, since the .pmtiles file itself carries no such record. GET /api/offline-tiles//api/terrain-tiles include it as meta, shown in the panel next to the existing size/updated line. GET …​/check-update compares that recorded build against each source’s live upstream state (a Geofabrik HEAD per country for offline, the Mapterhorn manifest for terrain — without rebuilding anything) and reports update_available: true/false/null — null means "don’t know" (no local build yet, or a check itself failed), kept as its own route rather than folded into the plain status endpoints above since those are on the page-load hot path (the topbar’s offline-status dot) and this one reaches an external service, only worth doing when the Offline data panel is actually open.

Camper-relevant POIs (Overpass)

rake oversouth,west,north,east,out.geojson queries Overpass for drinking water, dump stations, LPG refill, public toilets, laundromats, supermarkets/convenience stores, and springs within a bbox, and writes the result as a .geojson file (lib/maptool/overpass_pois.rb). Nodes only (nothing that’s mapped as a way/relation, e.g. a supermarket traced as a building outline rather than a single point). Also reachable from inside the running app itself, without the command line — see Import plugins below, which wraps this exact query as plugins/overpass-pois.

Import plugins

The sidebar’s import icon (logged in only) lists whatever’s under plugins/ and runs one for the current map viewport with a click — no area-picker to fill in, the visible area is the area. Each plugin is a directory with a plugin.json manifest ({"label": "…​", "command": "./run.sh"}) plus whatever script that command names; the entire interface a plugin needs to know about maptool is "read --south/--west/--north/--east, print a GeoJSON FeatureCollection to stdout" (lib/maptool/plugins.rb) — the result is saved as a new .geojson file the same way an upload would be.

plugins/overpass-pois ships as the reference example (a thin wrapper around the rake overpass:pois logic above). A plugin that scrapes a site whose terms of service forbid that — Park4Night, iOverlander, Freecampsites — doesn’t have to live in this (public) repo to work here: drop its directory under plugins/ locally and just don’t git add it, same as any other local-only file this app already treats that way (data/offline.pmtiles, the kiwix ZIM, …​).

Offline Wikipedia (kiwix-serve)

A point popup’s "Nearby on Wikipedia" link needs kiwix-serve running with a Wikipedia ZIM loaded — set KIWIX_URL to wherever the server can reach it (docker-compose.yaml already runs a kiwix service pointed at data/kiwix/wikipedia.zim and wires KIWIX_URL to it via its compose service name; blank/unset disables the link, same as every other optional integration here). Used only server-side: production sits behind a reverse proxy that forwards only to this app, so there’s no address a browser could reach kiwix-serve at directly — GET /kiwix/* (app.rb) proxies to it instead, and the kiwix service runs with --urlRootLocation=/kiwix so its own emitted links/assets already carry that same prefix. Not part of the repo — a ZIM is a large, self-contained archive, obtain one locally:

• Kiwix’s own library has pre-built Wikipedia ZIMs — a _maxi variant per language/region includes images, _nopic is much smaller if browsing images offline doesn’t matter here.

• Download it to data/kiwix/wikipedia.zim (the exact filename docker-compose.yaml’s `kiwix service expects — rename after downloading, Kiwix’s own filenames are much more specific). A symlink there works too (ln -s /wherever/the-real-file.zim data/kiwix/wikipedia.zim), so the real file doesn’t have to sit inside this repo’s checkout at all — the compose file’s volume binds that exact path, not the data/kiwix/ directory, specifically so this resolves correctly (a symlink inside an already-mounted directory wouldn’t: the container would have to resolve it itself, in its own filesystem, where the host path it points to doesn’t exist).

Once it’s there, docker compose up -d kiwix (or a full restart) picks it up — GET /api/kiwix reports available: true and the "Nearby on Wikipedia" link appears, pointed at kiwix-serve’s own geo search (GET /kiwix/search?pattern=&latitude=&longitude=&distance=) through this app’s own proxy rather than at kiwix-serve directly — it already has a full search UI of its own, this just makes it reachable through the same origin/reverse proxy as the rest of the app.

GNSS position (gpsd)

The "GNSS" control needs gpsd running and pointed at a receiver (gpsd /dev/ttyUSB0, or whatever device the receiver shows up as). On openSUSE, gpsd is a package with systemd units (gpsd.service / gpsd.socket); on other distributions, check your package manager. No app-side configuration is needed — it just connects to 127.0.0.1:2947, gpsd’s default port.

To test without hardware or gpsd itself installed/running, run bin/gnss_simulator.rb — it speaks gpsd’s own JSON wire protocol directly on gpsd’s default port (127.0.0.1:2947), so Maptool::GpsdClient connects to it exactly as it would to real gpsd, no app-side changes needed. Drives a loop around a center point (Bratislava by default) at a driving-ish pace instead of sitting at a fixed position, since a stationary "GNSS position" isn’t much to look at on the map:

bin/gnss_simulator.rb                       # loop around Bratislava, 300m radius
bin/gnss_simulator.rb 48.85 2.35 500         # custom center/radius (Paris, 500m)

GET /api/gnss/position should report a fix within a couple of seconds, moving on every poll.

To test against gpsd itself instead (e.g. to exercise the actual NMEA-parsing path, not just gpsd’s JSON output), feed it a fake NMEA stream over TCP instead of a real serial device — gpsd accepts a tcp://host:port source:

# terminal 1: a TCP server repeating one NMEA GGA/RMC sentence pair
python3 - <<'PY' 5001
import socket, sys, time

def checksum(s):
    cs = 0
    for ch in s:
        cs ^= ord(ch)
    return f"{cs:02X}"

def sentence(body):
    return f"${body}*{checksum(body)}\r\n"

port = int(sys.argv[1])
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", port))
srv.listen(1)
conn, _ = srv.accept()
gga = "GPGGA,123519,4808.916,N,01706.462,E,1,08,0.9,140.4,M,46.9,M,,"
rmc = "GPRMC,123519,A,4808.916,N,01706.462,E,000.5,084.4,230824,003.1,W"
while True:
    conn.sendall((sentence(gga) + sentence(rmc)).encode())
    time.sleep(1)
PY

# terminal 2: point gpsd at it instead of a real device
gpsd -N -n tcp://127.0.0.1:5001

GET /api/gnss/position should then report the fake fix (Bratislava Castle, in the sentences above) within a couple of seconds.

Tests

bundle exec rake test runs the whole suite (test/test_*.rb, Minitest — a default gem, but pinned in the Gemfile’s `:test group for reproducibility, same reasoning as rake itself). test_app.rb drives app.rb’s routes end to end through `Rack::MockRequest rather than a real server; everything else tests one lib/maptool/ module in isolation. MAPTOOL_DATA_DIR (see app.rb) is pointed at a throwaway Dir.mktmpdir by test/test_helper.rb before app.rb loads, so running the suite never touches real data/ — the app-level tests also wipe that directory between each test (see AppTestHelper#teardown), since GPX_STORE/LIVETRACK_STORE are process-wide constants shared by every test in the run.

LiveTrackClient’s own tests use a synthetic fixture (no network) — its parsing was checked against real Garmin sessions by hand during development, not by the suite, since that needs a live, unexpired session token to fetch. `Fit’s tests likewise build a minimal FIT file by hand rather than shipping a real device fixture; correctness against real files (position/altitude fields, definition-message framing, compressed-timestamp headers) was checked by hand against `python-fitparse’s test fixtures during development — see `docs/architecture.adoc.

Continuous integration

.gitea/workflows/ci.yml runs on every push/PR against a self-hosted Gitea Actions runner: ruby -c on every .rb file, a require_relative "app" smoke check that the app loads without error, bundle exec rake test (see above), and node --check on every file under public/js/ — no JS test suite, just a syntax check.

.gitea/workflows/renovate.yml runs Renovate (the renovate/renovate image) weekly (Mondays, also triggerable manually), opening PRs for outdated dependencies (Bundler Gemfile/Gemfile.lock, auto-detected — no renovate.json needed) against this Gitea instance (RENOVATE_PLATFORM=gitea). It runs on the native deploy runner (see docs/deployment.adoc) as a one-shot docker run, not a persistent service. Needs a RENOVATE_TOKEN repo secret — a Gitea personal access token (Settings → Applications → Generate New Token, with repository read/write scope) from whichever account should author the PRs; RENOVATE_ONBOARDING=false skips Renovate’s usual first-PR "configure me" step and goes straight to dependency update PRs using its defaults.