maptool/docs/development.adoc

= Development guide
:toc:
:toclevels: 2

== Web app only

[source,shell]
----
bundle install
bundle exec puma -C config/puma.rb
----

Open http://127.0.0.1:4567. For auto-reload during development, run
via the `rerun` gem instead:

[source,shell]
----
bundle exec rerun -- bundle exec puma -C config/puma.rb
----

TIP: Browsers apply heuristic caching to static files even without
explicit cache headers, so an edit to `public/` (or `views/`) can
appear not to have taken effect on the next reload. Force a real
reload (e.g. Ctrl/Cmd+Shift+R) if that happens.

== Offline map data

To enable the offline style, place a PMTiles file at
`data/offline.pmtiles` (see `docs/architecture.adoc` for how it is
served and styled — the same OpenMapTiles schema/style the online map
uses, just from local data). Logged-in users can generate this from
the app itself — see "In-app regeneration" below — or it can be built
by hand:

* Download one or more country extracts from
  https://download.geofabrik.de[Geofabrik] (`.osm.pbf`, ODbL/OSM
  attribution) — e.g.
  `curl -O https://download.geofabrik.de/europe/slovakia-latest.osm.pbf`.
* If more than one country, merge them with
  https://github.com/osmcode/osmium-tool[osmium-tool]:
  `osmium merge slovakia-latest.osm.pbf austria-latest.osm.pbf -o merged.osm.pbf`
  (each Geofabrik file is already clipped to that country's real
  border, so no separate bbox/polygon clip step is needed either
  before or after — skip this step entirely for a single country).
* Tile it with https://github.com/onthegomap/planetiler[Planetiler]'s
  default profile, which *is* OpenMapTiles — no `--profile` flag
  needed — writing PMTiles directly:
  `java -jar planetiler.jar --osm-path=merged.osm.pbf --output=data/offline.pmtiles`
  (needs Java 21+; `planetiler.jar` from
  https://github.com/onthegomap/planetiler/releases/latest[the latest release]).

Once the file is in place, `GET /api/offline-tiles` reports
`available: true` and the map defaults to the offline style.

== Offline terrain data (hillshade)

The sidebar's "Terrain shading" toggle works online out of the box
(public https://github.com/tilezen/joerd[Terrarium] elevation tiles,
no API key). For it to work offline too, place a PMTiles archive of
Terrarium-encoded `raster-dem` tiles at `data/terrain.pmtiles`
(`GET /api/terrain-tiles` reports `available: true` once it's there,
same pattern as offline map data above). Also generatable in-app (see
below) or by hand, the same `go-pmtiles extract` mechanism as
`data/offline.pmtiles` above — but from
https://mapterhorn.com[Mapterhorn], a Protomaps-affiliated project
publishing pre-built Terrarium-encoded terrain as PMTiles (a global
`planet.pmtiles`, z0-12, plus ~450 regional chunks covering z13 and up
— see https://mapterhorn.com/data-access/[their own data-access page]
for the full manifest and per-source license/attribution details, CC-BY-4.0
and similar, varying by region):

[source,shell]
----
pmtiles extract https://download.mapterhorn.com/planet.pmtiles planet-extract.pmtiles --bbox=west,south,east,north --maxzoom=13 --overfetch=0
# only if a regional chunk covers the bbox too (see download_urls.json) —
# most bboxes hit exactly one, a bbox straddling a chunk boundary hits two:
pmtiles extract https://download.mapterhorn.com/<chunk>.pmtiles chunk-extract.pmtiles --bbox=west,south,east,north --maxzoom=13 --overfetch=0
pmtiles merge planet-extract.pmtiles chunk-extract.pmtiles data/terrain.pmtiles
----

`--overfetch=0`: the default (0.05) merges nearby byte ranges into fewer,
bigger requests to save round-trips — for `planet.pmtiles` (global z0-12,
tens of GB) over a bbox spanning several countries this can collapse the
whole extract into a *single* multi-GB request with no way to resume after
a mid-transfer HTTP/2 stream reset. `--overfetch=0` trades that one
fragile mega-request for many small ones instead (spread over
`--download-threads`, default 4) — slower end to end, but a dropped
connection only loses one small request's worth of progress, not the
whole extract.

`--maxzoom=13`: raster DEM tiles don't compress away like vector ones do
(every tile is a real image, even over flat/empty terrain), so each extra
zoom level roughly quadruples storage — the app's own build
(`PmtilesBuilder::TERRAIN_MAX_ZOOM`) caps at 13, each regional chunk's
own native minzoom, rather than the online hillshade layer's z15. A
12-country selection at z15 was headed well past 181GB; z13 is still
plenty for the hillshade toggle and the planned slope readout.

Tiles are 512px WebP, not the 256px PNG this app's *online* Terrarium
source uses — `public/js/dem.js`'s own point-elevation lookups
(`shadow.js`'s terrain-shadow overlay) read the actual decoded tile
size rather than assuming one, so this isn't a problem either way.

== Offline hiking trails

The sidebar's "Hiking trails" overlay works online out of the box
(waymarkedtrails.org raster tiles, `public/js/hiking_trails.js`) —
clicking a trail fires a one-off Overpass query for its name/colour/
network/distance. For it to work offline too, one `data/hiking_trails/
<CODE>.geojson` per country is generated automatically as the last step
of every `offline.pmtiles` build (`Maptool::HikingTrails`,
`lib/maptool/hiking_trails.rb`), straight from the *same* per-country
`.osm.pbf` files already downloaded for `offline.pmtiles` itself — no
separate download, no third-party service:

[source,shell]
----
osmium tags-filter country.osm.pbf r/route=hiking -o hiking.osm.pbf
osmium add-locations-to-ways hiking.osm.pbf -o hiking-located.osm.pbf
osmium cat hiking-located.osm.pbf -f opl,locations_on_ways=true,add_metadata=false -o hiking.opl
----

`add-locations-to-ways` matters: a plain Geofabrik PBF's ways only carry
bare node *references*, not coordinates, so skipping this step silently
produces ways with empty x/y in the OPL output and every route gets
dropped for having no usable geometry — not an error, just wrong output
(verified against a real extract before this was caught). No general OSM
tool stitches a route relation's member ways into one line for you
(routes are a semantic grouping, not a geometry type), so
`HikingTrails.parse_opl` does that walk itself — the same thing
waymarkedtrails.org and this app's own online Overpass click handler
already do.

Best-effort per country: an extraction problem on one country's PBF logs
a warning and is skipped rather than failing the whole `offline.pmtiles`
build, or even the other countries in the same selection, over what's a
bonus overlay, not the core map (`PmtilesBuilder.extract_hiking_trails`).
Cached the same way `geofabrik_cache` is (skipped once a country's own
extract is already newer than its source `.osm.pbf`), so growing the
region selection by one country only extracts that one. Coordinates are
rounded to 5 decimal places (~1m, plenty for a rendered line) — Slovakia
alone is ~3000 routes, ~30MB unrounded.

One file per country rather than one combined file: `GET /api/hiking-trails`
lists which country codes have an extract (`countries: [...]`), and the
Overlays panel's "Hiking trails" item turns that into a checklist —
each ticked country fetches its own `GET /tiles/hiking_trails/<CODE>.geojson`
only then, not on page load and not for countries never ticked. Offline
click hit-tests the loaded vector layer(s) directly
(`queryRenderedFeatures`) instead of querying Overpass, so it resolves
instantly with no network at all. `rake hiking:trails[pbf,out]` runs the
extraction by hand against one `.osm.pbf` for testing.

`DELETE /api/offline-tiles` removes the whole `data/hiking_trails/`
directory along with `offline.pmtiles` and the Geofabrik cache — there's
no separate delete/rebuild route for it, since it's not an independent
build.

== In-app regeneration

A logged-in user can also generate both files above from the app
itself, no local tooling needed — the sidebar's "Offline data" icon
(Offline hub group) shows a checklist of European countries
(`lib/maptool/data/europe_countries.geojson`, Natural Earth 110m
admin-0, public domain) plus each file's status (size, last updated).
Ticking countries and clicking "Apply changes" saves the selection
(`data/offline_regions.json`, an array of ISO A2 codes) and rebuilds
both files to cover it.

There's no standalone per-country *built* file — `offline.pmtiles`/
`terrain.pmtiles` are always one combined extract for the *whole
current selection*. "Remove a country" is just "change the selection,
then Apply" (unticking a country and applying rebuilds without it).
terrain.pmtiles re-extracts its selection from scratch on
every Apply — fine given that extraction is byte-range requests
against a remote archive, not full downloads. offline.pmtiles, by
contrast, downloads a whole country's `.osm.pbf` from Geofabrik (can be
multi-GB), so it keeps a per-country cache under
`data/geofabrik_cache/` instead of re-fetching on every Apply — a
country already cached is only re-downloaded if Geofabrik's own
`Last-Modified` for it has moved on since (checked with a cheap `HEAD`,
see `PmtilesBuilder.cached_extract`), so growing or shrinking the
selection only touches whatever's actually new to it.

This wraps exactly the manual workflows documented above
(`lib/maptool/pmtiles_builder.rb`). offline.pmtiles downloads each
selected country's Geofabrik extract, `osmium merge`s them if there's
more than one, then tiles the result with Planetiler — the server
needs `osmium` and `java` + `planetiler.jar` installed and on `PATH`
(override with the `OSMIUM_BIN`/`JAVA_BIN`/`PLANETILER_JAR` env vars if
they live somewhere else; the Docker image installs all three, see
`Dockerfile`). terrain.pmtiles still uses `go-pmtiles`
(`PMTILES_BIN`) against Mapterhorn, unrelated to this — it extracts
`planet.pmtiles` (always) plus whichever Mapterhorn regional chunks
intersect the selection, `merge`s them if there was more than one —
always safe regardless of how many/which countries are selected, since
each chunk is Mapterhorn's own pre-partitioned, non-overlapping source
file (extracting the *same* source once per country and merging those,
by contrast, would fail — adjacent countries' region-extracts share
border tiles, since extraction copies whole tiles rather than clipping
vector content to the polygon, and `pmtiles merge` refuses any
overlapping tile id between inputs).

A build runs in the background (`GET /api/offline-regions` to list/see
the current selection, `POST` to change it, then `POST
/api/offline-tiles/build` / `/api/terrain-tiles/build`, polled via the
matching `*/build-status` route) — only one of each kind at a time;
starting a second while one's running gets a 409, as does starting
either with nothing selected yet. While running, `build-status` also
includes a `step` string (e.g. `"fetching PL (3/4) — done: SK, CZ —
pending: HU"`, or `"tiling with planetiler"`) — see
`PmtilesBuilder.run_job`'s `report_step`.

`DELETE /api/offline-tiles` removes offline.pmtiles, its sidecar meta,
and the whole per-country download cache (`data/geofabrik_cache/`) —
`DELETE /api/terrain-tiles` does the same for terrain.pmtiles. Both are
a manual "reclaim the space" / "start over" action (409 while that
kind's build is running), separate from the region picker's Apply,
which only changes what a *future* build covers and never frees
anything already on disk. `DELETE
/api/offline-tiles/cache/:code` removes just one country's cached
extract, e.g. to force a clean re-download of it without touching the
rest of the cache. `GET /api/offline-tiles/cache` lists what's
actually in that cache right now (code/size/Last-Modified per file) —
not the same thing as offline.pmtiles' own `meta.sources`, which only
reflects the last successful build: a country can be cached without
ever having been built (ticked, not yet Applied) or built from without
still being selected (unticked since). The region picker shows this
next to each country's checkbox, with its own delete button, rather
than only offering "delete everything" on the whole offline.pmtiles
row.

Each successful build also writes a small sidecar (`offline.pmtiles.
meta.json` — each source country's code/Geofabrik slug/Last-Modified;
`terrain.pmtiles.meta.json` — the Mapterhorn source names/md5sums)
recording what was actually built, since the `.pmtiles` file itself
carries no such record. `GET /api/offline-tiles`/`/api/terrain-tiles`
include it as `meta`, shown in the panel next to the existing
size/updated line. `GET .../check-update` compares that recorded build
against each source's live upstream state (a Geofabrik `HEAD` per
country for offline, the Mapterhorn manifest for terrain — without
rebuilding anything) and reports `update_available: true/false/null` —
`null` means "don't know" (no local build yet, or a check itself
failed), kept as its own route rather than folded into the plain
status endpoints above since those are on the page-load hot path (the
topbar's offline-status dot) and this one reaches an external service,
only worth doing when the Offline data panel is actually open.

== Camper-relevant POIs (Overpass)

`rake overpass:pois[south,west,north,east,out.geojson]` queries
https://overpass-api.de[Overpass] for drinking water, dump stations,
LPG refill, public toilets, laundromats, supermarkets/convenience
stores, and springs within a bbox, and writes the result as a
`.geojson` file (`lib/maptool/overpass_pois.rb`). Nodes only (nothing
that's mapped as a way/relation, e.g. a supermarket traced as a
building outline rather than a single point). Also reachable from
inside the running app itself, without the command line — see Import
plugins below, which wraps this exact query as `plugins/overpass-pois`.

== Import plugins

The sidebar's import icon (logged in only) lists whatever's under
`plugins/` and runs one for the current map viewport with a click — no
area-picker to fill in, the visible area *is* the area. Each plugin is
a directory with a `plugin.json` manifest (`{"label": "...", "command":
"./run.sh"}`) plus whatever script that command names; the entire
interface a plugin needs to know about maptool is "read
`--south`/`--west`/`--north`/`--east`, print a GeoJSON
`FeatureCollection` to stdout" (`lib/maptool/plugins.rb`) — the result
is saved as a new `.geojson` file the same way an upload would be.

`plugins/overpass-pois` ships as the reference example (a thin wrapper
around the `rake overpass:pois` logic above). A plugin that scrapes a
site whose terms of service forbid that — Park4Night, iOverlander,
Freecampsites — doesn't have to live in this (public) repo to work
here: drop its directory under `plugins/` locally and just don't
`git add` it, same as any other local-only file this app already
treats that way (`data/offline.pmtiles`, the kiwix ZIM, ...).

== Offline Wikipedia (kiwix-serve)

A point popup's "Nearby on Wikipedia" link needs
https://github.com/kiwix/kiwix-tools[kiwix-serve] running with a
Wikipedia https://library.kiwix.org[ZIM] loaded — set `KIWIX_URL` to
wherever *the server* can reach it (`docker-compose.yaml` already runs
a `kiwix` service pointed at `data/kiwix/wikipedia.zim` and wires
`KIWIX_URL` to it via its compose service name; blank/unset disables
the link, same as every other optional integration here). Used only
server-side: production sits behind a reverse proxy that forwards only
to this app, so there's no address a browser could reach kiwix-serve
at directly — `GET /kiwix/*` (`app.rb`) proxies to it instead, and the
`kiwix` service runs with `--urlRootLocation=/kiwix` so its own
emitted links/assets already carry that same prefix. Not part of the
repo — a ZIM is a large, self-contained archive, obtain one locally:

* https://library.kiwix.org/#lang=eng&category=wikipedia[Kiwix's own
  library] has pre-built Wikipedia ZIMs — a `_maxi` variant per
  language/region includes images, `_nopic` is much smaller if
  browsing images offline doesn't matter here.
* Download it to `data/kiwix/wikipedia.zim` (the exact filename
  `docker-compose.yaml`'s `kiwix` service expects — rename after
  downloading, Kiwix's own filenames are much more specific). A symlink
  there works too (`ln -s /wherever/the-real-file.zim
  data/kiwix/wikipedia.zim`), so the real file doesn't have to sit inside
  this repo's checkout at all — the compose file's volume binds that exact
  path, not the `data/kiwix/` directory, specifically so this resolves
  correctly (a symlink *inside* an already-mounted directory wouldn't: the
  container would have to resolve it itself, in its own filesystem, where
  the host path it points to doesn't exist).

Once it's there, `docker compose up -d kiwix` (or a full restart)
picks it up — `GET /api/kiwix` reports `available: true` and the
"Nearby on Wikipedia" link appears, pointed at kiwix-serve's own geo
search (`GET /kiwix/search?pattern=&latitude=&longitude=&distance=`)
through this app's own proxy rather than at kiwix-serve directly — it
already has a full search UI of its own, this just makes it reachable
through the same origin/reverse proxy as the rest of the app.

== GNSS position (gpsd)

The "GNSS" control needs https://gpsd.io[gpsd] running and pointed at
a receiver (`gpsd /dev/ttyUSB0`, or whatever device the receiver shows
up as). On openSUSE, `gpsd` is a package with systemd units
(`gpsd.service` / `gpsd.socket`); on other distributions, check your
package manager. No app-side configuration is needed — it just
connects to `127.0.0.1:2947`, gpsd's default port.

To test without hardware or gpsd itself installed/running, run
`bin/gnss_simulator.rb` — it speaks gpsd's own JSON wire protocol
directly on gpsd's default port (`127.0.0.1:2947`), so
`Maptool::GpsdClient` connects to it exactly as it would to real
gpsd, no app-side changes needed. Drives a loop around a center point
(Bratislava by default) at a driving-ish pace instead of sitting at a
fixed position, since a stationary "GNSS position" isn't much to look
at on the map:

[source,shell]
----
bin/gnss_simulator.rb                       # loop around Bratislava, 300m radius
bin/gnss_simulator.rb 48.85 2.35 500         # custom center/radius (Paris, 500m)
----

`GET /api/gnss/position` should report a fix within a couple of
seconds, moving on every poll.

To test against gpsd itself instead (e.g. to exercise the actual
NMEA-parsing path, not just gpsd's JSON output), feed it a fake NMEA
stream over TCP instead of a real serial device — gpsd accepts a
`tcp://host:port` source:

[source,shell]
----
# terminal 1: a TCP server repeating one NMEA GGA/RMC sentence pair
python3 - <<'PY' 5001
import socket, sys, time

def checksum(s):
    cs = 0
    for ch in s:
        cs ^= ord(ch)
    return f"{cs:02X}"

def sentence(body):
    return f"${body}*{checksum(body)}\r\n"

port = int(sys.argv[1])
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", port))
srv.listen(1)
conn, _ = srv.accept()
gga = "GPGGA,123519,4808.916,N,01706.462,E,1,08,0.9,140.4,M,46.9,M,,"
rmc = "GPRMC,123519,A,4808.916,N,01706.462,E,000.5,084.4,230824,003.1,W"
while True:
    conn.sendall((sentence(gga) + sentence(rmc)).encode())
    time.sleep(1)
PY

# terminal 2: point gpsd at it instead of a real device
gpsd -N -n tcp://127.0.0.1:5001
----

`GET /api/gnss/position` should then report the fake fix (Bratislava
Castle, in the sentences above) within a couple of seconds.

== Tests

`bundle exec rake test` runs the whole suite (`test/test_*.rb`,
Minitest — a default gem, but pinned in the `Gemfile`'s `:test` group
for reproducibility, same reasoning as `rake` itself). `test_app.rb`
drives `app.rb`'s routes end to end through `Rack::MockRequest` rather
than a real server; everything else tests one `lib/maptool/` module in
isolation. `MAPTOOL_DATA_DIR` (see `app.rb`) is pointed at a throwaway
`Dir.mktmpdir` by `test/test_helper.rb` before `app.rb` loads, so
running the suite never touches real `data/` — the app-level tests
also wipe that directory between each test (see `AppTestHelper#teardown`),
since `GPX_STORE`/`LIVETRACK_STORE` are process-wide constants shared
by every test in the run.

`LiveTrackClient`'s own tests use a synthetic fixture (no network) —
its parsing was checked against real Garmin sessions by hand during
development, not by the suite, since that needs a live, unexpired
session token to fetch. `Fit`'s tests likewise build a minimal FIT
file by hand rather than shipping a real device fixture; correctness
against real files (position/altitude fields, definition-message
framing, compressed-timestamp headers) was checked by hand against
`python-fitparse`'s test fixtures during development — see
`docs/architecture.adoc`.

== Continuous integration

`.gitea/workflows/ci.yml` runs on every push/PR against a self-hosted
Gitea Actions runner: `ruby -c` on every `.rb` file, a `require_relative
"app"` smoke check that the app loads without error, `bundle exec rake
test` (see above), and `node --check` on every file under
`public/js/` — no JS test suite, just a syntax check.

`.gitea/workflows/renovate.yml` runs
https://docs.renovatebot.com[Renovate] (the `renovate/renovate` image)
weekly (Mondays, also triggerable manually), opening PRs for outdated
dependencies (Bundler `Gemfile`/`Gemfile.lock`, auto-detected — no
`renovate.json` needed) against this Gitea instance
(`RENOVATE_PLATFORM=gitea`). It runs on the native `deploy` runner (see
`docs/deployment.adoc`) as a one-shot `docker run`, not a persistent
service. Needs a `RENOVATE_TOKEN` repo secret — a Gitea personal
access token (*Settings → Applications → Generate New Token*, with
repository read/write scope) from whichever account should author the
PRs; `RENOVATE_ONBOARDING=false` skips Renovate's usual first-PR
"configure me" step and goes straight to dependency update PRs using
its defaults.