maptool/docs/development.adoc

= Development guide
:toc:
:toclevels: 2

== Web app only

[source,shell]
----
bundle install
bundle exec puma -C config/puma.rb
----

Open http://127.0.0.1:4567. For auto-reload during development, run
via the `rerun` gem instead:

[source,shell]
----
bundle exec rerun -- bundle exec puma -C config/puma.rb
----

TIP: Browsers apply heuristic caching to static files even without
explicit cache headers, so an edit to `public/` (or `views/`) can
appear not to have taken effect on the next reload. Force a real
reload (e.g. Ctrl/Cmd+Shift+R) if that happens.

`/theme_preview.html` is a standalone debug page (not linked from the
app itself) for comparing offline base-map themes side by side: a bare
`protomaps-themes-base` theme, an iframe of the real app's offline
style, and one of its online style, with pan/zoom kept in sync across
all three via the same URL-hash mechanism `map.js` already uses for
shareable links. It's a plain static file, so nothing gates it to
development — it's reachable on a production instance too, same as any
other file under `public/`. Harmless if it stays that way (it shows
nothing `/` doesn't already), but worth knowing if that's ever not the
case.

== Offline map data

To enable the offline style, place a PMTiles file at
`data/offline.pmtiles` (see `docs/architecture.adoc` for how it is
served and styled). Logged-in users can generate this from the app
itself — see "In-app regeneration" below — or it can be built by hand:

* Extract a region from Protomaps' own daily full-planet basemap build
  (https://build-metadata.protomaps.dev/builds.json[build index],
  files at `https://build.protomaps.com/<key>`, ~120GB, ODbL/OSM
  attribution) with the
  https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI's `extract`
  subcommand, which pulls only the byte ranges covering a bbox over
  HTTP range requests rather than the whole file:
  `pmtiles extract https://build.protomaps.com/<key> data/offline.pmtiles --bbox=west,south,east,north --maxzoom=15`.
  (The old `app.protomaps.com` extract-builder UI this section used to
  point at has since redirected elsewhere — this is the same
  underlying mechanism, just the CLI instead of a web UI.)
* To convert some other vector tile source (e.g. an `.mbtiles` file)
  into PMTiles instead, use the same CLI's `convert` subcommand:
  `pmtiles convert in.mbtiles data/offline.pmtiles`. Its schema is
  what `protomaps-themes-base` (the styling library used for the
  offline style) expects.

Once the file is in place, `GET /api/offline-tiles` reports
`available: true` and the map defaults to the offline style.

== Offline terrain data (hillshade)

The sidebar's "Terrain shading" toggle works online out of the box
(public https://github.com/tilezen/joerd[Terrarium] elevation tiles,
no API key). For it to work offline too, place a PMTiles archive of
Terrarium-encoded `raster-dem` tiles at `data/terrain.pmtiles`
(`GET /api/terrain-tiles` reports `available: true` once it's there,
same pattern as offline map data above). Also generatable in-app (see
below) or by hand, the same `go-pmtiles extract` mechanism as
`data/offline.pmtiles` above — but from
https://mapterhorn.com[Mapterhorn], a Protomaps-affiliated project
publishing pre-built Terrarium-encoded terrain as PMTiles (a global
`planet.pmtiles`, z0-12, plus ~450 regional chunks covering z13 and up
— see https://mapterhorn.com/data-access/[their own data-access page]
for the full manifest and per-source license/attribution details, CC-BY-4.0
and similar, varying by region):

[source,shell]
----
pmtiles extract https://download.mapterhorn.com/planet.pmtiles planet-extract.pmtiles --bbox=west,south,east,north --maxzoom=15
# only if a regional chunk covers the bbox too (see download_urls.json) —
# most bboxes hit exactly one, a bbox straddling a chunk boundary hits two:
pmtiles extract https://download.mapterhorn.com/<chunk>.pmtiles chunk-extract.pmtiles --bbox=west,south,east,north --maxzoom=15
pmtiles merge planet-extract.pmtiles chunk-extract.pmtiles data/terrain.pmtiles
----

Tiles are 512px WebP, not the 256px PNG this app's *online* Terrarium
source uses — `public/js/dem.js`'s own point-elevation lookups
(`shadow.js`'s terrain-shadow overlay) read the actual decoded tile
size rather than assuming one, so this isn't a problem either way.

== In-app regeneration

A logged-in user can also generate both files above from the app
itself, no local tooling needed — the sidebar's "Offline data" icon
(Offline hub group) shows a checklist of European countries
(`lib/maptool/data/europe_countries.geojson`, Natural Earth 110m
admin-0, public domain) plus each file's status (size, last updated).
Ticking countries and clicking "Apply changes" saves the selection
(`data/offline_regions.json`, an array of ISO A2 codes) and rebuilds
both files to cover it.

There's no standalone per-country file to individually delete —
`offline.pmtiles`/`terrain.pmtiles` are always one combined extract for
the *whole current selection*. "Remove a country" and "refresh a
country" are both just "change the selection, then Apply" (unticking a
country and applying rebuilds without it; nothing to separately
delete). Every Apply re-extracts the entire selection from scratch, not
just what changed — fine given extraction is fast (byte-range requests
against remote archives, not per-tile fetches), but worth revisiting if
someone selects half of Europe at once and finds it slow.

This wraps exactly the manual `go-pmtiles` workflows documented above
(`lib/maptool/pmtiles_builder.rb`) — the server needs the
https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI installed and
on `PATH` (override with the `PMTILES_BIN` env var if it lives
somewhere else), using `--region=<geojson>` (a `FeatureCollection` of
the selected countries — go-pmtiles treats multiple features as their
union already, no union/dissolve math needed on this side) rather than
`--bbox=` for precise, per-country extraction. terrain.pmtiles extracts
`planet.pmtiles` (always) plus whichever Mapterhorn regional chunks
intersect the selection, `merge`s them if there was more than one —
always safe regardless of how many/which countries are selected, since
each chunk is Mapterhorn's own pre-partitioned, non-overlapping source
file (extracting the *same* source once per country and merging those,
by contrast, would fail — adjacent countries' region-extracts share
border tiles, since extraction copies whole tiles rather than clipping
vector content to the polygon, and `pmtiles merge` refuses any
overlapping tile id between inputs).

A build runs in the background (`GET /api/offline-regions` to list/see
the current selection, `POST` to change it, then `POST
/api/offline-tiles/build` / `/api/terrain-tiles/build`, polled via the
matching `*/build-status` route) — only one of each kind at a time;
starting a second while one's running gets a 409, as does starting
either with nothing selected yet.

Each successful build also writes a small sidecar (`offline.pmtiles.
meta.json` / `terrain.pmtiles.meta.json` — the Protomaps build date used,
or the Mapterhorn source names/md5sums) recording what was actually
built, since the `.pmtiles` file itself carries no such record. `GET
/api/offline-tiles`/`/api/terrain-tiles` include it as `meta`, shown in
the panel next to the existing size/updated line. `GET .../check-update`
compares that recorded build against the live upstream index/manifest
(without rebuilding anything) and reports `update_available: true/false/
null` — `null` means "don't know" (no local build yet, or the upstream
check itself failed), kept as its own route rather than folded into the
plain status endpoints above since those are on the page-load hot path
(the topbar's offline-status dot) and this one reaches an external
service, only worth doing when the Offline data panel is actually open.

== Camper-relevant POIs (Overpass)

`rake overpass:pois[south,west,north,east,out.geojson]` queries
https://overpass-api.de[Overpass] for drinking water, dump stations,
LPG refill, public toilets, laundromats, supermarkets/convenience
stores, and springs within a bbox, and writes the result as a
`.geojson` file (`lib/maptool/overpass_pois.rb`). Nodes only (nothing
that's mapped as a way/relation, e.g. a supermarket traced as a
building outline rather than a single point). Also reachable from
inside the running app itself, without the command line — see Import
plugins below, which wraps this exact query as `plugins/overpass-pois`.

== Import plugins

The sidebar's import icon (logged in only) lists whatever's under
`plugins/` and runs one for the current map viewport with a click — no
area-picker to fill in, the visible area *is* the area. Each plugin is
a directory with a `plugin.json` manifest (`{"label": "...", "command":
"./run.sh"}`) plus whatever script that command names; the entire
interface a plugin needs to know about maptool is "read
`--south`/`--west`/`--north`/`--east`, print a GeoJSON
`FeatureCollection` to stdout" (`lib/maptool/plugins.rb`) — the result
is saved as a new `.geojson` file the same way an upload would be.

`plugins/overpass-pois` ships as the reference example (a thin wrapper
around the `rake overpass:pois` logic above). A plugin that scrapes a
site whose terms of service forbid that — Park4Night, iOverlander,
Freecampsites — doesn't have to live in this (public) repo to work
here: drop its directory under `plugins/` locally and just don't
`git add` it, same as any other local-only file this app already
treats that way (`data/offline.pmtiles`, the kiwix ZIM, ...).

== Offline Wikipedia (kiwix-serve)

A point popup's "Nearby on Wikipedia" link needs
https://github.com/kiwix/kiwix-tools[kiwix-serve] running with a
Wikipedia https://library.kiwix.org[ZIM] loaded — set `KIWIX_URL` to
wherever *the server* can reach it (`docker-compose.yaml` already runs
a `kiwix` service pointed at `data/kiwix/wikipedia.zim` and wires
`KIWIX_URL` to it via its compose service name; blank/unset disables
the link, same as every other optional integration here). Used only
server-side: production sits behind a reverse proxy that forwards only
to this app, so there's no address a browser could reach kiwix-serve
at directly — `GET /kiwix/*` (`app.rb`) proxies to it instead, and the
`kiwix` service runs with `--urlRootLocation=/kiwix` so its own
emitted links/assets already carry that same prefix. Not part of the
repo — a ZIM is a large, self-contained archive, obtain one locally:

* https://library.kiwix.org/#lang=eng&category=wikipedia[Kiwix's own
  library] has pre-built Wikipedia ZIMs — a `_maxi` variant per
  language/region includes images, `_nopic` is much smaller if
  browsing images offline doesn't matter here.
* Download it to `data/kiwix/wikipedia.zim` (the exact filename
  `docker-compose.yaml`'s `kiwix` service expects — rename after
  downloading, Kiwix's own filenames are much more specific). A symlink
  there works too (`ln -s /wherever/the-real-file.zim
  data/kiwix/wikipedia.zim`), so the real file doesn't have to sit inside
  this repo's checkout at all — the compose file's volume binds that exact
  path, not the `data/kiwix/` directory, specifically so this resolves
  correctly (a symlink *inside* an already-mounted directory wouldn't: the
  container would have to resolve it itself, in its own filesystem, where
  the host path it points to doesn't exist).

Once it's there, `docker compose up -d kiwix` (or a full restart)
picks it up — `GET /api/kiwix` reports `available: true` and the
"Nearby on Wikipedia" link appears, pointed at kiwix-serve's own geo
search (`GET /kiwix/search?pattern=&latitude=&longitude=&distance=`)
through this app's own proxy rather than at kiwix-serve directly — it
already has a full search UI of its own, this just makes it reachable
through the same origin/reverse proxy as the rest of the app.

== GNSS position (gpsd)

The "GNSS" control needs https://gpsd.io[gpsd] running and pointed at
a receiver (`gpsd /dev/ttyUSB0`, or whatever device the receiver shows
up as). On openSUSE, `gpsd` is a package with systemd units
(`gpsd.service` / `gpsd.socket`); on other distributions, check your
package manager. No app-side configuration is needed — it just
connects to `127.0.0.1:2947`, gpsd's default port.

To test without hardware or gpsd itself installed/running, run
`bin/gnss_simulator.rb` — it speaks gpsd's own JSON wire protocol
directly on gpsd's default port (`127.0.0.1:2947`), so
`Maptool::GpsdClient` connects to it exactly as it would to real
gpsd, no app-side changes needed. Drives a loop around a center point
(Bratislava by default) at a driving-ish pace instead of sitting at a
fixed position, since a stationary "GNSS position" isn't much to look
at on the map:

[source,shell]
----
bin/gnss_simulator.rb                       # loop around Bratislava, 300m radius
bin/gnss_simulator.rb 48.85 2.35 500         # custom center/radius (Paris, 500m)
----

`GET /api/gnss/position` should report a fix within a couple of
seconds, moving on every poll.

To test against gpsd itself instead (e.g. to exercise the actual
NMEA-parsing path, not just gpsd's JSON output), feed it a fake NMEA
stream over TCP instead of a real serial device — gpsd accepts a
`tcp://host:port` source:

[source,shell]
----
# terminal 1: a TCP server repeating one NMEA GGA/RMC sentence pair
python3 - <<'PY' 5001
import socket, sys, time

def checksum(s):
    cs = 0
    for ch in s:
        cs ^= ord(ch)
    return f"{cs:02X}"

def sentence(body):
    return f"${body}*{checksum(body)}\r\n"

port = int(sys.argv[1])
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", port))
srv.listen(1)
conn, _ = srv.accept()
gga = "GPGGA,123519,4808.916,N,01706.462,E,1,08,0.9,140.4,M,46.9,M,,"
rmc = "GPRMC,123519,A,4808.916,N,01706.462,E,000.5,084.4,230824,003.1,W"
while True:
    conn.sendall((sentence(gga) + sentence(rmc)).encode())
    time.sleep(1)
PY

# terminal 2: point gpsd at it instead of a real device
gpsd -N -n tcp://127.0.0.1:5001
----

`GET /api/gnss/position` should then report the fake fix (Bratislava
Castle, in the sentences above) within a couple of seconds.

== Tests

`bundle exec rake test` runs the whole suite (`test/test_*.rb`,
Minitest — a default gem, but pinned in the `Gemfile`'s `:test` group
for reproducibility, same reasoning as `rake` itself). `test_app.rb`
drives `app.rb`'s routes end to end through `Rack::MockRequest` rather
than a real server; everything else tests one `lib/maptool/` module in
isolation. `MAPTOOL_DATA_DIR` (see `app.rb`) is pointed at a throwaway
`Dir.mktmpdir` by `test/test_helper.rb` before `app.rb` loads, so
running the suite never touches real `data/` — the app-level tests
also wipe that directory between each test (see `AppTestHelper#teardown`),
since `GPX_STORE`/`LIVETRACK_STORE` are process-wide constants shared
by every test in the run.

`LiveTrackClient`'s own tests use a synthetic fixture (no network) —
its parsing was checked against real Garmin sessions by hand during
development, not by the suite, since that needs a live, unexpired
session token to fetch. `Fit`'s tests likewise build a minimal FIT
file by hand rather than shipping a real device fixture; correctness
against real files (position/altitude fields, definition-message
framing, compressed-timestamp headers) was checked by hand against
`python-fitparse`'s test fixtures during development — see
`docs/architecture.adoc`.

== Continuous integration

`.gitea/workflows/ci.yml` runs on every push/PR against a self-hosted
Gitea Actions runner: `ruby -c` on every `.rb` file, a `require_relative
"app"` smoke check that the app loads without error, `bundle exec rake
test` (see above), and `node --check` on every file under
`public/js/` — no JS test suite, just a syntax check.

`.gitea/workflows/renovate.yml` runs
https://docs.renovatebot.com[Renovate] (the `renovate/renovate` image)
weekly (Mondays, also triggerable manually), opening PRs for outdated
dependencies (Bundler `Gemfile`/`Gemfile.lock`, auto-detected — no
`renovate.json` needed) against this Gitea instance
(`RENOVATE_PLATFORM=gitea`). It runs on the native `deploy` runner (see
`docs/deployment.adoc`) as a one-shot `docker run`, not a persistent
service. Needs a `RENOVATE_TOKEN` repo secret — a Gitea personal
access token (*Settings → Applications → Generate New Token*, with
repository read/write scope) from whichever account should author the
PRs; `RENOVATE_ONBOARDING=false` skips Renovate's usual first-PR
"configure me" step and goes straight to dependency update PRs using
its defaults.