maptool/docs/development.adoc

= Development guide
:toc:
:toclevels: 2

== Web app only

[source,shell]
----
bundle install
bundle exec puma -C config/puma.rb
----

Open http://127.0.0.1:4567. For auto-reload during development, run
via the `rerun` gem instead:

[source,shell]
----
bundle exec rerun -- bundle exec puma -C config/puma.rb
----

TIP: Browsers apply heuristic caching to static files even without
explicit cache headers, so an edit to `public/` (or `views/`) can
appear not to have taken effect on the next reload. Force a real
reload (e.g. Ctrl/Cmd+Shift+R) if that happens.

== Offline map data

To enable the offline style, place a PMTiles file at
`data/offline.pmtiles` (see `docs/architecture.adoc` for how it is
served and styled — the same OpenMapTiles schema/style the online map
uses, just from local data). Logged-in users can generate this from
the app itself — see "In-app regeneration" below — or it can be built
by hand:

* Download one or more country extracts from
  https://download.geofabrik.de[Geofabrik] (`.osm.pbf`, ODbL/OSM
  attribution) — e.g.
  `curl -O https://download.geofabrik.de/europe/slovakia-latest.osm.pbf`.
* If more than one country, merge them with
  https://github.com/osmcode/osmium-tool[osmium-tool]:
  `osmium merge slovakia-latest.osm.pbf austria-latest.osm.pbf -o merged.osm.pbf`
  (each Geofabrik file is already clipped to that country's real
  border, so no separate bbox/polygon clip step is needed either
  before or after — skip this step entirely for a single country).
* Tile it with https://github.com/onthegomap/planetiler[Planetiler]'s
  default profile, which *is* OpenMapTiles — no `--profile` flag
  needed — writing PMTiles directly:
  `java -jar planetiler.jar --osm-path=merged.osm.pbf --output=data/offline.pmtiles`
  (needs Java 21+; `planetiler.jar` from
  https://github.com/onthegomap/planetiler/releases/latest[the latest release]).

Once the file is in place, `GET /api/offline-tiles` reports
`available: true` and the map defaults to the offline style.

== Offline terrain data (hillshade)

The sidebar's "Terrain shading" toggle works online out of the box
(public https://github.com/tilezen/joerd[Terrarium] elevation tiles,
no API key). For it to work offline too, place a PMTiles archive of
Terrarium-encoded `raster-dem` tiles at `data/terrain.pmtiles`
(`GET /api/terrain-tiles` reports `available: true` once it's there,
same pattern as offline map data above). Also generatable in-app (see
below) or by hand, the same `go-pmtiles extract` mechanism as
`data/offline.pmtiles` above — but from
https://mapterhorn.com[Mapterhorn], a Protomaps-affiliated project
publishing pre-built Terrarium-encoded terrain as PMTiles (a global
`planet.pmtiles`, z0-12, plus ~450 regional chunks covering z13 and up
— see https://mapterhorn.com/data-access/[their own data-access page]
for the full manifest and per-source license/attribution details, CC-BY-4.0
and similar, varying by region):

[source,shell]
----
pmtiles extract https://download.mapterhorn.com/planet.pmtiles planet-extract.pmtiles --bbox=west,south,east,north --maxzoom=15
# only if a regional chunk covers the bbox too (see download_urls.json) —
# most bboxes hit exactly one, a bbox straddling a chunk boundary hits two:
pmtiles extract https://download.mapterhorn.com/<chunk>.pmtiles chunk-extract.pmtiles --bbox=west,south,east,north --maxzoom=15
pmtiles merge planet-extract.pmtiles chunk-extract.pmtiles data/terrain.pmtiles
----

Tiles are 512px WebP, not the 256px PNG this app's *online* Terrarium
source uses — `public/js/dem.js`'s own point-elevation lookups
(`shadow.js`'s terrain-shadow overlay) read the actual decoded tile
size rather than assuming one, so this isn't a problem either way.

== In-app regeneration

A logged-in user can also generate both files above from the app
itself, no local tooling needed — the sidebar's "Offline data" icon
(Offline hub group) shows a checklist of European countries
(`lib/maptool/data/europe_countries.geojson`, Natural Earth 110m
admin-0, public domain) plus each file's status (size, last updated).
Ticking countries and clicking "Apply changes" saves the selection
(`data/offline_regions.json`, an array of ISO A2 codes) and rebuilds
both files to cover it.

There's no standalone per-country file to individually delete —
`offline.pmtiles`/`terrain.pmtiles` are always one combined extract for
the *whole current selection*. "Remove a country" and "refresh a
country" are both just "change the selection, then Apply" (unticking a
country and applying rebuilds without it; nothing to separately
delete). Every Apply re-extracts the entire selection from scratch, not
just what changed — fine given extraction is fast (byte-range requests
against remote archives, not per-tile fetches), but worth revisiting if
someone selects half of Europe at once and finds it slow.

This wraps exactly the manual workflows documented above
(`lib/maptool/pmtiles_builder.rb`). offline.pmtiles downloads each
selected country's Geofabrik extract, `osmium merge`s them if there's
more than one, then tiles the result with Planetiler — the server
needs `osmium` and `java` + `planetiler.jar` installed and on `PATH`
(override with the `OSMIUM_BIN`/`JAVA_BIN`/`PLANETILER_JAR` env vars if
they live somewhere else; the Docker image installs all three, see
`Dockerfile`). terrain.pmtiles still uses `go-pmtiles`
(`PMTILES_BIN`) against Mapterhorn, unrelated to this — it extracts
`planet.pmtiles` (always) plus whichever Mapterhorn regional chunks
intersect the selection, `merge`s them if there was more than one —
always safe regardless of how many/which countries are selected, since
each chunk is Mapterhorn's own pre-partitioned, non-overlapping source
file (extracting the *same* source once per country and merging those,
by contrast, would fail — adjacent countries' region-extracts share
border tiles, since extraction copies whole tiles rather than clipping
vector content to the polygon, and `pmtiles merge` refuses any
overlapping tile id between inputs).

A build runs in the background (`GET /api/offline-regions` to list/see
the current selection, `POST` to change it, then `POST
/api/offline-tiles/build` / `/api/terrain-tiles/build`, polled via the
matching `*/build-status` route) — only one of each kind at a time;
starting a second while one's running gets a 409, as does starting
either with nothing selected yet.

Each successful build also writes a small sidecar (`offline.pmtiles.
meta.json` — each source country's code/Geofabrik slug/Last-Modified;
`terrain.pmtiles.meta.json` — the Mapterhorn source names/md5sums)
recording what was actually built, since the `.pmtiles` file itself
carries no such record. `GET /api/offline-tiles`/`/api/terrain-tiles`
include it as `meta`, shown in the panel next to the existing
size/updated line. `GET .../check-update` compares that recorded build
against each source's live upstream state (a Geofabrik `HEAD` per
country for offline, the Mapterhorn manifest for terrain — without
rebuilding anything) and reports `update_available: true/false/null` —
`null` means "don't know" (no local build yet, or a check itself
failed), kept as its own route rather than folded into the plain
status endpoints above since those are on the page-load hot path (the
topbar's offline-status dot) and this one reaches an external service,
only worth doing when the Offline data panel is actually open.

== Camper-relevant POIs (Overpass)

`rake overpass:pois[south,west,north,east,out.geojson]` queries
https://overpass-api.de[Overpass] for drinking water, dump stations,
LPG refill, public toilets, laundromats, supermarkets/convenience
stores, and springs within a bbox, and writes the result as a
`.geojson` file (`lib/maptool/overpass_pois.rb`). Nodes only (nothing
that's mapped as a way/relation, e.g. a supermarket traced as a
building outline rather than a single point). Also reachable from
inside the running app itself, without the command line — see Import
plugins below, which wraps this exact query as `plugins/overpass-pois`.

== Import plugins

The sidebar's import icon (logged in only) lists whatever's under
`plugins/` and runs one for the current map viewport with a click — no
area-picker to fill in, the visible area *is* the area. Each plugin is
a directory with a `plugin.json` manifest (`{"label": "...", "command":
"./run.sh"}`) plus whatever script that command names; the entire
interface a plugin needs to know about maptool is "read
`--south`/`--west`/`--north`/`--east`, print a GeoJSON
`FeatureCollection` to stdout" (`lib/maptool/plugins.rb`) — the result
is saved as a new `.geojson` file the same way an upload would be.

`plugins/overpass-pois` ships as the reference example (a thin wrapper
around the `rake overpass:pois` logic above). A plugin that scrapes a
site whose terms of service forbid that — Park4Night, iOverlander,
Freecampsites — doesn't have to live in this (public) repo to work
here: drop its directory under `plugins/` locally and just don't
`git add` it, same as any other local-only file this app already
treats that way (`data/offline.pmtiles`, the kiwix ZIM, ...).

== Offline Wikipedia (kiwix-serve)

A point popup's "Nearby on Wikipedia" link needs
https://github.com/kiwix/kiwix-tools[kiwix-serve] running with a
Wikipedia https://library.kiwix.org[ZIM] loaded — set `KIWIX_URL` to
wherever *the server* can reach it (`docker-compose.yaml` already runs
a `kiwix` service pointed at `data/kiwix/wikipedia.zim` and wires
`KIWIX_URL` to it via its compose service name; blank/unset disables
the link, same as every other optional integration here). Used only
server-side: production sits behind a reverse proxy that forwards only
to this app, so there's no address a browser could reach kiwix-serve
at directly — `GET /kiwix/*` (`app.rb`) proxies to it instead, and the
`kiwix` service runs with `--urlRootLocation=/kiwix` so its own
emitted links/assets already carry that same prefix. Not part of the
repo — a ZIM is a large, self-contained archive, obtain one locally:

* https://library.kiwix.org/#lang=eng&category=wikipedia[Kiwix's own
  library] has pre-built Wikipedia ZIMs — a `_maxi` variant per
  language/region includes images, `_nopic` is much smaller if
  browsing images offline doesn't matter here.
* Download it to `data/kiwix/wikipedia.zim` (the exact filename
  `docker-compose.yaml`'s `kiwix` service expects — rename after
  downloading, Kiwix's own filenames are much more specific). A symlink
  there works too (`ln -s /wherever/the-real-file.zim
  data/kiwix/wikipedia.zim`), so the real file doesn't have to sit inside
  this repo's checkout at all — the compose file's volume binds that exact
  path, not the `data/kiwix/` directory, specifically so this resolves
  correctly (a symlink *inside* an already-mounted directory wouldn't: the
  container would have to resolve it itself, in its own filesystem, where
  the host path it points to doesn't exist).

Once it's there, `docker compose up -d kiwix` (or a full restart)
picks it up — `GET /api/kiwix` reports `available: true` and the
"Nearby on Wikipedia" link appears, pointed at kiwix-serve's own geo
search (`GET /kiwix/search?pattern=&latitude=&longitude=&distance=`)
through this app's own proxy rather than at kiwix-serve directly — it
already has a full search UI of its own, this just makes it reachable
through the same origin/reverse proxy as the rest of the app.

== GNSS position (gpsd)

The "GNSS" control needs https://gpsd.io[gpsd] running and pointed at
a receiver (`gpsd /dev/ttyUSB0`, or whatever device the receiver shows
up as). On openSUSE, `gpsd` is a package with systemd units
(`gpsd.service` / `gpsd.socket`); on other distributions, check your
package manager. No app-side configuration is needed — it just
connects to `127.0.0.1:2947`, gpsd's default port.

To test without hardware or gpsd itself installed/running, run
`bin/gnss_simulator.rb` — it speaks gpsd's own JSON wire protocol
directly on gpsd's default port (`127.0.0.1:2947`), so
`Maptool::GpsdClient` connects to it exactly as it would to real
gpsd, no app-side changes needed. Drives a loop around a center point
(Bratislava by default) at a driving-ish pace instead of sitting at a
fixed position, since a stationary "GNSS position" isn't much to look
at on the map:

[source,shell]
----
bin/gnss_simulator.rb                       # loop around Bratislava, 300m radius
bin/gnss_simulator.rb 48.85 2.35 500         # custom center/radius (Paris, 500m)
----

`GET /api/gnss/position` should report a fix within a couple of
seconds, moving on every poll.

To test against gpsd itself instead (e.g. to exercise the actual
NMEA-parsing path, not just gpsd's JSON output), feed it a fake NMEA
stream over TCP instead of a real serial device — gpsd accepts a
`tcp://host:port` source:

[source,shell]
----
# terminal 1: a TCP server repeating one NMEA GGA/RMC sentence pair
python3 - <<'PY' 5001
import socket, sys, time

def checksum(s):
    cs = 0
    for ch in s:
        cs ^= ord(ch)
    return f"{cs:02X}"

def sentence(body):
    return f"${body}*{checksum(body)}\r\n"

port = int(sys.argv[1])
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", port))
srv.listen(1)
conn, _ = srv.accept()
gga = "GPGGA,123519,4808.916,N,01706.462,E,1,08,0.9,140.4,M,46.9,M,,"
rmc = "GPRMC,123519,A,4808.916,N,01706.462,E,000.5,084.4,230824,003.1,W"
while True:
    conn.sendall((sentence(gga) + sentence(rmc)).encode())
    time.sleep(1)
PY

# terminal 2: point gpsd at it instead of a real device
gpsd -N -n tcp://127.0.0.1:5001
----

`GET /api/gnss/position` should then report the fake fix (Bratislava
Castle, in the sentences above) within a couple of seconds.

== Tests

`bundle exec rake test` runs the whole suite (`test/test_*.rb`,
Minitest — a default gem, but pinned in the `Gemfile`'s `:test` group
for reproducibility, same reasoning as `rake` itself). `test_app.rb`
drives `app.rb`'s routes end to end through `Rack::MockRequest` rather
than a real server; everything else tests one `lib/maptool/` module in
isolation. `MAPTOOL_DATA_DIR` (see `app.rb`) is pointed at a throwaway
`Dir.mktmpdir` by `test/test_helper.rb` before `app.rb` loads, so
running the suite never touches real `data/` — the app-level tests
also wipe that directory between each test (see `AppTestHelper#teardown`),
since `GPX_STORE`/`LIVETRACK_STORE` are process-wide constants shared
by every test in the run.

`LiveTrackClient`'s own tests use a synthetic fixture (no network) —
its parsing was checked against real Garmin sessions by hand during
development, not by the suite, since that needs a live, unexpired
session token to fetch. `Fit`'s tests likewise build a minimal FIT
file by hand rather than shipping a real device fixture; correctness
against real files (position/altitude fields, definition-message
framing, compressed-timestamp headers) was checked by hand against
`python-fitparse`'s test fixtures during development — see
`docs/architecture.adoc`.

== Continuous integration

`.gitea/workflows/ci.yml` runs on every push/PR against a self-hosted
Gitea Actions runner: `ruby -c` on every `.rb` file, a `require_relative
"app"` smoke check that the app loads without error, `bundle exec rake
test` (see above), and `node --check` on every file under
`public/js/` — no JS test suite, just a syntax check.

`.gitea/workflows/renovate.yml` runs
https://docs.renovatebot.com[Renovate] (the `renovate/renovate` image)
weekly (Mondays, also triggerable manually), opening PRs for outdated
dependencies (Bundler `Gemfile`/`Gemfile.lock`, auto-detected — no
`renovate.json` needed) against this Gitea instance
(`RENOVATE_PLATFORM=gitea`). It runs on the native `deploy` runner (see
`docs/deployment.adoc`) as a one-shot `docker run`, not a persistent
service. Needs a `RENOVATE_TOKEN` repo secret — a Gitea personal
access token (*Settings → Applications → Generate New Token*, with
repository read/write scope) from whichever account should author the
PRs; `RENOVATE_ONBOARDING=false` skips Renovate's usual first-PR
"configure me" step and goes straight to dependency update PRs using
its defaults.