maptool/docs/development.adoc

= Development guide
:toc:
:toclevels: 2

== Web app only

[source,shell]
----
bundle install
bundle exec puma -C config/puma.rb
----

Open http://127.0.0.1:4567. For auto-reload during development, run
via the `rerun` gem instead:

[source,shell]
----
bundle exec rerun -- bundle exec puma -C config/puma.rb
----

TIP: Browsers apply heuristic caching to static files even without
explicit cache headers, so an edit to `public/` (or `views/`) can
appear not to have taken effect on the next reload. Force a real
reload (e.g. Ctrl/Cmd+Shift+R) if that happens.

== Offline map data

To enable the offline style, place a PMTiles file at
`data/offline.pmtiles` (see `docs/architecture.adoc` for how it is
served and styled). Logged-in users can generate this from the app
itself — see "In-app regeneration" below — or it can be built by hand:

* Extract a region from Protomaps' own daily full-planet basemap build
  (https://build-metadata.protomaps.dev/builds.json[build index],
  files at `https://build.protomaps.com/<key>`, ~120GB, ODbL/OSM
  attribution) with the
  https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI's `extract`
  subcommand, which pulls only the byte ranges covering a bbox over
  HTTP range requests rather than the whole file:
  `pmtiles extract https://build.protomaps.com/<key> data/offline.pmtiles --bbox=west,south,east,north --maxzoom=15`.
  (The old `app.protomaps.com` extract-builder UI this section used to
  point at has since redirected elsewhere — this is the same
  underlying mechanism, just the CLI instead of a web UI.)
* To convert some other vector tile source (e.g. an `.mbtiles` file)
  into PMTiles instead, use the same CLI's `convert` subcommand:
  `pmtiles convert in.mbtiles data/offline.pmtiles`. Its schema is
  what `protomaps-themes-base` (the styling library used for the
  offline style) expects.

Once the file is in place, `GET /api/offline-tiles` reports
`available: true` and the map defaults to the offline style.

== Offline terrain data (hillshade)

The sidebar's "Terrain shading" toggle works online out of the box
(public https://github.com/tilezen/joerd[Terrarium] elevation tiles,
no API key). For it to work offline too, place a PMTiles archive of
Terrarium-encoded `raster-dem` tiles at `data/terrain.pmtiles`
(`GET /api/terrain-tiles` reports `available: true` once it's there,
same pattern as offline map data above). Also generatable in-app (see
below) or by hand, the same `go-pmtiles extract` mechanism as
`data/offline.pmtiles` above — but from
https://mapterhorn.com[Mapterhorn], a Protomaps-affiliated project
publishing pre-built Terrarium-encoded terrain as PMTiles (a global
`planet.pmtiles`, z0-12, plus ~450 regional chunks covering z13 and up
— see https://mapterhorn.com/data-access/[their own data-access page]
for the full manifest and per-source license/attribution details, CC-BY-4.0
and similar, varying by region):

[source,shell]
----
pmtiles extract https://download.mapterhorn.com/planet.pmtiles planet-extract.pmtiles --bbox=west,south,east,north --maxzoom=15
# only if a regional chunk covers the bbox too (see download_urls.json) —
# most bboxes hit exactly one, a bbox straddling a chunk boundary hits two:
pmtiles extract https://download.mapterhorn.com/<chunk>.pmtiles chunk-extract.pmtiles --bbox=west,south,east,north --maxzoom=15
pmtiles merge planet-extract.pmtiles chunk-extract.pmtiles data/terrain.pmtiles
----

Tiles are 512px WebP, not the 256px PNG this app's *online* Terrarium
source uses — `public/js/dem.js`'s own point-elevation lookups
(`shadow.js`'s terrain-shadow overlay) read the actual decoded tile
size rather than assuming one, so this isn't a problem either way.

== In-app regeneration

A logged-in user can also generate both files above from the app
itself, no local tooling needed — the sidebar's "Offline data" icon
(Offline hub group) shows a checklist of European countries
(`lib/maptool/data/europe_countries.geojson`, Natural Earth 110m
admin-0, public domain) plus each file's status (size, last updated).
Ticking countries and clicking "Apply changes" saves the selection
(`data/offline_regions.json`, an array of ISO A2 codes) and rebuilds
both files to cover it.

There's no standalone per-country file to individually delete —
`offline.pmtiles`/`terrain.pmtiles` are always one combined extract for
the *whole current selection*. "Remove a country" and "refresh a
country" are both just "change the selection, then Apply" (unticking a
country and applying rebuilds without it; nothing to separately
delete). Every Apply re-extracts the entire selection from scratch, not
just what changed — fine given extraction is fast (byte-range requests
against remote archives, not per-tile fetches), but worth revisiting if
someone selects half of Europe at once and finds it slow.

This wraps exactly the manual `go-pmtiles` workflows documented above
(`lib/maptool/pmtiles_builder.rb`) — the server needs the
https://github.com/protomaps/go-pmtiles[go-pmtiles] CLI installed and
on `PATH` (override with the `PMTILES_BIN` env var if it lives
somewhere else), using `--region=<geojson>` (a `FeatureCollection` of
the selected countries — go-pmtiles treats multiple features as their
union already, no union/dissolve math needed on this side) rather than
`--bbox=` for precise, per-country extraction. terrain.pmtiles extracts
`planet.pmtiles` (always) plus whichever Mapterhorn regional chunks
intersect the selection, `merge`s them if there was more than one —
always safe regardless of how many/which countries are selected, since
each chunk is Mapterhorn's own pre-partitioned, non-overlapping source
file (extracting the *same* source once per country and merging those,
by contrast, would fail — adjacent countries' region-extracts share
border tiles, since extraction copies whole tiles rather than clipping
vector content to the polygon, and `pmtiles merge` refuses any
overlapping tile id between inputs).

A build runs in the background (`GET /api/offline-regions` to list/see
the current selection, `POST` to change it, then `POST
/api/offline-tiles/build` / `/api/terrain-tiles/build`, polled via the
matching `*/build-status` route) — only one of each kind at a time;
starting a second while one's running gets a 409, as does starting
either with nothing selected yet.

== Camper-relevant POIs (Overpass)

`rake overpass:pois[south,west,north,east,out.geojson]` queries
https://overpass-api.de[Overpass] for drinking water, dump stations,
LPG refill, public toilets, laundromats, supermarkets/convenience
stores, and springs within a bbox, and writes the result as a
`.geojson` file (`lib/maptool/overpass_pois.rb`). Nodes only (nothing
that's mapped as a way/relation, e.g. a supermarket traced as a
building outline rather than a single point). Also reachable from
inside the running app itself, without the command line — see Import
plugins below, which wraps this exact query as `plugins/overpass-pois`.

== Import plugins

The sidebar's import icon (logged in only) lists whatever's under
`plugins/` and runs one for the current map viewport with a click — no
area-picker to fill in, the visible area *is* the area. Each plugin is
a directory with a `plugin.json` manifest (`{"label": "...", "command":
"./run.sh"}`) plus whatever script that command names; the entire
interface a plugin needs to know about maptool is "read
`--south`/`--west`/`--north`/`--east`, print a GeoJSON
`FeatureCollection` to stdout" (`lib/maptool/plugins.rb`) — the result
is saved as a new `.geojson` file the same way an upload would be.

`plugins/overpass-pois` ships as the reference example (a thin wrapper
around the `rake overpass:pois` logic above). A plugin that scrapes a
site whose terms of service forbid that — Park4Night, iOverlander,
Freecampsites — doesn't have to live in this (public) repo to work
here: drop its directory under `plugins/` locally and just don't
`git add` it, same as any other local-only file this app already
treats that way (`data/offline.pmtiles`, the kiwix ZIM, ...).

== Offline Wikipedia (kiwix-serve)

A point popup's "Nearby on Wikipedia" link needs
https://github.com/kiwix/kiwix-tools[kiwix-serve] running with a
Wikipedia https://library.kiwix.org[ZIM] loaded — set `KIWIX_URL` to
wherever *the server* can reach it (`docker-compose.yaml` already runs
a `kiwix` service pointed at `data/kiwix/wikipedia.zim` and wires
`KIWIX_URL` to it via its compose service name; blank/unset disables
the link, same as every other optional integration here). Used only
server-side: production sits behind a reverse proxy that forwards only
to this app, so there's no address a browser could reach kiwix-serve
at directly — `GET /kiwix/*` (`app.rb`) proxies to it instead, and the
`kiwix` service runs with `--urlRootLocation=/kiwix` so its own
emitted links/assets already carry that same prefix. Not part of the
repo — a ZIM is a large, self-contained archive, obtain one locally:

* https://library.kiwix.org/#lang=eng&category=wikipedia[Kiwix's own
  library] has pre-built Wikipedia ZIMs — a `_maxi` variant per
  language/region includes images, `_nopic` is much smaller if
  browsing images offline doesn't matter here.
* Download it to `data/kiwix/wikipedia.zim` (the exact filename
  `docker-compose.yaml`'s `kiwix` service expects — rename after
  downloading, Kiwix's own filenames are much more specific). A symlink
  there works too (`ln -s /wherever/the-real-file.zim
  data/kiwix/wikipedia.zim`), so the real file doesn't have to sit inside
  this repo's checkout at all — the compose file's volume binds that exact
  path, not the `data/kiwix/` directory, specifically so this resolves
  correctly (a symlink *inside* an already-mounted directory wouldn't: the
  container would have to resolve it itself, in its own filesystem, where
  the host path it points to doesn't exist).

Once it's there, `docker compose up -d kiwix` (or a full restart)
picks it up — `GET /api/kiwix` reports `available: true` and the
"Nearby on Wikipedia" link appears, pointed at kiwix-serve's own geo
search (`GET /kiwix/search?pattern=&latitude=&longitude=&distance=`)
through this app's own proxy rather than at kiwix-serve directly — it
already has a full search UI of its own, this just makes it reachable
through the same origin/reverse proxy as the rest of the app.

== GNSS position (gpsd)

The "GNSS" control needs https://gpsd.io[gpsd] running and pointed at
a receiver (`gpsd /dev/ttyUSB0`, or whatever device the receiver shows
up as). On openSUSE, `gpsd` is a package with systemd units
(`gpsd.service` / `gpsd.socket`); on other distributions, check your
package manager. No app-side configuration is needed — it just
connects to `127.0.0.1:2947`, gpsd's default port.

To test without hardware or gpsd itself installed/running, run
`bin/gnss_simulator.rb` — it speaks gpsd's own JSON wire protocol
directly on gpsd's default port (`127.0.0.1:2947`), so
`Maptool::GpsdClient` connects to it exactly as it would to real
gpsd, no app-side changes needed. Drives a loop around a center point
(Bratislava by default) at a driving-ish pace instead of sitting at a
fixed position, since a stationary "GNSS position" isn't much to look
at on the map:

[source,shell]
----
bin/gnss_simulator.rb                       # loop around Bratislava, 300m radius
bin/gnss_simulator.rb 48.85 2.35 500         # custom center/radius (Paris, 500m)
----

`GET /api/gnss/position` should report a fix within a couple of
seconds, moving on every poll.

To test against gpsd itself instead (e.g. to exercise the actual
NMEA-parsing path, not just gpsd's JSON output), feed it a fake NMEA
stream over TCP instead of a real serial device — gpsd accepts a
`tcp://host:port` source:

[source,shell]
----
# terminal 1: a TCP server repeating one NMEA GGA/RMC sentence pair
python3 - <<'PY' 5001
import socket, sys, time

def checksum(s):
    cs = 0
    for ch in s:
        cs ^= ord(ch)
    return f"{cs:02X}"

def sentence(body):
    return f"${body}*{checksum(body)}\r\n"

port = int(sys.argv[1])
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", port))
srv.listen(1)
conn, _ = srv.accept()
gga = "GPGGA,123519,4808.916,N,01706.462,E,1,08,0.9,140.4,M,46.9,M,,"
rmc = "GPRMC,123519,A,4808.916,N,01706.462,E,000.5,084.4,230824,003.1,W"
while True:
    conn.sendall((sentence(gga) + sentence(rmc)).encode())
    time.sleep(1)
PY

# terminal 2: point gpsd at it instead of a real device
gpsd -N -n tcp://127.0.0.1:5001
----

`GET /api/gnss/position` should then report the fake fix (Bratislava
Castle, in the sentences above) within a couple of seconds.

== Tests

`bundle exec rake test` runs the whole suite (`test/test_*.rb`,
Minitest — a default gem, but pinned in the `Gemfile`'s `:test` group
for reproducibility, same reasoning as `rake` itself). `test_app.rb`
drives `app.rb`'s routes end to end through `Rack::MockRequest` rather
than a real server; everything else tests one `lib/maptool/` module in
isolation. `MAPTOOL_DATA_DIR` (see `app.rb`) is pointed at a throwaway
`Dir.mktmpdir` by `test/test_helper.rb` before `app.rb` loads, so
running the suite never touches real `data/` — the app-level tests
also wipe that directory between each test (see `AppTestHelper#teardown`),
since `GPX_STORE`/`LIVETRACK_STORE` are process-wide constants shared
by every test in the run.

`LiveTrackClient`'s own tests use a synthetic fixture (no network) —
its parsing was checked against real Garmin sessions by hand during
development, not by the suite, since that needs a live, unexpired
session token to fetch. `Fit`'s tests likewise build a minimal FIT
file by hand rather than shipping a real device fixture; correctness
against real files (position/altitude fields, definition-message
framing, compressed-timestamp headers) was checked by hand against
`python-fitparse`'s test fixtures during development — see
`docs/architecture.adoc`.

== Continuous integration

`.gitea/workflows/ci.yml` runs on every push/PR against a self-hosted
Gitea Actions runner: `ruby -c` on every `.rb` file, a `require_relative
"app"` smoke check that the app loads without error, `bundle exec rake
test` (see above), and `node --check` on every file under
`public/js/` — no JS test suite, just a syntax check.

`.gitea/workflows/renovate.yml` runs
https://docs.renovatebot.com[Renovate] (the `renovate/renovate` image)
weekly (Mondays, also triggerable manually), opening PRs for outdated
dependencies (Bundler `Gemfile`/`Gemfile.lock`, auto-detected — no
`renovate.json` needed) against this Gitea instance
(`RENOVATE_PLATFORM=gitea`). It runs on the native `deploy` runner (see
`docs/deployment.adoc`) as a one-shot `docker run`, not a persistent
service. Needs a `RENOVATE_TOKEN` repo secret — a Gitea personal
access token (*Settings → Applications → Generate New Token*, with
repository read/write scope) from whichever account should author the
PRs; `RENOVATE_ONBOARDING=false` skips Renovate's usual first-PR
"configure me" step and goes straight to dependency update PRs using
its defaults.