From 079ac37787a4091fc55c09fe56d9f20c54a95772 Mon Sep 17 00:00:00 2001 From: Roman Priesol Date: Sun, 13 Sep 2026 10:31:59 +0200 Subject: [PATCH] Wire offline hiking trails into the real build, API, and renderer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit data/hiking_trails.geojson now generates automatically as the last step of every offline.pmtiles build (PmtilesBuilder.extract_hiking_trails, best-effort — a failure here doesn't fail the whole build), merged across all selected countries with cross-border dedup by OSM relation id (HikingTrails.extract_merged). Served at GET /tiles/hiking_trails.geojson and GET /api/hiking-trails, fetched client-side only once the overlay is actually toggled on. hiking_trails.js now prefers this local file whenever it exists, regardless of which base map (online/offline) is active — same "prefer local when available" story as HillshadeControl already uses for terrain. Offline click hit-tests the real vector layer directly (queryRenderedFeatures) instead of querying Overpass — instant, no network. Coordinates are rounded to 5dp to keep the combined file smaller; colour tag values are normalized against a known-safe set before ever reaching a style paint property. Verified extract_merged (new: combines multiple countries' extracts, dedupes overlapping relations) against two real cached country PBFs on brick directly, not just fixtures. --- TODO.adoc | 5 +- app.rb | 25 ++++++ docs/development.adoc | 44 ++++++++++ lib/maptool/hiking_trails.rb | 30 ++++++- lib/maptool/pmtiles_builder.rb | 27 ++++++ public/js/hiking_trails.js | 148 ++++++++++++++++++++++++++++----- public/js/map.js | 5 ++ public/js/overlays.js | 4 + test/test_app.rb | 9 ++ test/test_hiking_trails.rb | 37 +++++++++ test/test_pmtiles_builder.rb | 61 +++++++++++++- 11 files changed, 370 insertions(+), 25 deletions(-) diff --git a/TODO.adoc b/TODO.adoc index ab056a8..6c467af 100644 --- a/TODO.adoc +++ b/TODO.adoc @@ -12,8 +12,9 @@ independence from connectivity. * [x] Shaded relief / DEM layer — hillshade toggle in the sidebar, sourced from public Terrarium tiles online; reads `data/terrain.pmtiles` for offline if one's been generated (see docs/development.adoc), otherwise the toggle is online-only for now ** [ ] Terrain slope readout for parking (numeric, not just visual shading) * [ ] Contour lines from OSM for hikes and mountain passes -* [x] Hiking trail overlay — Waymarked Trails raster tiles (`public/js/hiking_trails.js`), colored per-route by the OSM `colour` tag they already render (the KST/KČT paint-blaze colors in Slovakia/Czechia, plain red elsewhere), so no styling logic needed here. Click queries a one-off Overpass `around` search (racing 3 public mirrors) for the route's name/colour/network/operator/distance, no server involved -** [x] Prototype: an offline extraction path exists after all, no custom Planetiler profile needed — `Maptool::HikingTrails.extract` (`lib/maptool/hiking_trails.rb`, `rake hiking:trails[pbf,out]`) runs `osmium tags-filter r/route=hiking` + `osmium cat -f opl` against the *same* Geofabrik `.osm.pbf` already downloaded for `offline.pmtiles` — no new download, no new dependency. Not wired into the actual offline build/rendering yet, still deferred until there's a route planner to pair it with (see the entry above) +* [x] Hiking trail overlay — Waymarked Trails raster tiles online (`public/js/hiking_trails.js`), colored per-route by the OSM `colour` tag they already render (the KST/KČT paint-blaze colors in Slovakia/Czechia, plain red elsewhere). Click queries a one-off Overpass `around` search (racing 3 public mirrors) for the route's name/colour/network/operator/distance +** [x] Offline copy wired in for real, no custom Planetiler profile needed — `Maptool::HikingTrails` (`lib/maptool/hiking_trails.rb`) runs `osmium tags-filter r/route=hiking` + `add-locations-to-ways` + `cat -f opl` against the *same* per-country Geofabrik `.osm.pbf` already downloaded for `offline.pmtiles`, no new download. Generated automatically as the last step of every offline build (`PmtilesBuilder.extract_hiking_trails`, best-effort — a failure here doesn't fail the whole build) into `data/hiking_trails.geojson`, served at `GET /tiles/hiking_trails.geojson` only once the overlay's actually toggled on. Offline click uses a local `queryRenderedFeatures` hit-test instead of Overpass — instant, no network. See docs/development.adoc +*** [ ] File size for a many-country selection could get large (Slovakia alone: ~3000 routes) — coordinates are rounded to 5dp but there's no real tiling/simplification yet; revisit if a big selection makes the browser fetch noticeably slow * [x] Overpass/OSM scripts to extract POIs (drinking water, dump stations, LPG, public toilets, laundromats, shops, springs) — `rake overpass:pois`, see docs/development.adoc; writes a `.geojson` to drop in `data/gpx/` or upload. Also runnable from inside the app itself now, see the import plugins entry below (`plugins/overpass-pois` wraps this same query). Each category now carries a real `POI_ICONS` glyph and color (not p4n2gpx's plain red marker) plus a popup description built from opening_hours/operator/website tags * [x] Import plugins: a pluggable way to add more data sources (Park4Night, iOverlander, Freecampsites, ...) without their scrapers necessarily living in this (public) repo — sidebar icon (logged in only) lists whatever's under `plugins/`, "Run for this area" uses the current map viewport as the bbox, no separate area-picker needed. Contract: a `plugin.json` manifest + a script that reads `--south`/`--west`/`--north`/`--east` and prints a GeoJSON `FeatureCollection` to stdout (`lib/maptool/plugins.rb`, `GET /api/plugins`, `POST /api/plugins/:name/run`) — output saved the same way an upload would be. `plugins/overpass-pois` ships as the reference example; a private/ToS-sensitive scraper just drops its own directory in locally and is never `git add`ed ** [x] Park4Night — private plugin (`data/plugins/p4n2gpx/`, outside this repo per the "ToS-sensitive scraper" note above); iOverlander/Freecampsites still open, same story diff --git a/app.rb b/app.rb index 0246b79..50f0423 100644 --- a/app.rb +++ b/app.rb @@ -62,6 +62,7 @@ module Maptool OFFLINE_PMTILES_PATH = File.join(DATA_DIR, "offline.pmtiles") TERRAIN_PMTILES_PATH = File.join(DATA_DIR, "terrain.pmtiles") + HIKING_TRAILS_PATH = PmtilesBuilder.hiking_trails_path(DATA_DIR) GPX_STORE = GpxStore.new(File.join(DATA_DIR, "gpx")) LIVETRACK_STORE = LiveTrackStore.new(File.join(DATA_DIR, "livetrack.json")) @@ -356,6 +357,30 @@ module Maptool json ok: true end + # Generated as a step inside POST /api/offline-tiles/build (see + # PmtilesBuilder.extract_hiking_trails) rather than its own separate + # build — it's derived from the exact same per-country .osm.pbf cache + # offline.pmtiles itself already downloads, so there's nothing of its + # own to trigger or delete; DELETE /api/offline-tiles removes this + # file too. No `meta` argument here (unlike offline/terrain above) — + # parsing the whole GeoJSON just to report a feature count would cost + # real time on every page load's status check for what's a multi-MB + # file across several countries; size/mtime alone is enough to know + # whether it's there. + get "/api/hiking-trails" do + json pmtiles_file_status(HIKING_TRAILS_PATH, nil) + end + + # Fetched client-side only when the Hiking trails overlay is actually + # toggled on (public/js/hiking_trails.js), not on page load — same + # "on demand" story as the online raster tiles it replaces once this + # is available. + get "/tiles/hiking_trails.geojson" do + halt 404 unless File.exist?(HIKING_TRAILS_PATH) + + send_file HIKING_TRAILS_PATH, type: "application/geo+json" + end + # Pinged once per page load (public/js/map.js) — url is a same-origin # path (not kiwix-serve's own address, which the browser can't reach # directly behind the reverse proxy in front of this app), proxied by diff --git a/docs/development.adoc b/docs/development.adoc index 5f05856..46f952e 100644 --- a/docs/development.adoc +++ b/docs/development.adoc @@ -101,6 +101,50 @@ source uses — `public/js/dem.js`'s own point-elevation lookups (`shadow.js`'s terrain-shadow overlay) read the actual decoded tile size rather than assuming one, so this isn't a problem either way. +== Offline hiking trails + +The sidebar's "Hiking trails" overlay works online out of the box +(waymarkedtrails.org raster tiles, `public/js/hiking_trails.js`) — +clicking a trail fires a one-off Overpass query for its name/colour/ +network/distance. For it to work offline too, `data/hiking_trails.geojson` +is generated automatically as the last step of every `offline.pmtiles` +build (`Maptool::HikingTrails`, `lib/maptool/hiking_trails.rb`), straight +from the *same* per-country `.osm.pbf` files already downloaded for +`offline.pmtiles` itself — no separate download, no third-party service: + +[source,shell] +---- +osmium tags-filter country.osm.pbf r/route=hiking -o hiking.osm.pbf +osmium add-locations-to-ways hiking.osm.pbf -o hiking-located.osm.pbf +osmium cat hiking-located.osm.pbf -f opl,locations_on_ways=true,add_metadata=false -o hiking.opl +---- + +`add-locations-to-ways` matters: a plain Geofabrik PBF's ways only carry +bare node *references*, not coordinates, so skipping this step silently +produces ways with empty x/y in the OPL output and every route gets +dropped for having no usable geometry — not an error, just wrong output +(verified against a real extract before this was caught). No general OSM +tool stitches a route relation's member ways into one line for you +(routes are a semantic grouping, not a geometry type), so +`HikingTrails.parse_opl` does that walk itself — the same thing +waymarkedtrails.org and this app's own online Overpass click handler +already do. + +Best-effort: a hiking-trails extraction problem on one country's PBF +logs a warning and is skipped rather than failing the whole +`offline.pmtiles` build over what's a bonus overlay, not the core map +(`PmtilesBuilder.extract_hiking_trails`). Coordinates are rounded to 5 +decimal places (~1m, plenty for a rendered line) to keep the combined +file smaller — for a many-country selection it can still be a sizeable +GeoJSON (Slovakia alone: ~3000 routes, ~30MB unrounded), fetched +client-side only once the overlay is actually toggled on, not on page +load. `rake hiking:trails[pbf,out]` runs the extraction by hand against +one `.osm.pbf` for testing. + +`DELETE /api/offline-tiles` removes this file along with `offline.pmtiles` +and the Geofabrik cache — there's no separate delete/rebuild route for it, +since it's not an independent build. + == In-app regeneration A logged-in user can also generate both files above from the app diff --git a/lib/maptool/hiking_trails.rb b/lib/maptool/hiking_trails.rb index ae3cbf6..124040b 100644 --- a/lib/maptool/hiking_trails.rb +++ b/lib/maptool/hiking_trails.rb @@ -63,6 +63,22 @@ module Maptool end end + # Runs extract() over one .osm.pbf per selected country and combines the + # results into a single FeatureCollection — a route relation crossing a + # border (an E-path, say) shows up once per country whose extract it + # was pulled from, deduplicated here by OSM relation id (properties.id) + # rather than left to render as overlapping duplicate lines. + # extract: same injectable-seam reasoning as PmtilesBuilder's own + # download_extract:/fetch_manifest: — lets a test swap out the real + # (osmium-shelling-out) implementation for a canned one. + def self.extract_merged(pbf_paths, extract: method(:extract)) + seen = {} + pbf_paths.each do |pbf_path| + extract.call(pbf_path)["features"].each { |feature| seen[feature["properties"]["id"]] ||= feature } + end + { "type" => "FeatureCollection", "features" => seen.values } + end + # Pure — no subprocess — so it's cheaply testable against a canned OPL # fixture, same "keep the real logic testable without the real tool # installed" reasoning OverpassPois.to_geojson already gives for staying @@ -81,7 +97,7 @@ module Maptool ways[head[1..].to_i] = way_coordinates(fields) when "r" tags = tags_field(fields) - relations << { tags: tags, members: members_field(fields) } if tags["route"] == "hiking" + relations << { id: head[1..].to_i, tags: tags, members: members_field(fields) } if tags["route"] == "hiking" end end @@ -99,6 +115,7 @@ module Maptool distance = lines.sum { |coords| TrackStats.compute(coords)["distance_meters"] } tags = relation[:tags] properties = { + "id" => relation[:id], "name" => tags["name"] || tags["description"], "distance_meters" => distance.round(1), } @@ -107,11 +124,20 @@ module Maptool { "type" => "Feature", "properties" => properties.compact, - "geometry" => { "type" => "MultiLineString", "coordinates" => lines }, + # Rounded to 5 decimal places (~1m) — full OSM float precision + # (~1cm) is pointless for a rendered line and roughly doubles the + # size of every coordinate pair once this many-country GeoJSON is + # actually served to a browser. + "geometry" => { "type" => "MultiLineString", "coordinates" => round_coordinates(lines) }, } end private_class_method :relation_feature + def self.round_coordinates(lines) + lines.map { |coords| coords.map { |lon, lat| [lon.round(5), lat.round(5)] } } + end + private_class_method :round_coordinates + def self.field(fields, letter) fields.find { |f| f.start_with?(letter) } end diff --git a/lib/maptool/pmtiles_builder.rb b/lib/maptool/pmtiles_builder.rb index f7966e7..264cd0d 100644 --- a/lib/maptool/pmtiles_builder.rb +++ b/lib/maptool/pmtiles_builder.rb @@ -5,6 +5,7 @@ require "json" require "tmpdir" require "fileutils" require "etc" +require_relative "hiking_trails" module Maptool # Regenerates data/offline.pmtiles and data/terrain.pmtiles for the @@ -289,12 +290,37 @@ module Maptool run_planetiler!("--osm-path=#{osm_path}", "--output=#{tmp_path}", "--download", "--threads=#{Etc.nprocessors}") File.rename(tmp_path, File.join(data_dir, "offline.pmtiles")) File.write(offline_meta_path(data_dir), JSON.generate(sources: sources.map { |s| s.slice("code", "slug", "last_modified") })) + + report_step.call("extracting hiking trails") + extract_hiking_trails(sources, data_dir) ensure FileUtils.remove_entry(scratch_dir) if scratch_dir && Dir.exist?(scratch_dir) File.delete(tmp_path) if tmp_path && File.exist?(tmp_path) end private_class_method :extract_offline + # Runs over the same per-country .osm.pbf files sources above already + # downloaded/cached for offline.pmtiles itself — no separate download. + # Best-effort: a hiking-trails extraction problem (a malformed PBF + # region, osmium hiccup on one particular file) shouldn't fail the + # whole offline.pmtiles build over what's a bonus overlay, not the + # core map — logged into the sidecar meta's own "error" field instead, + # so it's visible without taking build_offline itself down. + def self.extract_hiking_trails(sources, data_dir) + geojson = HikingTrails.extract_merged(sources.map { |s| s["path"] }) + tmp_path = "#{hiking_trails_path(data_dir)}.tmp" + File.write(tmp_path, JSON.generate(geojson)) + File.rename(tmp_path, hiking_trails_path(data_dir)) + rescue HikingTrails::ExtractError => e + File.delete(tmp_path) if tmp_path && File.exist?(tmp_path) + warn "hiking trails extraction failed, skipping: #{e.message}" + end + private_class_method :extract_hiking_trails + + def self.hiking_trails_path(data_dir) + File.join(data_dir, "hiking_trails.geojson") + end + # Renders a report_step string like "fetching PL (3/4) — done: SK, CZ — # pending: HU" — so GET .../build-status shows not just "it's running" # but which of the current build's sources is in flight and which are @@ -442,6 +468,7 @@ module Maptool File.delete(File.join(data_dir, "offline.pmtiles")) if File.exist?(File.join(data_dir, "offline.pmtiles")) File.delete(offline_meta_path(data_dir)) if File.exist?(offline_meta_path(data_dir)) + File.delete(hiking_trails_path(data_dir)) if File.exist?(hiking_trails_path(data_dir)) cache_dir = geofabrik_cache_dir(data_dir) FileUtils.remove_entry(cache_dir) if Dir.exist?(cache_dir) end diff --git a/public/js/hiking_trails.js b/public/js/hiking_trails.js index 8ad9916..77fa30e 100644 --- a/public/js/hiking_trails.js +++ b/public/js/hiking_trails.js @@ -34,21 +34,45 @@ const SAFE_COLOR = /^(#[0-9a-fA-F]{3,8}|[a-zA-Z]+)$/; // Pre-rendered raster tiles, not vector — waymarkedtrails.org already // colors each route by its OSM `colour` tag (the KST/KČT-style paint-blaze // colors used in Slovakia/Czechia), falling back to plain red where a route -// has none, so there's no styling to do here ourselves. Online only, same -// "quietly renders nothing without a network" story as the Wikipedia/Radar/ -// Contour lines overlays — an offline copy (baked into offline.pmtiles) is -// a TODO for once there's a route planner to pair it with, see TODO.adoc. +// has none, so there's no styling to do here ourselves. Used when there's +// no local offline extraction to draw from instead (see _addOfflineLayer +// below) — same "quietly renders nothing without a network" story as the +// Wikipedia/Radar/Contour lines overlays. const TILE_URL = "https://tile.waymarkedtrails.org/hiking/{z}/{x}/{y}.png"; const ATTRIBUTION = 'Trails © Waymarked Trails (CC-BY-SA), map data © OpenStreetMap contributors'; const OPACITY = 0.85; +// Offline: a real vector line layer, GeoJSON generated at build time by +// Maptool::HikingTrails straight from the same Geofabrik .osm.pbf +// offline.pmtiles itself is built from (see lib/maptool/hiking_trails.rb, +// PmtilesBuilder.extract_offline) — no network needed at all, and no +// waiting on Overpass for click details either (queried locally via +// queryRenderedFeatures instead, see _handleClick below). +const OFFLINE_GEOJSON_URL = "/tiles/hiking_trails.geojson"; +// waymarkedtrails.org's own fallback for an untagged route. +const DEFAULT_TRAIL_COLOR = "#c1272d"; +// Real-world `colour` values seen in production data (see the 2026-09-13 +// extraction against SK.osm.pbf) — anything outside this set (typos like +// "yelllow", multi-value combos like "blue, yellow", stray hex codes) falls +// back to DEFAULT_TRAIL_COLOR rather than being trusted as a CSS value +// straight from OSM tag text. +const KNOWN_TRAIL_COLORS = new Set(["red", "yellow", "blue", "green", "orange", "purple", "brown", "black", "white", "grey", "gray", "darkblue"]); + +function normalizedTrailColor(raw) { + if (!raw) return DEFAULT_TRAIL_COLOR; + const trimmed = raw.trim(); + if (KNOWN_TRAIL_COLORS.has(trimmed.toLowerCase())) return trimmed.toLowerCase(); + return /^#[0-9a-fA-F]{3,8}$/.test(trimmed) ? trimmed : DEFAULT_TRAIL_COLOR; +} + // A plain on/off Overlays item, same shape as ContourControl — one static // raster overlay, no scrub/forecast controls to expand into. export class HikingTrailsControl { constructor(map) { this._map = map; this._active = false; + this._offlineAvailable = false; this._clickHandler = (event) => this._handleClick(event); // setStyle() (switching Online/Offline) tears down all custom sources/ @@ -59,6 +83,14 @@ export class HikingTrailsControl { }); } + // Called once map.js's GET /api/hiking-trails check resolves — same + // "prefer the local file whenever it exists" story as + // OverlaysControl#setHillshadeOfflineAvailable, independent of whether + // the Online or Offline *base map* happens to be selected right now. + setOfflineAvailable(available) { + this._offlineAvailable = available; + } + mount(li) { li.innerHTML = `