Prototype: extract offline hiking trails from the Geofabrik PBF we already have

No custom Planetiler profile needed after all — Maptool::HikingTrails
runs osmium tags-filter r/route=hiking against the same .osm.pbf
PmtilesBuilder already downloads per country for offline.pmtiles, then
osmium cat -f opl (locations_on_ways=true) to get member-way geometry
back as plain text, no separate node-id lookup table to build.

No general OSM tool stitches a route relation into one line (routes
are a semantic grouping, not a geometry type) — parse_opl does that
walk itself, same as the Overpass-based click handler already does
client-side. Kept pure/testable apart from the two osmium subprocess
calls, mirroring OverpassPois.fetch vs .to_geojson.

rake hiking:trails[pbf,out] to try it by hand against a real cached
extract. Not wired into the actual offline build or rendering yet —
still just a prototype, per TODO.adoc.

Rakefile

@@ -24,6 +24,22 @@ namespace :overpass do
   end
 end
 
+namespace :hiking do
+  desc "Extract route=hiking relations (name/colour/network/operator, full geometry) from a local " \
+       ".osm.pbf (e.g. one of PmtilesBuilder's own data/geofabrik_cache/*.osm.pbf) and write GeoJSON. " \
+       "Prototype — see lib/maptool/hiking_trails.rb. Usage: rake hiking:trails[input.osm.pbf,out.geojson]"
+  task :trails, %i[pbf out] do |_t, args|
+    %i[pbf out].each { |key| abort("usage: rake hiking:trails[input.osm.pbf,out.geojson]") unless args[key] }
+
+    require_relative "lib/maptool/hiking_trails"
+
+    puts "Extracting hiking routes from #{args[:pbf]}..."
+    geojson = Maptool::HikingTrails.extract(args[:pbf])
+    File.write(args[:out], JSON.generate(geojson))
+    puts "Wrote #{geojson["features"].size} route(s) to #{args[:out]}"
+  end
+end
+
 namespace :gpx do
   desc "Backfill properties.id on Note-category features saved before edit/delete (and its id targeting) existed"
   task :backfill_note_ids do

TODO.adoc

@@ -12,7 +12,8 @@ independence from connectivity.
 * [x] Shaded relief / DEM layer — hillshade toggle in the sidebar, sourced from public Terrarium tiles online; reads `data/terrain.pmtiles` for offline if one's been generated (see docs/development.adoc), otherwise the toggle is online-only for now
 ** [ ] Terrain slope readout for parking (numeric, not just visual shading)
 * [ ] Contour lines from OSM for hikes and mountain passes
-* [x] Hiking trail overlay — Waymarked Trails raster tiles (`public/js/hiking_trails.js`), colored per-route by the OSM `colour` tag they already render (the KST/KČT paint-blaze colors in Slovakia/Czechia, plain red elsewhere), so no styling logic needed here. Online only for now — an offline copy (baked into `offline.pmtiles`, needing a custom Planetiler profile for route relations) is deferred until there's a route planner/router to pair it with
+* [x] Hiking trail overlay — Waymarked Trails raster tiles (`public/js/hiking_trails.js`), colored per-route by the OSM `colour` tag they already render (the KST/KČT paint-blaze colors in Slovakia/Czechia, plain red elsewhere), so no styling logic needed here. Click queries a one-off Overpass `around` search (racing 3 public mirrors) for the route's name/colour/network/operator/distance, no server involved
+** [x] Prototype: an offline extraction path exists after all, no custom Planetiler profile needed — `Maptool::HikingTrails.extract` (`lib/maptool/hiking_trails.rb`, `rake hiking:trails[pbf,out]`) runs `osmium tags-filter r/route=hiking` + `osmium cat -f opl` against the *same* Geofabrik `.osm.pbf` already downloaded for `offline.pmtiles` — no new download, no new dependency. Not wired into the actual offline build/rendering yet, still deferred until there's a route planner to pair it with (see the entry above)
 * [x] Overpass/OSM scripts to extract POIs (drinking water, dump stations, LPG, public toilets, laundromats, shops, springs) — `rake overpass:pois`, see docs/development.adoc; writes a `.geojson` to drop in `data/gpx/` or upload. Also runnable from inside the app itself now, see the import plugins entry below (`plugins/overpass-pois` wraps this same query). Each category now carries a real `POI_ICONS` glyph and color (not p4n2gpx's plain red marker) plus a popup description built from opening_hours/operator/website tags
 * [x] Import plugins: a pluggable way to add more data sources (Park4Night, iOverlander, Freecampsites, ...) without their scrapers necessarily living in this (public) repo — sidebar icon (logged in only) lists whatever's under `plugins/`, "Run for this area" uses the current map viewport as the bbox, no separate area-picker needed. Contract: a `plugin.json` manifest + a script that reads `--south`/`--west`/`--north`/`--east` and prints a GeoJSON `FeatureCollection` to stdout (`lib/maptool/plugins.rb`, `GET /api/plugins`, `POST /api/plugins/:name/run`) — output saved the same way an upload would be. `plugins/overpass-pois` ships as the reference example; a private/ToS-sensitive scraper just drops its own directory in locally and is never `git add`ed
 ** [x] Park4Night — private plugin (`data/plugins/p4n2gpx/`, outside this repo per the "ToS-sensitive scraper" note above); iOverlander/Freecampsites still open, same story

lib/maptool/hiking_trails.rb

@@ -0,0 +1,160 @@
+require "open3"
+require "tmpdir"
+require "json"
+require_relative "track_stats"
+
+module Maptool
+  # Prototype: extracts route=hiking relations (name/colour/network/operator,
+  # full member-way geometry) straight out of a Geofabrik .osm.pbf — the same
+  # file PmtilesBuilder already downloads per country for offline.pmtiles, so
+  # this needs no separate download or third-party service. Two osmium
+  # subprocess steps do the heavy lifting (osmium is already a server
+  # dependency, see PmtilesBuilder/docs/development.adoc):
+  #
+  #  1. `osmium tags-filter FILE r/route=hiking -o hiking.osm.pbf` — pulls
+  #     out just the hiking relations *and* (by default, no -R) every way/
+  #     node they reference, so the output is small and self-contained.
+  #  2. `osmium cat hiking.osm.pbf -f opl,locations_on_ways=true,add_metadata=false`
+  #     — OPL is a plain-text, one-object-per-line format (see
+  #     https://osmcode.org/opl-file-format/); locations_on_ways embeds each
+  #     way's node coordinates directly in its own line, so no separate
+  #     node-id -> [lon,lat] table needs building, just two passes over text.
+  #
+  # No general OSM tool stitches a route relation's member ways into one
+  # line for you (routes are a semantic grouping, not a geometry type) —
+  # every hiking-trail renderer, including this app's own Overpass-based
+  # click handler (public/js/hiking_trails.js) and waymarkedtrails.org
+  # itself, does this same member-way walk by hand.
+  module HikingTrails
+    class ExtractError < StandardError; end
+
+    ROUTE_TAGS = %w[name description ref colour network operator].freeze
+
+    def self.osmium_bin
+      ENV.fetch("OSMIUM_BIN", "osmium")
+    end
+
+    # pbf_path: a Geofabrik-style .osm.pbf, e.g. one of
+    # PmtilesBuilder.cached_extracts' own files. Returns a GeoJSON
+    # FeatureCollection (MultiLineString per relation, one sub-line per
+    # member way — not one big concatenated LineString, since a relation's
+    # ways aren't always contiguous/ordered and jumping across a gap would
+    # draw a spurious straight connector).
+    def self.extract(pbf_path)
+      Dir.mktmpdir("maptool-hiking-trails") do |dir|
+        filtered = File.join(dir, "hiking.osm.pbf")
+        run!(osmium_bin, "tags-filter", pbf_path, "r/route=hiking", "-o", filtered)
+
+        opl_path = File.join(dir, "hiking.opl")
+        run!(osmium_bin, "cat", filtered, "-f", "opl,locations_on_ways=true,add_metadata=false", "-o", opl_path)
+
+        parse_opl(File.read(opl_path))
+      end
+    end
+
+    # Pure — no subprocess — so it's cheaply testable against a canned OPL
+    # fixture, same "keep the real logic testable without the real tool
+    # installed" reasoning OverpassPois.to_geojson already gives for staying
+    # separate from OverpassPois.fetch.
+    def self.parse_opl(text)
+      ways = {} # way id -> [[lon, lat], ...]
+      relations = [] # { tags:, members: [{type:, id:}, ...] }
+
+      text.each_line do |line|
+        fields = line.chomp.split(" ")
+        head = fields.first
+        next unless head
+
+        case head[0]
+        when "w"
+          ways[head[1..].to_i] = way_coordinates(fields)
+        when "r"
+          tags = tags_field(fields)
+          relations << { tags: tags, members: members_field(fields) } if tags["route"] == "hiking"
+        end
+      end
+
+      features = relations.filter_map { |relation| relation_feature(relation, ways) }
+      { "type" => "FeatureCollection", "features" => features }
+    end
+
+    def self.relation_feature(relation, ways)
+      lines = relation[:members]
+              .select { |member| member[:type] == "w" }
+              .filter_map { |member| ways[member[:id]] }
+              .select { |coords| coords.size >= 2 }
+      return nil if lines.empty?
+
+      distance = lines.sum { |coords| TrackStats.compute(coords)["distance_meters"] }
+      tags = relation[:tags]
+      properties = {
+        "name" => tags["name"] || tags["description"],
+        "distance_meters" => distance.round(1),
+      }
+      ROUTE_TAGS.each { |key| properties[key] ||= tags[key] }
+
+      {
+        "type" => "Feature",
+        "properties" => properties.compact,
+        "geometry" => { "type" => "MultiLineString", "coordinates" => lines },
+      }
+    end
+    private_class_method :relation_feature
+
+    def self.field(fields, letter)
+      fields.find { |f| f.start_with?(letter) }
+    end
+    private_class_method :field
+
+    def self.tags_field(fields)
+      raw = field(fields, "T")&.delete_prefix("T") || ""
+      return {} if raw.empty?
+
+      raw.split(",").to_h { |pair| pair.split("=", 2).map { |part| unescape(part.to_s) } }
+    end
+    private_class_method :tags_field
+
+    def self.members_field(fields)
+      raw = field(fields, "M")&.delete_prefix("M") || ""
+      return [] if raw.empty?
+
+      raw.split(",").map do |member|
+        { type: member[0], id: member[1..].split("@", 2).first.to_i }
+      end
+    end
+    private_class_method :members_field
+
+    # "n123x14.123y49.456,n124x14.130y49.460,..." -> [[14.123, 49.456], ...].
+    # A member id with no embedded location (shouldn't happen with
+    # locations_on_ways=true and -R omitted, but a way with a dangling/
+    # missing node reference is possible in real-world data) is skipped
+    # rather than raising — one bad node shouldn't drop the whole route.
+    def self.way_coordinates(fields)
+      raw = field(fields, "N")&.delete_prefix("N") || ""
+      return [] if raw.empty?
+
+      raw.split(",").filter_map do |node_ref|
+        match = node_ref.match(/\An\d+x(?<lon>-?[\d.]+)y(?<lat>-?[\d.]+)\z/)
+        [match[:lon].to_f, match[:lat].to_f] if match
+      end
+    end
+    private_class_method :way_coordinates
+
+    # OPL escapes space/newline/comma/equals/@/% as %<hex>% (see
+    # https://osmcode.org/opl-file-format/#_encoding) — only tag keys/
+    # values and usernames can contain them; the field-letter prefixes and
+    # id/comma/@ separators we split on above are never themselves escaped.
+    def self.unescape(str)
+      str.gsub(/%([0-9a-fA-F]{1,6})%/) { [Regexp.last_match(1).to_i(16)].pack("U") }
+    end
+    private_class_method :unescape
+
+    def self.run!(*args)
+      stdout, stderr, status = Open3.capture3(*args)
+      raise ExtractError, "#{args.first} exited with status #{status.exitstatus}: #{[stdout, stderr].map(&:strip).reject(&:empty?).join("\n")}" unless status.success?
+    rescue Errno::ENOENT, Errno::EACCES => e
+      raise ExtractError, "could not run '#{args.first}' (#{e.message}) — is osmium-tool installed and on PATH?"
+    end
+    private_class_method :run!
+  end
+end

test/test_hiking_trails.rb

@@ -0,0 +1,103 @@
+require_relative "test_helper"
+require "maptool/hiking_trails"
+
+# No real osmium calls — extract() is a thin subprocess-plus-parse wrapper
+# (same "not independently tested" reasoning OverpassPois.fetch gets), but
+# parse_opl is pure text in, GeoJSON out, and that's the part actually worth
+# protecting: the OPL escaping rules and the way-id -> relation-member
+# stitching are easy to get subtly wrong and osmium isn't installed in every
+# environment this test suite runs in.
+class HikingTrailsTest < Minitest::Test
+  def test_parse_opl_builds_a_multilinestring_from_member_way_geometry
+    opl = <<~OPL
+      w1 Thighway=path Nn10x20.087y49.179,n11x20.088y49.18
+      w2 Thighway=path Nn11x20.088y49.18,n12x20.089y49.181
+      r1 Troute=hiking,type=route,name=Rysy Mw1@,w2@
+    OPL
+
+    features = Maptool::HikingTrails.parse_opl(opl)["features"]
+
+    assert_equal 1, features.size
+    feature = features.first
+    assert_equal "MultiLineString", feature["geometry"]["type"]
+    assert_equal [[[20.087, 49.179], [20.088, 49.18]], [[20.088, 49.18], [20.089, 49.181]]], feature["geometry"]["coordinates"]
+    assert_equal "Rysy", feature["properties"]["name"]
+  end
+
+  def test_parse_opl_skips_relations_that_are_not_route_hiking
+    opl = <<~OPL
+      w1 Thighway=path Nn10x20.087y49.179,n11x20.088y49.18
+      r1 Ttype=multipolygon Mw1@outer
+    OPL
+
+    assert_empty Maptool::HikingTrails.parse_opl(opl)["features"]
+  end
+
+  def test_parse_opl_falls_back_to_description_when_there_is_no_name
+    opl = <<~OPL
+      w1 Thighway=path Nn10x20.087y49.179,n11x20.088y49.18
+      r1 Troute=hiking,type=route,description=Rázc.%20%pri%20%Žabom%20%potoku%20%-%20%Rysy Mw1@
+    OPL
+
+    feature = Maptool::HikingTrails.parse_opl(opl)["features"].first
+
+    assert_equal "Rázc. pri Žabom potoku - Rysy", feature["properties"]["name"]
+  end
+
+  def test_parse_opl_unescapes_commas_and_equals_in_tag_values
+    opl = <<~OPL
+      w1 Thighway=path Nn10x20.087y49.179,n11x20.088y49.18
+      r1 Troute=hiking,type=route,name=Salt%2c%%20%Pepper%3d%Co Mw1@
+    OPL
+
+    feature = Maptool::HikingTrails.parse_opl(opl)["features"].first
+
+    assert_equal "Salt, Pepper=Co", feature["properties"]["name"]
+  end
+
+  def test_parse_opl_carries_over_colour_network_operator_ref
+    opl = <<~OPL
+      w1 Thighway=path Nn10x20.087y49.179,n11x20.088y49.18
+      r1 Troute=hiking,type=route,name=Trail,colour=red,network=rwn,operator=KST,ref=0933 Mw1@
+    OPL
+
+    properties = Maptool::HikingTrails.parse_opl(opl)["features"].first["properties"]
+
+    assert_equal "red", properties["colour"]
+    assert_equal "rwn", properties["network"]
+    assert_equal "KST", properties["operator"]
+    assert_equal "0933", properties["ref"]
+  end
+
+  def test_parse_opl_computes_distance_from_member_way_geometry
+    # Roughly 111.2m per 0.001 degree of latitude at the equator.
+    opl = <<~OPL
+      w1 Thighway=path Nn10x0.0y0.0,n11x0.0y0.001
+      r1 Troute=hiking,type=route,name=Trail Mw1@
+    OPL
+
+    distance = Maptool::HikingTrails.parse_opl(opl)["features"].first["properties"]["distance_meters"]
+
+    assert_in_delta 111.2, distance, 1.0
+  end
+
+  def test_parse_opl_drops_a_relation_whose_member_ways_are_all_missing
+    opl = <<~OPL
+      r1 Troute=hiking,type=route,name=Ghost Mw999@
+    OPL
+
+    assert_empty Maptool::HikingTrails.parse_opl(opl)["features"]
+  end
+
+  def test_parse_opl_ignores_a_dangling_member_reference_but_keeps_the_rest
+    opl = <<~OPL
+      w1 Thighway=path Nn10x20.087y49.179,n11x20.088y49.18
+      r1 Troute=hiking,type=route,name=Trail Mw1@,w999@
+    OPL
+
+    features = Maptool::HikingTrails.parse_opl(opl)["features"]
+
+    assert_equal 1, features.size
+    assert_equal 1, features.first["geometry"]["coordinates"].size
+  end
+end