Support a persistent, private plugins directory alongside the built-in ones

plugins/ ships baked into the image and gets wiped/replaced by the
checkout sync on every deploy (docs/deployment.adoc), so a private
plugin dropped there — the exact use case the plugin contract was
designed for — didn't survive a redeploy the way uploaded GPX/kiwix
data under data/ does.

Plugins.list/run now accept either one directory or an array, and
app.rb checks the built-in plugins/ plus data/plugins/ (already
covered by the existing ./data bind mount, no new volume needed).

app.rb

@@ -67,11 +67,21 @@ module Maptool
     # Not under DATA_DIR: unlike everything else there (uploads, PMTiles,
     # the kiwix ZIM), a plugin is code, not local data — plugins/overpass-pois
     # ships checked into this repo as the reference example of the
-    # contract. A private/scraper plugin just lives in its own directory
+    # contract. Overridable so tests can point it at a throwaway fixture
-    # here too, simply never `git add`ed, no special-casing needed.
+    # directory. In production this whole directory is baked into the
-    # Overridable so tests can point it at a throwaway fixture directory.
+    # image (see Dockerfile's `COPY . .`) and wiped/replaced on every
+    # deploy along with the rest of the checkout (docs/deployment.adoc) —
+    # fine for a plugin checked into this repo, not for a private one.
     PLUGINS_DIR = ENV.fetch("MAPTOOL_PLUGINS_DIR") { File.join(__dir__, "plugins") }
 
+    # A second, optional plugin directory under DATA_DIR — the same
+    # persistent bind mount as gpx/kiwix (docs/deployment.adoc), so a
+    # private/scraper plugin (one whose ToS forbids living in this public
+    # repo) dropped in data/plugins/ survives a redeploy the way PLUGINS_DIR
+    # above can't. Checked after PLUGINS_DIR (see Plugins.list/run), so a
+    # same-named private plugin never shadows the built-in one.
+    PLUGINS_DIRS = [PLUGINS_DIR, File.join(DATA_DIR, "plugins")].freeze
+
     # Shared secret for the Garmin LiveTrack webhook (a Google Apps Script
     # watching Gmail for LiveTrack emails, see docs/deployment.adoc) — not a
     # Gitea-authenticated user, so it can't use the OAuth2 login flow above.
@@ -342,7 +352,7 @@ module Maptool
     # for the contract) — read-only, doesn't need require_login! the way
     # actually running one does below.
     get "/api/plugins" do
-      json Plugins.list(PLUGINS_DIR)
+      json Plugins.list(PLUGINS_DIRS)
     end
 
     # Runs one plugin for the given bbox (the current map viewport, from
@@ -358,7 +368,7 @@ module Maptool
 
       south, west, north, east = bbox.map(&:to_f)
       begin
-        geojson = Plugins.run(PLUGINS_DIR, params[:name], [south, west, north, east])
+        geojson = Plugins.run(PLUGINS_DIRS, params[:name], [south, west, north, east])
       rescue Plugins::RunError => e
         halt 502, json(error: e.message)
       end

docker-compose.yaml

@@ -45,7 +45,9 @@ services:
       # Uploaded GPX files and the offline PMTiles extract — kept outside
       # the image so a rebuild/redeploy doesn't wipe them (see
       # docs/development.adoc for how the deploy job treats this
-      # directory).
+      # directory). A private plugin (app.rb's PLUGINS_DIRS) also belongs
+      # under here, at data/plugins/<name>/ — nothing further to mount,
+      # this one bind already covers the whole data/ tree.
       - ./data:/app/data
 
   # Fakes a GNSS fix (a loop around Bratislava, see

docs/deployment.adoc

@@ -50,6 +50,11 @@ expect this to flip the other way, `/srv/docker` symlinked to
   files, `offline.pmtiles`, `terrain.pmtiles`, and
   `kiwix/wikipedia.zim` (see `docs/development.adoc` for each).
   Persisted via that bind mount, so a redeploy never touches it.
+  Also where a private import plugin belongs (`data/plugins/<name>/`,
+  see `lib/maptool/plugins.rb`'s `PLUGINS_DIRS`) — `build/plugins/`
+  only holds what's checked into this public repo, and gets wiped and
+  replaced by the checkout sync on every deploy same as the rest of
+  `build/`.
 * `.env` — regenerated every deploy (see below), lives here rather
   than in `build/` for the same reason as the symlink: `docker
   compose`/`podman compose` auto-load `.env` from the compose file's

lib/maptool/plugins.rb

@@ -4,7 +4,7 @@ require "open3"
 module Maptool
   # Runs an external "importer" script that turns a bbox into a GeoJSON
   # FeatureCollection, and lists what's available. Each plugin is a
-  # directory under PLUGINS_DIR with a plugin.json manifest ({"label":
+  # directory under one of `dirs` with a plugin.json manifest ({"label":
   # "...", "command": "./run.rb"}) plus whatever script it names — the
   # entire interface a plugin needs to know about maptool is "read
   # --south/--west/--north/--east, print GeoJSON to stdout". Kept this thin
@@ -12,6 +12,13 @@ module Maptool
   # forbids that live in its own private repo, dropped into plugins/
   # locally, never touching this (public) one — see plugins/overpass-pois
   # for a real, in-repo example of the contract itself.
+  #
+  # `dirs` takes either one directory or an array of them (wrapped with
+  # Array() below) — app.rb passes both the built-in plugins/ (baked into
+  # the image, wiped and replaced on every deploy along with the rest of
+  # the checkout) and an optional second directory under the persistent
+  # data/ bind mount, so a private plugin dropped there survives a
+  # redeploy the same way an uploaded GPX file does.
   module Plugins
     class RunError < StandardError; end
 
@@ -21,27 +28,32 @@ module Maptool
     # filesystem, same spirit as GpxStore's own path-escape guard.
     NAME_PATTERN = /\A[A-Za-z0-9_-]+\z/
 
-    def self.list(dir)
+    def self.list(dirs)
-      Dir.glob(File.join(dir, "*", "plugin.json")).filter_map do |manifest_path|
+      Array(dirs).flat_map do |dir|
-        name = File.basename(File.dirname(manifest_path))
+        Dir.glob(File.join(dir, "*", "plugin.json")).filter_map do |manifest_path|
-        manifest = JSON.parse(File.read(manifest_path))
+          name = File.basename(File.dirname(manifest_path))
-        { "name" => name, "label" => manifest["label"] || name }
+          manifest = JSON.parse(File.read(manifest_path))
-      rescue JSON::ParserError
+          { "name" => name, "label" => manifest["label"] || name }
-        nil
+        rescue JSON::ParserError
-      end.sort_by { |p| p["label"] }
+          nil
+        end
+      end.uniq { |plugin| plugin["name"] }.sort_by { |p| p["label"] }
     end
 
     # bbox: [south, west, north, east]. Raises RunError with a short
     # message on anything that isn't a clean run — an unknown plugin, a
     # non-zero exit, or output that isn't a GeoJSON FeatureCollection — so
     # the route can turn it into a single error response without knowing
-    # any of these details itself.
+    # any of these details itself. When the same name exists under more
-    def self.run(dir, name, bbox)
+    # than one of `dirs`, the first one that has it wins — same precedence
+    # as list's `uniq` above.
+    def self.run(dirs, name, bbox)
       raise RunError, "unknown plugin" unless name =~ NAME_PATTERN
 
-      plugin_dir = File.join(dir, name)
+      plugin_dir = Array(dirs).map { |dir| File.join(dir, name) }.find { |dir| File.file?(File.join(dir, "plugin.json")) }
+      raise RunError, "unknown plugin" unless plugin_dir
+
       manifest_path = File.join(plugin_dir, "plugin.json")
-      raise RunError, "unknown plugin" unless File.file?(manifest_path)
 
       command = JSON.parse(File.read(manifest_path))["command"]
       raise RunError, "plugin has no command" unless command

test/test_plugins.rb

@@ -52,4 +52,38 @@ class PluginsTest < Minitest::Test
     File.chmod(0o755, File.join(@plugin_dir, "run.sh"))
     assert_raises(Maptool::Plugins::RunError) { Maptool::Plugins.run(@dir, "hello", [1, 2, 3, 4]) }
   end
+
+  # Multiple dirs (app.rb's PLUGINS_DIRS: the built-in plugins/ plus an
+  # optional private one under data/plugins/) — list/run both accept an
+  # array, not just the single-dir form the tests above exercise.
+
+  def test_list_merges_plugins_across_several_dirs
+    other_dir = Dir.mktmpdir("maptool-plugins-test-2")
+    other_plugin_dir = File.join(other_dir, "private")
+    FileUtils.mkdir_p(other_plugin_dir)
+    File.write(File.join(other_plugin_dir, "plugin.json"), JSON.generate(label: "Private", command: "./run.sh"))
+
+    plugins = Maptool::Plugins.list([@dir, other_dir])
+
+    assert_equal [{ "name" => "hello", "label" => "Hello" }, { "name" => "private", "label" => "Private" }], plugins
+  ensure
+    FileUtils.remove_entry(other_dir)
+  end
+
+  def test_list_prefers_the_first_dir_when_the_same_name_exists_in_both
+    other_dir = Dir.mktmpdir("maptool-plugins-test-2")
+    FileUtils.mkdir_p(File.join(other_dir, "hello"))
+    File.write(File.join(other_dir, "hello", "plugin.json"), JSON.generate(label: "Shadowed", command: "./run.sh"))
+
+    plugins = Maptool::Plugins.list([@dir, other_dir])
+
+    assert_equal [{ "name" => "hello", "label" => "Hello" }], plugins
+  ensure
+    FileUtils.remove_entry(other_dir)
+  end
+
+  def test_run_finds_a_plugin_in_a_later_dir_when_earlier_dirs_do_not_have_it
+    result = Maptool::Plugins.run([Dir.mktmpdir, @dir], "hello", [1, 2, 3, 4])
+    assert_equal({ "type" => "FeatureCollection", "features" => [] }, result)
+  end
 end