From a908cd7abc1ce3e49d949595d8f0581fb1117c88 Mon Sep 17 00:00:00 2001 From: Roman Priesol Date: Tue, 8 Sep 2026 21:12:54 +0200 Subject: [PATCH] Fix the real cause of slow podman deploys: add --build, drop podman rmi podman-compose@.service was missing the --build flag that docker-compose@.service already has (added on cube.local: /etc/systemd/system/podman-compose@.service), so removing maptool:latest before every restart (my own earlier fix, meant to force a rebuild) also destroyed the build cache along with it - every deploy was redoing apt-get/bundle install from scratch (~3 min measured, vs ~20s before). --build lets a plain systemctl restart rebuild while still reusing unchanged layers, same as docker did, so the ci.yml deploy step collapses back to one line. --- .gitea/workflows/ci.yml | 24 +++++++++-------------- docs/deployment.adoc | 43 +++++++++++++++++++++++++++-------------- 2 files changed, 37 insertions(+), 30 deletions(-) diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index f153e63..6679049 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -99,19 +99,13 @@ jobs: # podman-compose@maptool.service (/etc/systemd/system/podman-compose@.service) # owns the container lifecycle — its ExecStartPre already tears down - # the previous containers/networks/volumes on every start, but not - # the built image, and its ExecStart runs a plain `podman compose up` - # (deliberately no `--build`), which only builds when the image tag - # isn't already present locally. So a bare restart would just - # relaunch the previous image; removing that tag first forces a - # fresh build from the code just synced above — purely local, no - # registry involved (same `maptool:latest` docker-compose.yaml - # already builds/tags; podman rmi never talks to a registry for a - # locally-built image). sudo is needed since the unit — and thus the - # image — is root's, not gitea-runner's own (gitea-runner has - # passwordless sudo for this on cube-host). + # the previous containers/networks/volumes on every start, and its + # ExecStart runs `podman compose up --build` (mirroring + # docker-compose@.service's own `--build`), so a plain restart always + # picks up the code just synced above while still reusing unchanged + # layers (apt-get/bundle install) as cache — unlike explicitly + # removing the image first, which would delete that cache along with + # it. sudo is needed since the unit is root's, not gitea-runner's own + # (gitea-runner has passwordless sudo for this on cube-host). - name: Deploy - run: | - sudo systemctl stop podman-compose@maptool.service - sudo podman rmi -f maptool:latest || true - sudo systemctl start podman-compose@maptool.service + run: sudo systemctl restart podman-compose@maptool.service diff --git a/docs/deployment.adoc b/docs/deployment.adoc index af7a8df..a0ef67e 100644 --- a/docs/deployment.adoc +++ b/docs/deployment.adoc @@ -46,10 +46,13 @@ https://docs.gitea.com/runner/installation/binary/), running as a systemd service (`gitea-runner.service`) under a dedicated `gitea-runner` system user, which needs: -* Passwordless `sudo` (confirmed via `sudo -l -U gitea-runner`) — - `podman-compose@maptool.service` is a root-owned systemd unit (see - below), and the image it builds lives in root's Podman storage, so - starting/stopping it and removing its image both require root. +* Passwordless `sudo` for managing any `podman-compose@` instance — + `/etc/sudoers.d/gitea-runner-podman` grants `NOPASSWD` on + `systemctl {start,stop,restart} podman-compose@*.service` and + `podman rmi -f *` (root-owned systemd units, same idea as the + `docker` group membership gitea-runner already has for + `docker-compose@` services, just scoped explicitly since Podman has + no daemon/socket-group equivalent to grant blanket access through). * Write access to `/srv/podman/maptool`. Its `config.yaml` (`/etc/gitea-runner/config.yaml`) declares a single @@ -222,14 +225,24 @@ on a push to `main`, with `runs-on: cube-host`: . Copies the checkout into `/srv/podman/maptool/` using plain `cp`/`rm`: clears everything in the target except `data/`, copies the fresh checkout over, then drops the copied `.git`. -. Stops `podman-compose@maptool.service`, force-removes the - `maptool:latest` image directly via `podman rmi`, then starts the - service again. The service (`/etc/systemd/system/podman-compose@.service`, - a systemd template instantiated as `podman-compose@maptool.service`) - owns the actual container lifecycle — its `ExecStartPre` tears down - the previous containers/networks/volumes on every start, and its - `ExecStart` runs a plain `podman compose up` (deliberately no - `--build`), which only builds when the image tag isn't already - present locally. Removing the tag first is what forces a fresh build - from the code just synced above; without it, a restart alone would - just relaunch the previous image unchanged. +. `sudo systemctl restart podman-compose@maptool.service`. The service + (`/etc/systemd/system/podman-compose@.service`, a systemd template + instantiated as `podman-compose@maptool.service`) owns the actual + container lifecycle — its `ExecStartPre` tears down the previous + containers/networks/volumes on every start, and its `ExecStart` runs + `podman compose up --build` (mirroring `docker-compose@.service`'s + own `--build`), so a plain restart always rebuilds from the code + just synced above while still reusing any layer whose inputs didn't + change (base image, `apt-get install`, `bundle install` when + `Gemfile`/`Gemfile.lock` are unchanged) as cache. ++ +CAUTION: An earlier version of this step explicitly removed the +`maptool:latest` image (`podman rmi -f`) before restarting, to force a +rebuild against a service whose `ExecStart` didn't have `--build` yet. +That "worked" but also destroyed the build cache along with the image +— since nothing else referenced those layers, Podman fully deleted +them rather than leaving them as reusable cache the way Docker leaves +a retagged image's old layers behind — turning every deploy into a +full `apt-get`/`bundle install` from scratch (~3 minutes measured, +vs. ~20s before). Fixed by adding `--build` to the unit instead (see +above); don't reintroduce the `podman rmi` step.