Full production-readiness pass: fixed every issue found during exhaustive
functional and restart-stability testing (admin/user/repo/fork/mirror+sync,
webhook delivery, real Gitea Actions CI job runs, paced and rapid-burst
container restarts), then brought all touched scripts into full lint
compliance. Most significant: Gitea Actions CI jobs failed 100% of the time
due to a nested-overlayfs Docker-in-Docker mount conflict, despite runners
appearing registered and online — only an actual job run surfaced it. Now
verified end-to-end across two full passes: a real workflow reaches
`status: success`, the full restart-stability suite (3x paced + 4x
rapid-burst restarts) shows identical runner UUIDs with no duplicate
registrations, and a final focused smoke test confirmed the lint-only edits
(exit-code and trap-line changes included) introduced zero regressions.
- rootfs/tmp/etc/docker/daemon.json: added `storage-driver: fuse-overlayfs`
to the baked default config. The inner (DinD) dockerd's default overlayfs
driver conflicted with the outer container's own overlay-backed root
filesystem, causing every job container creation to fail with
`failed to mount ... fstype: overlay ... err: invalid argument`.
fuse-overlayfs (already bundled in the image) mounts entirely in
userspace via FUSE, avoiding the kernel-level conflict; no new run-flag
requirement since `--privileged`/`SYS_ADMIN` already covers it.
- rootfs/usr/local/etc/docker/init.d/05-dockerd.sh: same
`storage-driver: fuse-overlayfs` fix added to both daemon.json generation
branches (with/without registry) as defense-in-depth for the case where
the baked `/etc/docker` seed directory is absent; also clears a stale
`/tmp/docker.pid` before each start attempt so dockerd doesn't refuse to
start after a restart when the PID namespace reset lets an unrelated
early-boot process reuse the old PID number; fixed a call to the
non-existent `symlink` function (missing its `__` prefix) that silently
no-op'd the `/var/lib/docker` symlink setup on every run — corrected to
`__symlink`, the actual function defined in functions/entrypoint.sh and
used consistently everywhere else in this codebase; also updated its
`su_cmd touch "$SERVICE_PID_FILE"` call site to `__su_cmd` to match the
function rename in functions/entrypoint.sh (see that entry below). Lint
pass: added `VERSION="..."` assignment and refreshed the `##@Version`
header timestamp; added `--` before 8 grep search patterns; wrapped 2
unconditional emoji echoes in the existing NO_COLOR conditional pattern.
- rootfs/usr/local/etc/docker/init.d/08-gitea.sh: updated the one
`su_cmd touch "$SERVICE_PID_FILE"` call site to `__su_cmd`, matching the
function rename in functions/entrypoint.sh (see that entry above) —
otherwise this script would call an undefined function on every service
start. Since this required touching the file anyway, brought it into the
same lint compliance as the other 5 scripts in this pass: added
`VERSION="..."` assignment and refreshed the `##@Version` header
timestamp (this script previously had neither); added `--` before 8
grep search patterns.
- rootfs/usr/local/etc/docker/init.d/zz-act_runner.sh: removed the legacy
single "gitea"-named runner registration and daemon-start blocks, which
duplicated the runner set already owned by start-runners/RUNNERS_START;
removed the second unconditional `__post_execute` invocation at the
bottom of the script that duplicated runner registration on every start;
switched RUNNER_IP_ADDRESS to loopback instead of the detected external
IP4_ADDRESS, which was transient/wrong under Docker-in-Docker networking
and caused "no route to host" registration failures; redirected the
cache-server and start-runners background job output to real log files
(instead of inheriting the __post_execute pipe) and disowned both jobs,
since a long-running background process inheriting that pipe's write end
would never let the reading `tee` see EOF, hanging __run_start_script
forever; also updated its `su_cmd touch "$SERVICE_PID_FILE"` call site to
`__su_cmd` to match the function rename in functions/entrypoint.sh (see
that entry below). Lint pass: refreshed the stale `##@Version`/`VERSION=`
timestamp since the file was substantially edited this pass; broke one
207-char line into multiple lines without changing behavior; confirmed
the 2 existing emoji echoes are already NO_COLOR-guarded.
- rootfs/usr/local/bin/start-runners: refactored the default RUNNER_LABELS
value into an array joined with a comma, avoiding one very long hardcoded
line. Lint pass: added the missing standard script header block and a
`VERSION="..."` assignment (this script previously had neither); wrapped
the fatal-error trap's emoji echo in a NO_COLOR conditional, matching the
existing pattern used elsewhere in the repo.
- rootfs/usr/local/bin/entrypoint.sh: the stale-PID-file restart guard now
also checks the live process's own cmdline for "entrypoint.sh" before
treating the recorded PID as still running, since the PID namespace
resets on every `docker restart` while `/run` persists, letting an
unrelated early-boot process coincidentally reuse the old PID number and
cause the guard to wrongly skip __start_init_scripts on a real restart.
Lint pass: added `VERSION="..."` assignment and refreshed the
`##@Version` header timestamp; added `--` before 6 grep search patterns
(including one missed on the first fix pass); replaced 2 bare `exit`
statements with explicit exit codes.
- rootfs/usr/local/etc/docker/functions/entrypoint.sh: same PID-reuse
false-positive fix applied to `__no_exit`'s monitor-loop guard, via a
`__no_exit_monitor_loop` marker embedded in the exec'd process's own
cmdline so a future restart can tell a genuine still-running monitor loop
apart from an unrelated process that reused its PID; also renamed the 6
`su_cmd()` helper definitions inside `__switch_to_user()` to `__su_cmd()`
to follow this codebase's `__`-prefix naming convention for internal
functions, and updated all 3 call sites (05-dockerd.sh, 08-gitea.sh,
zz-act_runner.sh — all `su_cmd touch "$SERVICE_PID_FILE"`) so the shared
function rename doesn't silently break any of the three init.d scripts
that source it. Lint pass: added
`VERSION="..."` assignment and refreshed the `##@Version` header
timestamp; added `--` before 15 grep search patterns; wrapped 4
unconditional emoji echoes in the existing NO_COLOR conditional pattern;
broke one 200-char line into multiple lines without changing behavior.
- rootfs/tmp/etc/gitea/app.ini: moved `ALLOWED_HOST_LIST` from the
deprecated `[webhook]` section to `[security]`, removing a
`[E] Deprecation:` warning logged on every gitea startup and admin-CLI
invocation (`[webhook].ALLOWED_HOST_LIST` is deprecated and slated for
removal in Gitea v28.0.0).
- TODO.AI.md: documented every bug found and fixed this pass, including
full verification details (exact error messages, root-cause tracing,
live confirmation steps, and regression-suite results) for each.
Brought the Dockerfile's LABEL block back in line with AI.md's OCI
label standard (lines 58-87).
- Dockerfile: removed `org.opencontainers.image.base.name` (belongs
on the base image, not the app image) and
`org.opencontainers.image.schema-version` (non-spec, redundant with
`version`); removed the duplicate `authors="${LICENSE}"` line and
the duplicate `source="https://docker.io/..."` line, replacing the
former with the correct `org.opencontainers.image.licenses="${LICENSE}"`
label and keeping a single `source` pointing at the github.com repo
- TODO.AI.md: marked this finding fixed
Added `VERSION="202608031200-git"` matching the existing `##@Version`
header, and added `--` before the pattern argument on all 15 grep
invocations in the file (the header/version mismatch and several of
the missing `--` guards were flagged by the `script-lint` agent;
completeness check found the pattern repeated across the whole file,
not just the originally-flagged subset). Also quoted the previously
bare `grep` pattern at the former line 544 (`grep -v grep` ->
`grep -v -- 'grep'`).
- rootfs/usr/local/etc/docker/init.d/zz-act_runner.sh: version stamp
+ grep -- fixes
- TODO.AI.md: marked this finding fixed
Replaced `echo "$SERVER_ADDRESS" | grep -q '://'` with the native bash
`[[ "$SERVER_ADDRESS" != *"://"* ]]` test, eliminating a useless
subshell/pipe and the missing `--` before the grep pattern that came
with it. Found by the `script-lint` agent while auditing the
act_runner cache-server integration.
- rootfs/usr/local/bin/start-runners: line 24 UUOC fix
- TODO.AI.md: marked the start-runners lint finding fixed; logged a
new, separate line-length violation on line 36 (`RUNNER_LABELS`
default is 781 chars) discovered during the same lint pass but not
yet actioned
App-breaking bug found during full runtime verification of the act_runner
cache-server feature: `/config/env` was never explicitly created. It only
came into existence as a side effect of `__create_env_file()` copying
`/usr/local/etc/docker/env/default.sample` into it — but that sample file
does not exist in this image's rootfs, so `__create_env_file()` returns
early without creating the directory. `05-dockerd.sh` and
`zz-act_runner.sh` then failed writing `/config/env/docker.local.sh` and
`/config/env/act_runner.local.sh` directly, logging
`No such file or directory` (non-fatal, but a real bug).
- rootfs/usr/local/bin/entrypoint.sh: added
`mkdir -p "/config/env" 2>/dev/null || true` alongside the other
`/config/*` directory creation lines
- TODO.AI.md: logged the fix and the upstream-template-sync follow-up
Verified: full end-to-end run with `--privileged --cap-add CHOWN
--cap-add SYS_TIME --cap-add SYS_ADMIN` (per README) shows no more
"No such file or directory" errors, gitea starts on port 80, dockerd
starts, act_runner cache server logs "Cache server has been started and
is listening on 44015", and both configured runners register
successfully ("Runner registered successfully." x2, RUNNERS_START=2).
`runners-cache.yaml` is generated with real (non-REPLACE_) cache config
values.
App-breaking bug found during full runtime verification of the act_runner
cache-server feature: gitea failed to start with `Command error: unknown
command: /config/gitea/app.ini` because its `--port` argument was empty.
Root cause: `__format_variables()` ran `printf '%s\n' $input | sort -Ru |
tr '\n' ' '` unconditionally. When `$input` word-splits to zero words
(whitespace-only, e.g. no port env vars set), `printf` with a `%s` format
still runs once with a missing arg, emitting a blank line — so the
function returned a single space `" "` instead of empty. That made
`ENV_PORTS` / `WEB_SERVER_PORTS` resolve to `" "`, which made
`SERVICE_PORT` in `08-gitea.sh` become `" "` — non-empty per `[ -n ... ]`
but rendering as an empty `--port` value to `gitea web`.
- rootfs/usr/local/etc/docker/functions/entrypoint.sh: replaced
`[ -z "$input" ] && return 0` with
`[[ "$input" =~ [^[:space:]] ]] || return 0` so whitespace-only input
is treated as empty before reaching the `printf` pipeline
- TODO.AI.md: logged the fix and the upstream-template-sync follow-up
Verified: rebuilt the image and confirmed `gitea will be running on port
80` / `gitea web --port 80 ...` in a full container run (previously
`--port` with no value).
App-breaking bug found during full runtime verification of the act_runner
cache-server feature: `__random_password()`'s `tr | head -c` pipeline
could be killed by SIGPIPE once `head -c` closes its input early, and
under `set -eo pipefail` this aborted the whole script.
- rootfs/usr/local/etc/docker/functions/entrypoint.sh: wrapped the
`tr -dc ... | head -c...` pipeline in `{ ... } || true` so a SIGPIPE
from `head` closing early no longer aborts the script
- TODO.AI.md: logged the fix and the upstream-template-sync follow-up
App-breaking bug found during full runtime verification of the act_runner
cache-server feature: the container died immediately on every startup,
printing only the initial banner line, with no error message.
Root cause: 26 occurrences of `[ "$DEBUGGER" = "on" ] && echo/printf/
__service_banner "..."` used as a bare (non-if-guarded) statement. Under
`set -eo pipefail`, when `$DEBUGGER` is not "on" (the default), the test
fails and the statement's exit status is nonzero, aborting the whole
script silently. Confirmed via `bash -x` trace pinpointing the exact
crash line.
- rootfs/usr/local/etc/docker/functions/entrypoint.sh: appended `|| true`
to all 26 occurrences of the pattern
- TODO.AI.md: logged the fix and the upstream-template-sync follow-up
(functions/entrypoint.sh is normally regenerated from casjay-dotfiles,
not hand-edited)
Turned on the act_runner Actions cache instead of leaving it disabled.
The standalone `cache-server` process was already being launched
unconditionally by zz-act_runner.sh but had no `external_secret`, so it
was refusing to start (the binary requires a non-empty secret even
though its own `cache.enabled` field is unused). Wired a shared,
randomly generated secret through the existing REPLACE_* templating
pattern so the "gitea" runner, the cache-server, and the extra
runner-N daemons spawned by start-runners all share one cache backend.
- rootfs/tmp/etc/act_runner/default_config.yaml: cache.enabled: true;
added external_server (pointed at the local cache-server) and
external_secret placeholders
- rootfs/tmp/etc/act_runner/cache_server.yaml: cache.enabled: true;
added external_secret placeholder (required for the binary to start)
- rootfs/usr/local/etc/docker/init.d/zz-act_runner.sh: added
__gen_cache_secret() (mirrors __gen_auth_token's persisted-token
pattern, stored at $CONF_DIR/tokens/cache_secret); substitutes
REPLACE_RUNNER_CACHE_SECRET into both the runner config and the
cache-server config; exports RUNNER_CACHE_HOST/PORT/SECRET for
start-runners; bumped version stamp
- rootfs/usr/local/bin/start-runners: generates a shared
runners-cache.yaml from the exported cache env vars and passes
--config to both `act_runner register` and `act_runner daemon` for
every runner-N instance, so they use the same external cache server
instead of an unshared per-process local cache
- TODO.AI.md: logged pre-existing script-lint findings (missing `--`
before grep queries throughout zz-act_runner.sh, a UUOC in
start-runners, an unpinned/stale docker.yaml CI workflow, and
forbidden OCI labels in the Dockerfile) surfaced incidentally by the
lint pass for this change but out of scope for it
casjay
