forgejo/TODO.AI.md

TODO.AI.md

App-breaking bug fixed — sshd_config AllowUsers mismatch (gitea→forgejo migration)

Found incidentally while sweeping for remaining "gitea" references during the forgejo rename.

• rootfs/tmp/etc/ssh/sshd_config: AllowUsers gitea referenced a system account that is never created — 08-forgejo.sh sets SERVICE_USER="git", and no script anywhere creates a gitea user. This bug predates the rename (the same mismatch existed in the gitea version) and would have silently blocked all git-over-SSH logins. Fixed to AllowUsers git.

Lint cleanup done — UUOC fixed (start-runners)

Verified clean by script-lint agent after fix.

• rootfs/usr/local/bin/start-runners: line 24 UUOC (echo | grep -q '://') replaced with [[ "$SERVER_ADDRESS" != *"://"* ]]; grep call removed entirely so the missing -- no longer applies.

Lint cleanup done — version stamp and grep -- fixed (zz-act_runner.sh)

Verified clean by script-lint agent after fix.

• rootfs/usr/local/etc/docker/init.d/zz-act_runner.sh: added VERSION="202608031200-git" matching the existing ##@Version header; added -- before the pattern argument on all 15 grep invocations in the file (not just the subset originally enumerated); quoted the bare grep pattern at the former line 544 (now grep -v -- 'grep').

Lint cleanup done — line-length violation fixed (start-runners)

• rootfs/usr/local/bin/start-runners: the 781-char RUNNER_LABELS="${RUNNER_LABELS:-...}" default literal was replaced with a _default_runner_labels array joined via IFS=,, only applied when RUNNER_LABELS is unset. Verified with bash -n and a line-length scan (no line exceeds 180 chars).

App-breaking bug fixed — DEBUGGER guard pattern under set -e (functions/entrypoint.sh)

Needs syncing back to the upstream template in casjay-dotfiles/scripts per the Docker Template Update Runbook in AI.md — functions/entrypoint.sh is normally regenerated, not hand-edited.

• 26x occurrences of [ "$DEBUGGER" = "on" ] && echo/printf/__service_banner "..." used as a bare statement: under set -e, this aborts the whole script silently whenever $DEBUGGER != "on" (the default). This was the root cause of the container dying immediately after printing only the startup banner. Fixed by appending || true to all 26 occurrences.

App-breaking bug fixed — __random_password() SIGPIPE (functions/entrypoint.sh)

Needs syncing back to the upstream template in casjay-dotfiles/scripts per the Docker Template Update Runbook in AI.md — functions/entrypoint.sh is normally regenerated, not hand-edited.

• __random_password() (~line 333): tr | head -c pipeline died under set -eo pipefail on SIGPIPE. Fixed by wrapping in { ... } || true.

App-breaking bug fixed — __format_variables() whitespace-only input (functions/entrypoint.sh)

Needs syncing back to the upstream template per AI.md's runbook.

• __format_variables() (~line 187): printf '%s\n' $input | sort -Ru | tr '\n' ' ' always emits at least one line even when $input word-splits to zero words (whitespace-only), because printf with a format containing %s runs once even with no args. This made ENV_PORTS / WEB_SERVER_PORTS resolve to a single space " " instead of empty when no port env vars were set, which made SERVICE_PORT in 08-gitea.sh become " " — passing the -n test but rendering as an empty --port arg to gitea web, which broke gitea's CLI argument parsing entirely (Command error: unknown command: /config/gitea/app.ini). Fixed by replacing the [ -z "$input" ] check with [[ "$input" =~ [^[:space:]] ]] || return 0.

App-breaking bug fixed — missing /config/env directory (bin/entrypoint.sh)

• /config/env directory was never explicitly created. It only came into existence as a side effect of __create_env_file() (functions/entrypoint.sh) copying /usr/local/etc/docker/env/default.sample into it — but that sample file/dir does not exist in this image's rootfs, so __create_env_file() returns early (line 960) without creating the directory. 05-dockerd.sh's __create_service_env() then fails writing /config/env/docker.local.sh directly (cat <<'EOF' >"/config/env/....local.sh", no tee suppression) with No such file or directory; zz-act_runner.sh hits the same error writing /config/env/act_runner.local.sh. Fixed by adding mkdir -p "/config/env" 2>/dev/null || true alongside the other /config/* directory creation lines (~line 241) in rootfs/usr/local/bin/entrypoint.sh. Needs syncing to the upstream template per AI.md's runbook.

OCI label cleanup done — forbidden labels removed from Dockerfile

• Removed org.opencontainers.image.base.name (belongs on the base image, not the app image) and org.opencontainers.image.schema-version (non-spec, redundant with version).
• Removed the duplicate org.opencontainers.image.authors="${LICENSE}" line and duplicate org.opencontainers.image.source="https://docker.io/..." line; the license value now correctly populates the (previously missing) org.opencontainers.image.licenses label per AI.md's OCI label standard (lines 58-87), and source keeps the single github.com URL.

App-breaking bug fixed — act_runner init.d hang + duplicate runner registration (zz-act_runner.sh)

Needs syncing back to the upstream template in casjay-dotfiles/scripts per the Docker Template Update Runbook in AI.md — the outer hooks (__post_execute, __run_start_script) are normally generated, not hand-edited, but this repo's app-specific runner logic already lives inline in them rather than in the *_local() stubs, so the fix was applied in place.

• Root cause of the hang: because EXEC_CMD_BIN='' for this service, __run_start_script (called at the script's tail) internally calls __post_execute synchronously through __post_execute | tee -p -a "/data/logs/init.txt". Inside __post_execute's own backgrounded subshell, /usr/local/bin/start-runners & was launched without redirecting its stdout/stderr, so it (and the long-running act_runner daemon processes it execs, which never exit) inherited the write end of that tee pipe. Since that write end never closed, tee never saw EOF, so it never exited, so __run_start_script never returned, so __start_init_scripts's ( source "$init" ) for this script never completed — the init.d loop hung forever after all services were actually up, and /data/logs/start.log's final printf (the completion marker) was never reached. Fixed by redirecting start-runners's stdout/stderr to "$LOG_DIR/runners.log" at its invocation site instead of inheriting the pipe.
• The script also called __post_execute a second time explicitly (__post_execute ... | tee ... & near the end of the script), duplicating the call __run_start_script already makes via its empty-EXEC_CMD_BIN branch. Removed the redundant explicit call.
• The script additionally registered and started a hardcoded, always-on single runner named "gitea" (in __run_pre_execute_checks and __post_execute) independent of and in addition to the RUNNERS_START-driven runners from start-runners — so RUNNERS_START=2 produced 3 registered runners (gitea, runner-1, runner-2) instead of 2. Removed both the legacy registration block and its matching daemon-start block; start-runners (already idempotent per-runner via its .runner-file check) is now the sole runner registration/startup path.
• Verified with bash -n and a live rebuild/retest: the init.d loop now completes and /data/logs/start.log gets its completion marker; only runner-1/runner-2 are registered (no gitea runner).

App-breaking bug fixed — act_runner registered against wrong/unreachable IP (zz-act_runner.sh)

Needs syncing back to the upstream template per AI.md's runbook (same file/hooks as above).

• RUNNER_IP_ADDRESS defaulted to $IP4_ADDRESS (the container's externally-detected IP, captured once very early in bin/entrypoint.sh via __get_ip4). In live testing this value did not match the container's actual eth0 address by the time act_runner daemon tried to connect, causing every runner to fail its first RPC with dial tcp <ip>:80: connect: no route to host and exit immediately (act_runner does not retry a failed initial connection). Since act_runner registers against gitea running in the very same container/network namespace, the detected external IP was never the right thing to use. Fixed by defaulting RUNNER_IP_ADDRESS to 127.0.0.1 instead (still overridable via the RUNNER_IP_ADDRESS env var).
• Verified live: after the fix, curl http://127.0.0.1/ inside the container succeeds immediately. A first retest still showed runners connecting to 172.17.0.2:80, but this was a false alarm caused by stale test volumes: a previous test run's .runner registration files (with the old baked-in address) were still present, and __register_runner's idempotency check correctly skipped re-registering over them. After fully removing the test volumes and starting a genuinely fresh container, runner registration and connectivity succeeded (Runner registered successfully. / declare successfully for both runner-1 and runner-2, no connection errors).

App-breaking bug fixed — cache-server fd leak hangs init.d loop (zz-act_runner.sh)

Needs syncing back to the upstream template per AI.md's runbook (same file/hooks as above).

• Same fd-leak class as the start-runners hang fixed above, but for act_runner cache-server: act_runner cache-server --config "$CACHE_CONFIG_FILE" 2>>/dev/stderr >>"$CACHE_LOG_FILE" & used 2>>/dev/stderr, which duplicates the process's current stderr fd — at that point in __post_execute's backgrounded subshell, still the write end of the tee -p -a "/data/logs/init.txt" pipe used by the synchronous __post_execute call in __run_start_script. Since cache-server is a long-running daemon that never exits, that pipe's write end never closed, so tee never saw EOF, hanging __run_start_script (and therefore the whole init.d loop) forever even though dockerd/gitea/act_runner/cache-server were all actually up and working. Fixed by redirecting to "$CACHE_LOG_FILE" 2>&1 instead of /dev/stderr.
• The redirect fix alone did not fully resolve the hang: the subshell launching start-runners/cache-server (the left side of __post_execute's pipe to tee -p -a /data/logs/init.txt) was still observed blocked in do_wait on its long-running child even with output redirected to a real file, preventing the pipe from ever seeing EOF. Added an explicit disown "$!" immediately after each background launch (act_runner cache-server and /usr/local/bin/start-runners) to fully detach them from the subshell's job table.
• Verified live on a genuinely fresh container (volumes removed first): /data/logs/start.log now gets its completion marker, no orphaned tee -p -a /data/logs/init.txt processes remain, and both runner-1/runner-2 register and declare successfully against gitea (runners.log shows Runner registered successfully. and declare successfully for both, no connection errors).

App-breaking bug fixed — dockerd fails to restart under rapid restart cycling (05-dockerd.sh)

Found while directly testing the user's "restart container many times" requirement with a rapid 4x-restart loop.

• docker restart reuses the same container filesystem (unlike a fresh run), so /tmp/docker.pid from the previous dockerd instance survives the restart. The PID namespace itself resets on every restart, so the low PID number written into that file (e.g. 461) can coincidentally be reused by an unrelated early-boot process in the new namespace. dockerd's own startup check saw /proc/<pid> exist and refused to start: failed to start daemon, ensure docker is not running or delete /tmp/docker.pid: process with PID 461 is still running.
• Observed impact: 05-dockerd.sh logged ❌ Service dockerd failed to start - check logs, but gitea and act_runner still reported starting successfully; the entrypoint logged ⚠️ Warning: 1 critical service(s) reported failures / ℹ️ Continuing with 1 failure(s) - container may still be functional and kept going in a degraded state, then the whole container crashed anyway roughly 2 minutes later (ExitCode=1).
• Fixed by removing /tmp/docker.pid unconditionally at the top of __run_pre_execute_checks in 05-dockerd.sh, before dockerd is started. This init script is the sole owner of the dockerd lifecycle (a separate real overlap-guard already exists via SERVICE_PID_FILE), so clearing docker's own pidfile before every start attempt is safe.
• Verified live on a rebuilt image: no failed to start daemon / stale-pidfile error appeared in the logs across either a 3x normally-paced restart loop or a 4x rapid back-to-back restart burst (see the verification note on the PID-reuse fix below — both fixes were tested together in the same runs).

App-breaking bug fixed — PID-reuse false positive kills container after restart (entrypoint.sh, functions/entrypoint.sh)

Found and fixed immediately after the dockerd stale-pidfile fix above, while retesting restart robustness on the rebuilt image. Same root bug class (a PID recorded in a file that persists across docker restart gets coincidentally reused by an unrelated process once the PID namespace resets), but hitting two different guards this time, one of which actively took down the whole container:

• functions/entrypoint.sh's __no_exit(): guarded re-entry with [ -f /run/.no_exit.pid ] && kill -0 "$no_exit_pid". After a restart, an unrelated early-boot process could reuse that old PID number, making the check wrongly believe the monitor loop was already running. __no_exit then returned 0 instead of execing the actual monitor loop, so bin/entrypoint.sh fell straight through to exit $? — the whole entrypoint process exited cleanly (ExitCode=0), killing the container roughly 2 minutes after a normal, correctly-paced restart, well after dockerd/gitea/act_runner had all logged successful starts. This is almost certainly the actual root cause behind earlier restart-loop crashes previously attributed only to the dockerd stale-pidfile bug.
• bin/entrypoint.sh's ENTRYPOINT_PID_FILE check (~line 433) has the identical hazard in the opposite direction: a false-positive "still alive" match sets START_SERVICES=no, which would silently skip __start_init_scripts entirely on a genuine restart (not observed in this test run, but reachable by the same mechanism).
• Fixed both by requiring the live PID's own /proc/<pid>/cmdline to actually match the expected process, not just kill -0 succeeding: __no_exit's exec'd monitor loop now embeds a __no_exit_monitor_loop marker comment in its own bash -c command text (visible in its own cmdline), and the re-check greps for it; ENTRYPOINT_PID_FILE's check greps the candidate PID's cmdline for entrypoint.sh.
• Verified live on a rebuilt image (fresh volumes): a 3x normally-paced restart loop (each followed by a 90s post-init settle, well past the previously-observed ~2min crash window) and a 4x rapid back-to-back restart burst (3s apart, no waiting for init between them) both completed with the container remaining running/ExitCode=0 throughout — no clean-exit crash, and no failed to start daemon ... still running dockerd error in the logs. Runner UUIDs for all 5 registered runners were identical before and after both test sequences (/config/act_runner/reg/ still holds exactly 5 directories, no duplicates), confirming the "same id, not new" requirement holds even under rapid restart cycling.

Config cleanup done — deprecated [webhook].ALLOWED_HOST_LIST moved to [security] (rootfs/tmp/etc/gitea/app.ini)

• Every gitea admin user create (and, by extension, every gitea startup) logged: [E] Deprecation: config option [webhook].ALLOWED_HOST_LIST present, please use [security].ALLOWED_HOST_LIST instead because this fallback will be/has been removed in v28.0.0.
• Root cause: the shipped app.ini template set ALLOWED_HOST_LIST = * under [webhook] only; no [security] equivalent existed, so gitea used the deprecated fallback on every run.
• Fix: added ALLOWED_HOST_LIST = * to [security] and removed it from [webhook].
• app.ini is a staged config file (rootfs/tmp/etc/gitea/app.ini → /config/gitea/app.ini), not a gen-dockerfile-generated file, so this is a normal in-repo edit, not a template/generation issue.
• Verified live on a rebuilt image: gitea admin user create no longer emits the deprecation warning, and admin login/API access still works normally after the move.
• Re-ran the full functional + restart-stability regression suite against this rebuilt image on fresh volumes: admin user creation, non-admin user creation, repo creation, fork (forker/testrepo), and GitHub mirror creation + manual mirror-sync trigger (mirror_last_sync_at moved off epoch) all passed. A 3x paced restart loop (90s settle each) and a 4x rapid back-to-back restart burst (3s apart) both left the container running/ExitCode=0 throughout, with dockerd, gitea web, and all 5 act_runner daemons (+ cache-server) alive afterward, and all 5 runner UUIDs byte-for-byte identical before and after (/config/act_runner/reg/ still exactly 5 directories, no duplicates).

App-breaking bug fixed — act_runner CI jobs cannot start: nested overlayfs mount failure (05-dockerd.sh)

• Every Gitea Actions job failed within ~13s with, both via act_runner and via a plain manual docker run inside the container: failed to mount ... fstype: overlay ... err: invalid argument. Reproduced with docker exec gitea-test docker run --rm <any image> ... — 100% failure rate, not specific to any one image or to act_runner's job-container logic.
• Root cause: the inner (DinD) dockerd defaulted to the overlayfs storage driver, same as the outer host/container's own root filesystem. Nesting an overlay2-driver dockerd inside a container whose own root is itself an overlay filesystem is a well-known Docker-in-Docker failure mode — the kernel rejects the resulting overlay-on-overlay mount with invalid argument on this kernel/host combination, even with index=off already set.
• Impact: the entire act_runner CI subsystem was non-functional — runners registered fine and showed as online, but every single job would fail immediately at the "Starting job container" step, regardless of workflow, label, or target image. This would not have been caught by container-startup/registration testing alone; it required actually running a real workflow to surface.
• Fix: added "storage-driver": "fuse-overlayfs" to both branches of the /config/docker/daemon.json generation in __run_pre_execute_checks_local (with-registry and without-registry cases). fuse-overlayfs is already bundled in the image (/usr/bin/fuse-overlayfs, /dev/fuse present) and avoids the kernel-level overlay-on-overlay conflict by mounting entirely in userspace via FUSE. The existing --privileged/SYS_ADMIN requirement (already documented in README.md as required for Docker-in-Docker) covers fuse-overlayfs's own requirements — no new run-flag requirement introduced.
• Verified live: manually patched /config/docker/daemon.json on the running container, restarted the inner dockerd, and confirmed docker info reports Storage Driver: fuse-overlayfs and docker run (both casjaysdev/alpine:latest and ubuntu:latest) now creates and starts containers successfully with no mount error, where it previously failed 100% of the time.
• Follow-up root cause found: the initial 05-dockerd.sh heredoc fix alone did not take effect on a fresh container/fresh volumes. Traced to __run_precopy's baked-/etc-seeding step: on first run it copies the whole baked /etc/docker directory (staged from rootfs/tmp/etc/docker/daemon.json via 03-files.sh) into /config/docker/ BEFORE __run_pre_execute_checks_local's [ ! -f "/config/docker/daemon.json" ] guard ever runs — so the guard always saw the file already present and the heredoc (containing the fix) never executed. Fixed by adding "storage-driver": "fuse-overlayfs" to the actual baked source file, rootfs/tmp/etc/docker/daemon.json, in addition to keeping the 05-dockerd.sh heredoc fix as defense-in-depth for any case where the baked /etc/docker dir is absent.
• Rebuilt the image and recreated the container on completely fresh volumes: confirmed /config/docker/daemon.json now contains "storage-driver": "fuse-overlayfs" and docker info reports Storage Driver: fuse-overlayfs via the normal init path (not a manual patch). Confirmed docker exec gitea-test docker run --rm ubuntu:latest echo ... succeeds with no mount error. Pushed a real runs-on: alpine Gitea Actions workflow via the contents API and confirmed the job reached "status":"success" in ~3 seconds via GET .../actions/tasks — full end-to-end confirmation, not just container-creation succeeding. This bug is now fully resolved and verified.
• Re-ran the full restart-stability + functional regression suite on this fix, on the same fresh container: 3x paced restarts (90s settle each) + 4x rapid-burst restarts (3s apart) — container stayed running/ExitCode:0/healthy throughout, all 5 runner UUIDs identical before/after (no duplicate registrations), and Storage Driver: fuse-overlayfs + prior admin user/repo state all survived the restarts intact. Also verified admin user creation, repo creation, fork (into a second forker user account), and mirror+sync (mirror_updated timestamp advanced from mirror-registration time to a fresh sync time) all still work with no regressions from this fix.

Non-issue — confirmed intentional (.gitea/workflows/docker.yaml)

• Uses a stale/unpinned action pattern (@v2-@v4, DockerHub-only, catthehacker/ubuntu:act-latest). AI.md PART 7 explicitly documents this as the legacy hand-crafted workflow: "Never overwrite it, and never use it as a template for new work — it uses tag-pinned actions and retired secret names. All new/updated workflows come from gen-dockerfile actions." No build.yml exists yet in this repo; generating one is a separate task (running gen-dockerfile actions), not a fix to this file.