Test The Stack

Question

All of my code-smell checks seem to be working great now, but my infrastructure-as-code still needs some work. It's not really working well and is hard for the agent to update as it gets more complicated. Any suggestions?

Solution

Yes! Automated application testing using our lifecycle operations calls.

Explanation

Everything so far checks that code is well-formed. Nothing so far checks that apps/demo-app actually comes up, updates, and tears down cleanly - that is what appctl-tester is for. It is a skill like any other, but what makes it work is that appctl itself is portable: appctl <verb> <app> --env <name> never changes shape, whichever app <app> names, so one generic test runner can drive any of them the same way.

Each app owns its own test cases, stored under its own test/cases/:

apps/demo-app/test/cases
├── 00-full-static-qa
│   └── fixtures
├── 01-full-functional-eval
└── 02-bad-tag-fails

Source: apps/demo-app/test/cases/

Three cases, three different shapes of "does this still work" - all run the same way, by the same runner, because appctl gave them a consistent surface to test against in the first place.

Static QA

00-full-static-qa lints, formats, and doc-checks just this app's own shell surface - apps/appctl, apps/demo-app/libs/appctl-* - using the exact same three wrapper scripts from two pages ago, just pointed at this case's own throwaway fixtures/*.conf files instead of ci/'s production ones. Scoped and checked in isolation: a lint failure anywhere else in the repo never fails this app's own case, and this case never widens to cover anything outside its own scope either.

Full Lifecycle

01-full-functional-eval drives the real lifecycle against a real local docker engine: build, deploy, test, update to a second real tag, test again, destroy. "Test" here means polling the actually served page - matching both the greeting text and the image tag baked into it - not just checking that a container exists and is marked healthy. A tag can resolve, pull cleanly, and still serve the wrong content; this case is the one that would notice.

Bad Tag

02-bad-tag-fails is the negative case: it builds and deploys normally first, then rewrites the pin to a tag that cannot resolve and expects appctl build itself to refuse it - the one place a new pin is ever fetched, so that is where a bad tag has to fail. A pass here means appctl correctly refused the bad tag, the same way shell-lint's own tests prove it catches a bad script.

Checksum Proof

A real run is expensive - a real docker engine, real containers, real waiting for healthy. So a run that already happened and passed is trusted again, until something it depended on changes, via two layers of committed proof. Per case, checksum.sha256 covers that case's own test, rc, and fixtures/ - nothing elsewhere, so editing the case invalidates only that case:

ef58854b...  apps/demo-app/test/cases/01-full-functional-eval/docs/test.md
5feceb66...  apps/demo-app/test/cases/01-full-functional-eval/rc
ec7d6249...  apps/demo-app/test/cases/01-full-functional-eval/test

Source: apps/demo-app/test/cases/01-full-functional-eval/checksum.sha256

Per app, one committed log under test/logs/ covers everything outside that app's own test/ - the shared substrate every case runs against (apps/appctl, this app's own libs/) - plus the environment it ran under, ending in a trailing RESULT: PASS:

# appctl-tester log - 2026-11-06T14:32:26Z - app=demo-app
# checksums (sha256sum -c compatible, paths relative to repo root)
7b080c96...  .agents/skills/appctl-tester/scripts/appctl-tester
0a91d97e...  apps/appctl
8e6dba02...  apps/demo-app/compose.yml
...                                      (one line per tracked file)
# environment
docker:  Docker version 26.1.5
compose: Docker Compose version v2.27.0
kernel:  Linux ... 7.2.11-zen1-1-zen ...
os:      Alpine Linux unknown
# steps
2026-11-06T14:32:30Z  pass     00-full-static-qa
2026-11-06T14:32:44Z  pass     01-full-functional-eval
2026-11-06T14:32:51Z  pass     02-bad-tag-fails
RESULT: PASS

Source (rotates, newest two kept): apps/demo-app/test/logs/

An app counts as verified only if both layers check out: every tracked file in the log still matches its checksum and the log still ends in RESULT: PASS, and every one of that app's own cases still has a matching checksum.sha256. Either one being stale or missing forces a full real run of every case for that app - never a partial re-run of just the stale case, which keeps the pass/fail decision binary per app instead of a web of per-case bookkeeping.

Commit Gate

appctl-tester is just one more entry in ci/scripts/, so the pre-commit loop from the previous page calls it the exact same way it calls everything else - --githook-precommit, unconditionally. The scoping happens inside the script, same principle as every other check: whichever app(s) have a staged file under apps/<name>/ are in scope, checked against their own committed proof; an app with nothing staged is left alone entirely. Checked, never run for real from a commit hook - a stale app gets a real run right then, but only because --githook-precommit itself forces verify-first, not because a human remembered to ask for it.

CI Verify Only

In .gitlab-ci.yml, ailm-10-appctl-tester-job calls bash ci/scripts/10-appctl-tester all - no --verify flag at all. It does not need one:

# CI=true always means verify-only, regardless of which form was
# used: CI never attempts a real run, it only ever checks that
# someone already did, locally, and committed the proof.
if [[ "${CI:-}" == 'true' ]]; then
  mode='verify'
fi

Source: .agents/skills/appctl-tester/scripts/appctl-tester

GitLab sets CI=true in every job automatically, and this line overrides whatever mode the arguments alone would have selected - even a CI job's own script: forgetting to pass --verify could never accidentally trigger a real run. Stale or missing proof for any app then fails the job outright (exit 3), naming exactly which app's proof was not current, rather than silently attempting - and likely failing differently - a real run with no engine or port to use.

appctl-tester --help lays out the same three ways to call it that drive everything above:

usage: ./appctl-tester <app-name>|all [--verify]
       ./appctl-tester --githook-precommit

  <app-name>           run that one app's test cases
  all                  run every app under apps/
  --verify              check each app's committed proof first;
                        run for real only if stale
  --githook-precommit  scope is whichever app(s) have a staged
                        file under apps/<name>/; none staged
                        means nothing to do

  (no --verify) always runs for real, regardless of staleness.
  CI=true always forces verify-only - CI never attempts a real run.

Source: .agents/skills/appctl-tester/scripts/appctl-tester

• <app-name> targets one app by its directory name under apps/ - demo-app, in every example on this page.
• all targets every app found under apps/, for a single command that proves the whole repo's app surface still works.
• --verify (only after <app-name>/all) checks committed proof first, running for real only if it is stale - the Checksum Proof section above is exactly what it checks.
• --githook-precommit is its own scope, not a modifier on the other two: whichever app(s) have a staged file, verify-first, always.
• (no flags beyond <app-name>/all) always runs for real, regardless of staleness - the deliberate "just run it" form, for manual, local use.
• CI=true is checked automatically, not passed as a flag at all, and always wins over every other form above.