scripts/dev.sh is the one entry point for working on nexcage on a
workstation: it checks the tools, builds, runs every test that works without a
Proxmox host, runs the GitHub-hosted CI jobs through
act, and times the lifecycle commands for
performance work. Nothing in it needs root. The make targets below call it,
except build, check and sim, which run zig and tests/sim/run.sh
directly.
scripts/dev.sh setup # once: podman socket, act's runner image, a busybox rootfs
scripts/dev.sh check # before a commit: zig fmt, unit tests, the simulator, the script tests
scripts/dev.sh all # before a pull request: check, then every CI job through act
| Command | make |
What it runs | Needs | Time here¹ |
|---|---|---|---|---|
doctor |
doctor |
Checks every tool below and says how to get what is missing | 1 s | |
setup |
dev-setup |
Starts podman’s API socket, builds act’s runner image, fetches busybox | a container engine | minutes, once: the act image is 1.7 GB |
build |
build (Debug only) |
Debug to zig-out/bin, ReleaseSafe to zig-out/release/bin |
zig 0.15.1 | seconds |
test |
check |
zig fmt --check, unit tests |
zig | 6 s |
sim |
sim |
Every command against fake pct/pvesh/pvesm (TESTING.md) |
zig, unshare, python3, tar, zstd | 12 s |
check |
test + sim + tests/scripts/run.sh, the tests of the shell scripts in scripts/ |
jq, dpkg-deb, readelf for the script tests | 20 s | |
crun |
The crun-backend image and what crun_build.yml checks on it, in its order: version, the features ABI, the features document, ps against crun, update’s flags, a Proxmox LXC container under --root, an isolation profile, --systemd-cgroup, a bundle from a foreign working directory, kill, exec’s process spec, a profile’s limits on update, memory values, the re-execution. Not the .deb, the runtime-spec suite or critest, which only act crun_build.yml:docker runs, nor the pod, which is cri |
a container engine | 3 min first, then a minute | |
cri |
A pod through containerd’s CRI with nexcage as the runtime handler: sandbox, CNI address, container, exec, update, removal |
a container engine | 5 s, plus the images | |
e2e |
e2e |
sim + crun + cri |
||
shell [CMD] |
dev-shell |
A shell with nexcage (crun backend), crun, containerd and crictl, or CMD run there | a container engine | |
act |
act |
The GitHub-hosted CI jobs, in the runner image | act, a container engine; gh logged in and deps/crun checked out for dependency_check.yml |
1.5–4 min per job |
all |
local-ci |
check, then act, whose crun_build.yml job is crun and cri as CI runs them |
12 min | |
pve-e2e |
proxmox_e2e.yml on the self-hosted Proxmox VE runner, for this branch |
gh, the branch pushed | 10 min | |
perf |
perf |
Times the lifecycle commands; see Performance | zig, python3, tar; unshare and zstd unless --suite crun; a container engine unless --suite lxc |
1 min for lxc, 4 min for both with --against |
¹ A 12-core workstation with rootless podman, warm caches.
Multi-step commands (check, e2e, act, all) run every step even after
one fails, keep each step’s output in zig-out/dev/logs/<step>.log, and end
with a summary; the exit status is 1 if any step failed.
zig version must print 0.15.1. From
https://ziglang.org/download/#release-0.15.1.~/.local/share/containers, and files that containers write into the
checkout stay yours. Docker works as well (CONTAINER_ENGINE=docker), but
act’s --bind then leaves root-owned files in the checkout.act_Linux_x86_64.tar.gz and checksums.txt from
https://github.com/nektos/act/releases, check the sum, and put act in
~/.local/bin. The curl … | sudo bash installer does the same but runs an
unreviewed script as root.scripts/dev.sh setup, then scripts/dev.sh doctor until it is all green.The simulator runs nexcage as uid 0 in an unprivileged user namespace. On
Ubuntu 24.04 and later AppArmor forbids mounts there by default;
doctor says so, and the fix is
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0.
crun, cri, shell and perf --suite crun build localhost/nexcage:ci
from the repository’s Dockerfile, as crun_build.yml builds its
nexcage:ci; the Dockerfile clones vendored crun at its pin and builds the
crun backend. With unchanged sources the engine’s layer cache makes a rebuild
take seconds. The CRI image localhost/nexcage:cri-test adds containerd,
crictl and the CNI plugins on top of it.
The tests run in a privileged container with its own cgroup namespace. Under rootless podman two things differ from CI’s rootful Docker, and the tests handle both:
cpu, memory and pids are delegated to a user’s cgroups, so
controllers are enabled one at a time; a single write naming cpuset or
io would fail as a whole and enable nothing. tests/crun/update.sh checks
--cpuset-cpus only where cpuset is in the container’s
cgroup.controllers, and prints a skip: line here.oom_score_adj and AppArmor profiles cannot be loaded.
tests/cri/pod_on_nexcage.sh sees the user namespace and sets
restrict_oom_score_adj and disable_apparmor, as containerd’s own startup
warning asks.conmon, which watches each of podman’s containers, writes an empty file named
oom into podman’s working directory when a process in the container is
killed for lack of memory, as critest makes happen. dev.sh runs containers
from zig-out/dev, so the file lands there; .gitignore names /oom for a
run by hand from the checkout.
scripts/dev.sh shell leaves you in that container with the checkout at
/src (read-only) and a busybox bundle at /bundle, its processes already
moved out of the root cgroup as the tests do — otherwise containers get no
cgroup of their own and ps fails. The bundle’s config.json is the one
crun, e2e or the crun perf suite last wrote; before any of them has run
there is none, and create fails. scripts/dev.sh shell CMD... runs CMD
there instead.
nexcage --runtime crun create --bundle /bundle demo
nexcage --runtime crun start demo && nexcage --runtime crun ps demo
nexcage --runtime crun delete --force demo
/src/tests/cri/pod_on_nexcage.sh
scripts/dev.sh act runs, by default, every job a pull request gets on a
GitHub-hosted runner except security.yml’s Semgrep, Trivy and gitleaks
jobs, which download their rules, databases and images, and the two that
build a binary only for a self-hosted job: ci.yml (build-test,
simulation, scripts), security.yml’s zizmor, version-check.yml,
crun_build.yml and memory_leak_check.yml, plus the crun job of
dependency_check.yml in its dry run. Name jobs to run others, and pass
act’s own options after --:
scripts/dev.sh act ci.yml:simulation
scripts/dev.sh act crun_build.yml:docker -- --verbose
.actrc maps ubuntu-latest, ubuntu-24.04 and ubuntu-22.04 to
localhost/nexcage-act:latest — catthehacker’s act image plus gh, built from
.github/act — turns pulling off, and adds --bind, because act’s copy of the
checkout drops the submodule’s .git file. dev.sh adds four things act
cannot know:
DOCKER_HOST pointing at podman’s socket.DOCKER_BUILDKIT=0 under podman. A job’s docker build goes to podman
through the socket, and the act image’s docker CLI then builds with buildx’s
docker-container driver, which keeps the result in its own cache and never
loads it: the job’s next docker run finds nothing, or an older image of the
same name and passes on that. The classic builder hands the build to podman.git worktree, a read-only mount of the main repository’s
.git, which the worktree’s .git file points into and which is otherwise
outside the container.crun_build.yml builds nexcage:ci, runs its checks on it and builds its
CRI and runtime-spec images FROM it: under podman, the job’s
docker build -t nexcage:ci and docker run nexcage:ci both take
localhost/nexcage:ci when that exists, and docker.io/library/nexcage:ci
otherwise. The first is the name crun, cri and shell give their
image, so the job’s build takes it over; they build theirs again before
they use it. A build FROM nexcage:ci, though, takes
docker.io/library/nexcage:ci first. With both names there, the job runs
its checks on its own build but builds its CRI image on whatever an earlier
run left as docker.io/library/nexcage:ci, and critest tests that. So
dev.sh refuses the job while one of its images has both names, and says
which to remove (podman rmi docker.io/library/nexcage:ci, most likely an
earlier act run’s), and fails it if one has both when it ends: something
else built that name while it ran. It renames and removes nothing itself.
It also runs one such job at a time: a second, from another worktree say,
waits for the first rather than building the same names in the middle of
its steps. crun, cri and shell build localhost/nexcage:ci,
:builder and :cri-test as well and take no turn, so one of them in
another checkout while such a job runs can still change what it tests:
run them one at a time.dependency_check.yml is scheduled, so dev.sh dispatches it, with
dry_run=true and the token gh auth token holds: the job reads the
repository’s open issues and the crun releases and prints what it would file,
edit or close. Without that input and with a token it does those things in
CageForge/nexcage, so keep --input dry_run=true when running act on it by
hand; without a gh login dev.sh skips the job and says so. It skips it too,
and says so, where deps/crun is not checked out — in a worktree, say — since
with --bind the workflow’s submodules: recursive checks nothing out:
git submodule update --init --depth 1 deps/crun first. The image builds
leave deps/crun out of their context (.dockerignore), since the Dockerfile
clones crun itself, so a checked-out one does not get in their way.
Two things act runs differently from GitHub’s runners:
ci.yml’s build-test fails at its last step, “Upload binary
(ReleaseSafe)”: act 0.2.89’s artifact server does not know the
mime_type field actions/upload-artifact@v7 sends, and the step fails
with “Failed to CreateArtifact: … Unexpected end of JSON input” after five
attempts. Every step before it is what CI runs; the job’s FAIL comes from
that one step.crun_build.yml’s runtime-spec suite: tests/runtime-tools/known-failures
is the list for rootful Docker, and under rootless podman five tests differ
from it, so the step fails here
(RUNTIME_SPEC_VALIDATION.md names them).Not in the default set: the jobs of proxmox_e2e.yml, k8s_e2e.yml and
buildagent.yml that need self-hosted runners, and the two GitHub-hosted ones
that only build for them, proxmox_e2e.yml’s binary and k8s_e2e.yml’s
build; and security.yml’s Semgrep, Trivy and gitleaks jobs, which
download their rules, databases and images.
The simulator shows what nexcage asks of Proxmox; only a Proxmox VE host shows
what Proxmox then does. scripts/dev.sh pve-e2e dispatches
proxmox_e2e.yml for the current branch on the self-hosted runner and follows
the run until it ends. The branch must be pushed at the commit you have
checked out; the script refuses otherwise, so the run is of the code in front
of you.
scripts/dev.sh perf # this checkout
scripts/dev.sh perf --against main # this checkout against main; exit 1 on a regression
scripts/dev.sh perf --suite lxc -n 40 # one suite, more passes
make perf PERF_ARGS='--against v0.13.0'
Both builds are ReleaseSafe: a Debug build’s allocator checks would be much of
what gets timed. --against REF builds REF in a worktree under
zig-out/dev/perf/, and the two builds run the same suite scripts — this
checkout’s — taking turns, so a change in the machine’s load lands on both:
the LXC suite runs both binaries in every pass against the same fake host,
the crun suite alternates the two images in four rounds. Each run keeps
samples.tsv, summary.json and report.txt in zig-out/dev/perf/<time>/,
linked from zig-out/dev/perf/latest.
Two suites:
lxc-sim (tests/perf/lxc_sim.sh) runs the Proxmox LXC lifecycle —
version, list with ten containers, create, state, start, exec,
kill, stop, snapshot, snapshots, rollback, delsnapshot,
delete, run, create from an 8 MB OCI bundle — against the
simulator’s fake tools. For each command:
calls: how many times it ran pct, pvesh, pvesm, pveam,
pveversion or zfs — every call the fake tools log. On a real node each
is a Perl process costing 0.3–1 s, so this is most of what a user of a real
node waits for, and it does not vary between runs. Any increase is a
regression.wall_us: nexcage’s own wall time, measured inside the namespace. The fakes
are shell scripts, so it includes a few milliseconds per call; it is for
comparing two builds on one machine, not a prediction for a Proxmox host.maxrss_kb: peak memory, from one run under GNU time.crun (tests/perf/crun.sh) runs features, create, start, state,
ps, exec, kill and delete on the crun backend inside the crun image,
and the same lifecycle through the image’s own crun binary, taking turns. The
report shows nexcage’s overhead over crun per operation. The reference is the
distribution’s crun, not the vendored libcrun nexcage links, so part of a small
difference can be the version.
tests/perf/report.py decides what is a regression. A change in time counts
only if the median rose by more than 10 % and 0.5 ms and three quarters of the
new samples are above the old median: time on a workstation is noisy, and this
asks for a shift, not a blip. Comparing main with itself this way stays
within ±3 %. An extra pct list and 5 ms put into the lookup of a container’s
init PID look like this, and the run exits 1:
suite/op main@e42238ca ms head@e42238ca+dirty ms change calls
lxc-sim/list 12.95 13.27 +2.4% 2 ok
lxc-sim/state-running 32.09 47.42 +47.8% 3 -> 4 MORE CALLS
lxc-sim/kill 21.00 36.29 +72.8% 2 -> 3 MORE CALLS
lxc-sim/stop 24.56 23.94 -2.5% 2 ok
--threshold and --min-delta-us change the bounds, and the script can be
run again on any samples.tsv:
python3 tests/perf/report.py zig-out/dev/perf/latest/samples.tsv \
--compare main@e42238c head@1a2b3c4 --threshold 5
| Path | What |
|---|---|
scripts/dev.sh |
The entry point |
.actrc, .github/act/Dockerfile |
act’s defaults and runner image |
tests/sim/lib.sh |
The fake Proxmox host, shared by the simulator and the perf suite |
tests/perf/lxc_sim.sh, tests/perf/crun.sh |
The perf suites; each writes samples |
tests/perf/report.py |
Summary, comparison, regression verdict |
tests/crun/features_check.py |
The features document check, shared with crun_build.yml |
tests/scripts/ |
Tests of the shell scripts in scripts/, against fakes of the tools they call |
zig-out/dev/ |
Logs, the busybox bundle, act’s artifacts, perf runs (make clean removes it) |
After make clean, run git worktree prune if perf --against had created a
worktree there.