nexcage

Kubernetes integration

What nexcage would need to run containers for Kubernetes, and how the work is staged in the tenant-nexcage tenant of the Cozystack cluster pskep.

Status as of 0.17.0: Kubernetes schedules pods onto nexcage.

Everything a kubelet’s containerd asks an OCI runtime is answered.

At 0.9.1 none of that was true: nexcage was a command-line lifecycle tool for LXC containers on one Proxmox VE host, and nothing in Kubernetes could schedule onto it. The way in was the one crun and runc take — containerd and CRI-O call an OCI runtime binary with the runtime-spec command line, so no daemon and no operator were needed, only that command line and the crun backend (vendored libcrun) behind it.

What is left is not the command line. It is the table below: no image service of its own, and the verbs nothing has asked for. For Kubernetes, more than one host was never a runtime question — a kubelet runs on each node and calls the binary there, the way runc is called.

What the runtime is missing

The gaps below are what separates the current CLI from something a kubelet, a containerd shim or an operator can drive. They are ordered by what blocks the next step, not by size.

# Gap Today Needed for
1 exec Done on both. Proxmox LXC runs pct exec; crun goes to libcrun_container_exec_process_file, so exec --process <file> — the shape an engine sends — works, and crictl exec runs a command in a pod. Both exit with the command’s status kubectl exec, CRI ExecSync, exec probes
2 OCI runtime-spec CLI Enough for podman and containerd to run containers. create <id> --bundle <dir>, --root <dir>, --console-socket and --pid-file work; the last two on the crun backend, refused on Proxmox LXC A containerd shim, and runc-compatible tooling generally
3 state Done. Proxmox LXC reports the bundle it recorded; crun hands the question to libcrun, so the output is crun state’s The same. A shim reads the bundle path back from state
4 Missing verbs delete --force, kill --all, features and ps are done — ps because the trace on the node showed Kubernetes asking for ps --format json and getting unknown command. pause and resume are done too — the cgroup freezer, on both backends — and update (0.13.0), because an in-place pod resize sends it. No events yet, and nothing has asked for it Pod lifecycle, metrics, cgroup updates on resize
5 No remote surface CLI only, must run as root on the PVE host Anything in a Kubernetes pod driving nexcage. A pod cannot call pct
6 No log handling The engine’s, not the runtime’s. The engine captures the container’s output to the file the kubelet reads, as it does for runc. Verified: kubectl logs and crictl logs return a nexcage container’s output Kubelet reads /var/log/pods/…/0.log; kubectl logs needs it
7 No CNI The engine’s, not the runtime’s. containerd and CRI-O create the sandbox’s network namespace, run the CNI plugins in it and hand the runtime a path to join. Verified on both: a pod gets an address from host-local and reaches another pod over TCP Pod IPs from the cluster CNI, NetworkPolicy, service routing
8 No sandbox model The engine’s, not the runtime’s. containerd and CRI-O create the pod sandbox and hand each container in it the sandbox’s namespaces to join by path in linux.namespaces. Verified: a sandbox and a container in it both run on nexcage Multi-container pods sharing a network namespace, the pause container
9 No image service Not the runtime’s job for Kubernetes: containerd and CRI-O implement ImageService themselves and hand the runtime a bundle whose rootfs is already unpacked — runc never pulls anything either. For the command-line use there is now images (what the cluster can create from, per node and storage), pull (an OCI image into a chosen storage) and rmi. Pull with credentials is not possible: oci-registry-pull has no parameter for them CRI ImageService: pull with credentials, list, remove, image FS stats
10 Fixed resources Done. A bundle’s linux.resources apply at create through libcrun, and update changes a running container’s limits: update --resources=- <id> with the document on stdin is what containerd sends for an in-place pod resize. Under an isolation profile a resize above the profile’s memory or pids cap, or to no limit, is refused naming the cap; containerd resends the pod’s declared limit with every update, the CPU manager’s cpuset updates included, so declare limits within the profile’s (CLI_REFERENCE.md). On Proxmox LXC the same settings become pct set in its terms (MiB, cores, a cgroup v2 weight), and what pct cannot express is refused by name Pod requests and limits, and resizing them in place
11 Single host Reads and lifecycle cover the cluster: list, state, start, stop, delete, update and the snapshot verbs find a container on any node and act through that node’s API. exec, kill, pause, resume and the pid in state stay local and say so. They reach into a container’s processes or its cgroup freezer from the host, and no API offers that. For the same reason, state reports a frozen container on another node as running. create --node <name> places a new one on a chosen node, from a template that node can read A node per PVE host, or a scheduler that targets more than one

One further gap is operational rather than functional: nexcage exposes no health or metrics endpoint a probe can use.

Constraints found in the pskep cluster

These decide which of the stages below can run where. All three were measured on the cluster, not assumed.

Constraint Consequence
No KubeVirt (no virtualmachines.kubevirt.io); the Cozystack app catalog here has bucket, etcd, info, ingress, monitoring, seaweedfs, tenant only A Proxmox VE test node cannot be run as a VM inside the cluster. It has to be a VM on a Proxmox host, declared through kubemox
Nodes are Talos and report user.max_user_namespaces=0 inside pods tests/sim/run.sh cannot run in a pod: it needs a user namespace with mounts. Changing it needs machine.sysctls in the Talos machine config
PodSecurity enforces baseline on tenant namespaces by default Privileged pods are rejected until the namespace is labelled otherwise
kubemox is connected to prox-home (ProxmoxConnection in tenant-proxmox, PVE 9.2.20, Ready) A VM or LXC container on that host can be declared as a Kubernetes object today

Because of the first two, the in-cluster job in deploy/kubernetes/tenant-nexcage builds and unit-tests nexcage but does not run the simulation suite; it reports whether the node would allow it, so the day the sysctl changes the job starts covering it.

Stages

flowchart TD
  S1["1. Build and test in-cluster<br/>Job in tenant-nexcage<br/>done"] --> S2["2. PVE test node<br/>kubemox VirtualMachine + E2E runner<br/>done"]
  S2 --> S3["3. OCI runtime-spec command line<br/>so a container engine can call nexcage<br/>done"]
  S3 --> S4["4. containerd and CRI-O<br/>running pods on nexcage<br/>both done"]
  S4 --> S5["5. Kubernetes scheduling a pod onto nexcage<br/>done on nexcage-e2e-1"]

Stage 1 — build and test inside the cluster

Done. deploy/kubernetes/tenant-nexcage/ holds the Tenant and a Job that fetches Zig, builds nexcage, runs the unit tests and the smoke checks in tenant-nexcage. It needs nothing from the runtime and no privileges.

What it does not cover: the simulation suite, and anything that calls pct.

Stage 2 — a Proxmox VE test node declared from Kubernetes

Done. deploy/kubernetes/tenant-nexcage/vm-e2e-node.yaml is a kubemox VirtualMachine in tenant-nexcage; kubemox clones the template nexcage-pve-tpl-v0-2 on prox-home into the node nexcage-e2e-1, a Debian 13 guest running Proxmox VE 9.2.20. pve-template/ holds the four scripts that build that template from the Debian cloud image, and scripts/register_e2e_runner.sh registers the Actions runner on the node — deliberately not in the template, because a registration belongs to one machine.

proxmox_e2e.yml now says runs-on: [self-hosted, pve9], so it lands on that node rather than on whichever host happens to hold the proxmox label, and it has a step for the registry path: nexcage run --name … docker.io/library/…, which only Proxmox VE 9.1 and later can do. On the node, by hand, the whole lifecycle works against real pct — create, start, state with the init’s host PID, kill, stop, delete — and nexcage run docker.io/library/redis:7 pulls the image through oci-registry-pull and starts it.

Building that node also turned up a defect in what nexcage ships: the release workflow built with Zig’s default native target, so the published binary was tuned to the GitHub runner’s CPU and died with SIGILL on the node’s Xeon E5-2697 v2. Both release paths now pass -Dcpu=baseline.

One node at a time: the guest’s hostname and address are baked into the template, because a Proxmox node keeps its configuration under /etc/pve/nodes/<hostname> and cannot be renamed after a clone.

Stage 3 — the OCI runtime-spec command line

Done. containerd and CRI-O do not talk to a runtime over an API: they exec a binary with the command line the runtime-spec defines, the same one runc and crun answer. Everything nexcage needs to be callable that way is CLI shape and the crun backend behind it.

exec is done, and so is the rest. In the order a container engine needs it:

  What the engine sends nexcage today
container id positional, create <id> done: with --bundle, the positional word is the id
global --root <dir> — containerd passes its own state directory done, before or after the command
state bundle filled in done, and start/stop keep it
delete --force done: shuts the container down first
kill --all accepted, and says what it really does on LXC
create --pid-file <file>, --console-socket <sock> done on crun: the pty’s master end arrives over SCM_RIGHTS and is a real tty. Refused on Proxmox LXC, which starts no process on create
also read ps, features, pause, resume, update all done: ps and features (0.10.0), pause/resume (0.12.0), update (0.13.0)

The crun backend took the bundle path from the container id — /var/lib/nexcage/bundles/<id> — and ignored the one it was given, so it looked where nothing had written. It uses the caller’s directory now, and honours --root; bundles are no longer confined to two directories, because a container engine picks its own and runc and crun accept any.

--console-socket was the first thing the crun backend asked for once it could report what libcrun says, and it is done: with the flag, create on a bundle whose spec sets process.terminal succeeds and the master end of the pty arrives on the caller’s socket —

RECEIVED_FDS=1
IS_A_TTY=True

— while the same bundle without it still answers use --console-socket with create when a terminal is used. That is the first runtime-spec create that completes on nexcage.

state followed, and it is libcrun’s own: the JSON is written by libcrun_container_state, so it matches crun state for the same container field for field rather than being a second rendering that can drift. Together with create that is enough for a caller to make a container and read it back:

$ nexcage --runtime crun state s1
{ "ociVersion": "1.3.0", "id": "s1", "pid": 30, "status": "created",
  "bundle": "/run/eb/s1", "rootfs": "rootfs", "created": "...",
  "systemd-scope": "", "owner": "root" }

start, kill and delete have now been run end to end on this backend for the first time: create --console-socket → start → "status": "running" → delete --force → gone. Two flags that were accepted and ignored now reach libcrun: delete --force, which the driver had hardcoded to false, and kill --all, which goes to libcrun_container_killall.

A container engine drives it. podman runs a container on nexcage end to end:

$ podman --runtime /usr/local/bin/nexcage run --rm localhost/tiny:1 \
      /bin/sh -c 'echo HELLO_FROM_NEXCAGE'
HELLO_FROM_NEXCAGE

# what podman sent the runtime
create --bundle /var/lib/containers/storage/overlay-containers/<id>/userdata \
       --pid-file /run/containers/.../pidfile <id>
start <id>
delete --force <id>

No unexpected options: the command line an engine sends is the one built in this stage. Two things had to change for it to work, and neither was on the list of missing verbs — which is the argument for running an engine against it rather than working down a checklist:

What was still missing at that point: ps, pause, resume and update did not exist (all four do now). podman did not ask for any of them to run a container. features and exec were also on this list and are done — the first because containerd asked at startup, the second because a kubelet cannot do without it.

--pid-file and --console-socket are where the Proxmox LXC backend stops being able to pretend: the runtime-spec means create to leave the container’s process alive and waiting, and pct create starts nothing, so there is no PID to write and no console to hand over. Those two belong to the crun backend, which is also where the rest of the list gets its answers.

The container work behind that command line belongs to the crun backend, which links vendored libcrun and already has create, start, kill, delete and exec. The Proxmox LXC backend stays what it is: a different surface, driven by pct, not by a container engine.

How to test it without containerd. podman drives an OCI runtime with exactly this command line, and it is already installed on the build agent:

podman --runtime /usr/local/bin/nexcage run --rm docker.io/library/alpine:3 echo hi

That is a far shorter loop than standing up containerd, and a runtime podman can drive is a runtime containerd can drive.

Stage 4 — containerd and CRI-O

containerd runs a container on nexcage:

$ ctr run --rm --runc-binary /usr/local/bin/nexcage \
      docker.io/library/alpine:3.20 t2 /bin/echo HELLO_FROM_NEXCAGE
HELLO_FROM_NEXCAGE

# what containerd sent
--root /run/containerd/runc/default --log <taskdir>/log.json --log-format json \
    create --bundle <dir> --pid-file <file> t2
… start t2
… delete t2
… delete --force t2

It took three rounds to get there, and each blocker was only visible once the one before it was gone — none of them was in the table above, which was predicting ps and features that containerd never asked for:

  1. unknown command '--log'. containerd puts --log <file> and --log-format json before the command; the loop that finds the command name did not skip them, so --log was taken for the command.
  2. open …/log.json: no such file. --log is not an option to swallow: containerd opens that file to find out why the runtime failed. nexcage writes its log there now, and --log-format json writes the shape runc writes.
  3. Time.UnmarshalJSON: input is not a JSON string. The time field was a number; Go’s time.Time wants RFC 3339. src/core/rfc3339.zig formats it.

Then a fourth, which podman had hidden: an OCI bundle’s config.json shadowed nexcage’s own. ./config.json was first in the configuration search path, a bundle holds a config.json that is a runtime spec, and containerd runs the runtime from the bundle directory — so the routing rules were replaced by a file that is not a configuration at all, silently. A file declaring ociVersion is skipped in the search path now, and refused outright when named with --config; and after 0.17.0 the working directory is not searched at all.

A CRI pod runs on nexcage. The runtime handler is the one a kubelet would name, and crictl drives the same interface the kubelet drives:

# /etc/containerd/config.toml
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nexcage]
  runtime_type = 'io.containerd.runc.v2'
  [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nexcage.options]
    BinaryName = '/usr/local/bin/nexcage'
    # As runc's handler has it. This run was in a container without systemd,
    # so cgroupfs; a node whose kubelet uses systemd's driver sets true, as
    # deploy/kubernetes/node/containerd.toml does.
    SystemdCgroup = false

$ crictl runp --runtime nexcage sandbox.json
$ crictl pods
POD ID          STATE   NAME     NAMESPACE   RUNTIME
6190c7b1cca16   Ready   nx-pod   default     nexcage

$ crictl logs "$CTR"
HELLO_FROM_NEXCAGE_CRI
uid=0(root) gid=0(root) groups=0(root),1(bin),…

One defect stood between the sandbox and the container, and it is the reason this had to be run rather than reasoned about: the runtime was not entering the bundle it was given. containerd’s shim serves a whole pod, so it runs the runtime from the sandbox’s directory while --bundle names the container’s own; an OCI spec’s root.path is relative ("rootfs"), and libcrun resolves it against the working directory. crun and runc chdir into the bundle for exactly this reason. Without it the container’s rootfs was looked for inside the sandbox’s — where /bin/sh genuinely is absent, because only /pause lives there. The pod sandbox worked throughout, because for it the two directories are the same one. tests/crun/foreign_cwd.sh is the check, and it needs no engine: create a container from any working directory that is not the bundle.

The other thing CRI asks for that no engine had asked for before is features, and it is answered now. containerd calls it once at startup; it never stopped a pod, because containerd records the failure and then assumes nothing, which is also the reason a wrong answer would be worse than none — what a runtime claims there, a kubelet believes.

So nexcage claims nothing of its own. The document comes from libcrun_container_get_features, the call behind crun features, so it is this build’s configuration read from this build:

$ nexcage --runtime crun features | jq -c '.linux.cgroup, .linux.mountExtensions'
{"v1":true,"v2":true,"systemd":true,"systemdUser":true}
{"idmap":{"enabled":true}}

exec is done on this backend too, which is what kubectl exec and an exec probe come down to. libcrun has three entry points for it; two take a runtime_spec_schema_config_schema_process *, which libocispec generates from a JSON schema and which is far larger than the two structs whose hand-written mirror had just put a features document at the wrong offset. The third takes the path of a file holding that process as JSON, and libcrun’s own parser builds the struct — a path has no layout to get wrong. So --process <file> hands the engine’s own file over as it is, and a command typed on a command line is written to a temporary spec of the same shape. A container made under an isolation profile is the exception: libcrun gets a copy of the engine’s file with every capability set cut to the container’s bounding set (see CLI_REFERENCE.md).

Parsing --process mattered more than adding the call. Without it the flag fell through as an unknown option and its value became the container id, so containerd’s exec was answered with exec is not implemented for the crun backend (/tmp/runc-process68868969) — the file’s path, in the place of a container.

$ crictl exec "$CTR" /bin/echo NEXCAGE_EXEC_OK
NEXCAGE_EXEC_OK

containerd reads two things from it — mountExtensions.idmap for user namespaces and rro in mountOptions for recursive read-only mounts — and both are libcrun’s answers rather than assertions maintained here. features carries no container id, and a container id is nexcage’s routing key, so the backend is resolved the way a container with no name would route. On the Proxmox LXC backend it refuses and says where the answer lives: pct creates the container there, so the hooks and seccomp a features document promises are not nexcage’s to report.

CRI-O runs a pod on nexcage too, and it did not predict containerd any more than containerd predicted podman:

# /etc/crio/crio.conf.d/10-nexcage.conf
[crio.runtime.runtimes.nexcage]
runtime_path = "/usr/local/bin/nexcage"
runtime_type = "oci"
runtime_root = "/run/nexcage-crio"

$ crictl runp --runtime nexcage sandbox.json && crictl logs "$CTR"
HELLO_FROM_NEXCAGE_CRIO
uid=0(root) gid=0(root) groups=0(root),1(bin),…

Two things stood in the way, and neither was a container operation:

  1. --root=/run/nexcage-crio was read as a command name. CRI-O writes a flag with an =, and every parser here read the two-word form only. It even mixes them: create goes through conmon, which writes --root=<dir>, while start, state, kill and delete come from CRI-O itself with --root <dir>. runc and crun accept either without noticing the question. --flag=value is now split into two words before anything looks at the vector, and the split stops at a bare --, so exec … -- env FOO=bar keeps its =.
  2. --version. CRI-O asks a runtime its version before it will use one, and asks with the flag. version as a subcommand was the only spelling. The flag maps onto the same command, so the two cannot drift.

After that the trace is crun’s, call for call, features included — and features, which nexcage only grew for containerd, is asked by CRI-O as well. What made both findings defects rather than guesses was the same control as before: two runtime handlers in one CRI-O config, one ending in nexcage and one in /usr/bin/crun, same pod, same bundles.

A third thing stood in the way and nothing here saw it, because no run gave nexcage systemd’s cgroup driver: --systemd-cgroup before the command was dropped. conmon puts it first of all whenever CRI-O’s cgroup_manager is systemd, which is CRI-O’s default and has no per-handler setting, and containerd’s shim puts it after --log-format for a handler with SystemdCgroup = true. nexcage read it only after the command, where no engine sends it, so libcrun stayed on its cgroupfs manager. That manager takes the kubelet’s kubepods-besteffort-pod<uid>.slice:crio:<id> for a directory name and makes it at the root of the hierarchy: outside the pod’s cgroup, its QoS class and kubepods.slice, and every limit the kubelet sets on them. It is read before the command now, and tests/crun/systemd_cgroup.sh checks it in each engine’s order. In a container without systemd, which is where that check runs, create then fails with cannot open sd-bus, as crun’s does.

A pod network, and it is not this runtime’s work. The engine creates the sandbox’s network namespace, runs the CNI plugins in it, and hands the runtime a path in linux.namespaces; joining a namespace by path is what libcrun already did for the sandbox’s pid, ipc and uts. So the verification is what matters here, and it holds on both engines with a bridge plus host-local:

containerd: pod 10.88.0.2, default via 10.88.0.1     CRI-O: pod 10.89.0.2, default via 10.89.0.1

# two pods, two network namespaces, both on nexcage
pod A 10.88.0.4 listens on 8080; pod B 10.88.0.5 connects   ->   SERVED
pod A 10.89.0.5 listens on 8080; pod B 10.89.0.6 connects   ->   SERVED

One thing looks like a failure and is not: ping to the gateway succeeds under containerd and fails under CRI-O. CapEff: 00000000000005fb in a CRI-O container has no CAP_NET_RAW (bit 13), which is CRI-O’s default set, and plain crun through a second handler fails the same way. TCP between pods needs no raw socket and works on both, which is what a cluster actually requires.

This is now a check rather than a story. tests/cri/pod_on_nexcage.sh creates a pod sandbox and a container in it through containerd’s CRI, on a CNI bridge, and the crun CI job runs it on every pull request. Two things in it are deliberate. The container’s image is not the sandbox’s, because the first version used pause for both and passed on a binary with the bundle defect — /pause exists in the sandbox’s rootfs too, so a container started from the wrong rootfs looked identical to one started from the right one. And the trace is asserted: without it the test would pass just as well with runc behind the handler. It fails on the pre-chdir binary with that binary’s own message, and passes on this one.

Of the gaps a pod needs, the CRI run settles three of them, and not because nexcage grew them: the shim captures the container’s output to the file the kubelet reads, the sandbox holds the namespaces its containers join, and the cgroup comes from the spec containerd writes, made the way the handler’s SystemdCgroup says, which has to be the kubelet’s driver. That is the shape of this whole stage — the engine owns the pod, the runtime owns the container. What is left from that row is what a cluster adds on top of a plain CNI bridge: NetworkPolicy and service routing, which belong to a CNI plugin and kube-proxy, not to a runtime.

Stage 5 — Kubernetes scheduling a pod onto nexcage

Done, on nexcage-e2e-1. Everything before this drove a container engine directly — podman, ctr, crictl, a standalone kubelet. This adds the two things only a cluster has, an API server and a RuntimeClass, so the pod arrives the way a user’s pod arrives.

tests/k8s/pod_on_node.sh is the run: it installs k3s if the node has none, adds one runtime to the containerd configuration k3s generates, and applies a pod that names the class.

# /var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.tmpl


[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nexcage]
  runtime_type = "io.containerd.runc.v2"
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nexcage.options]
  BinaryName = "/nexcage-traced"
  SystemdCgroup =

/nexcage-traced is a small shell wrapper the script writes. It appends each call’s arguments to /tmp/nexcage-node-trace.log, then execs the nexcage binary ($NEXCAGE, by default /usr/bin/nexcage, where the .deb puts it); that log is where the list of what Kubernetes asks a runtime, below, comes from. Outside the test, a plain setup points BinaryName straight at /usr/bin/nexcage.

SystemdCgroup came later, and a pod that is Ready does not show why it is needed. k3s run as a service gives its kubelet systemd’s cgroup driver, and containerd then names every pod’s cgroup kubepods-<qos>-pod<uid>.slice:cri-containerd:<id>, whatever the runtime. Only a handler with SystemdCgroup makes the shim pass --systemd-cgroup, which has systemd place the container under the pod’s slice. The trace from that node has no --systemd-cgroup on any call, and nexcage dropped the flag before the command anyway, so libcrun’s cgroupfs manager made that name a cgroup at the root of the hierarchy, beside kubepods.slice: no pod-level limit, QoS class or node-allocatable cap was over the pod, sandbox included. A pod’s own limits still held, which is why the checks read from inside a pod could not see it. The script now reads each pod’s container and sandbox from the host as well, /proc/<pid>/cgroup of the PID crictl reports, and fails unless both are in the pod’s cgroup under kubepods.slice, or under /kubepods/ on a kubelet with the cgroupfs driver.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata: { name: nexcage }
handler: nexcage
---
spec:
  runtimeClassName: nexcage

ok: the pod was accepted with runtimeClassName: nexcage
ok: the pod is Ready
ok: pod IP 10.42.0.5, from the cluster's CNI
ok: kubectl logs returns the container's output
ok: kubectl exec runs a command in the pod
ok: the pod was deleted
PASS: Kubernetes scheduled a pod onto nexcage on nexcage-e2e

What Kubernetes asks a runtime, recorded on that node — the whole surface, in one place at last:

features
create --bundle <dir> --pid-file <file> <id>      (with --root, --log, --log-format json)
start <id>
ps --format json <id>
exec --process /tmp/runc-process82872205 --detach --pid-file <file> <id>
kill <id> 15
kill <id> 9
kill --all <id> 9
delete <id>
delete --force <id>

delete --force <id> of a container libcrun no longer has — a second delete, or one after a reboot emptied /run — exits 0, as crun’s and runc’s does. It used to go to the Proxmox backend and fail on a node without Proxmox.

Two things in that list were worth naming. The exec line is the shape stage 4 added for containerd, --detach included: without it kubectl exec would have failed here. And ps --format json was asked for and did not exist — nexcage answered unknown command 'ps', containerd swallowed it without a word in the journal, and the pod was Ready, logged, exec’d and deleted regardless. Like features, it was found by running the thing rather than by reading a list.

ps is done now, and it is libcrun’s answer: libcrun_container_read_pids reads the container’s cgroup, children included, which is where crun ps and runc ps get theirs. The output follows crun’s — a JSON array of numbers, or a PID header and one per line — and not runc’s table, which runs the host’s ps -ef and filters it. tests/crun/ps.sh compares the list against crun ps for the same container rather than checking it looks plausible, because any list of live PIDs looks plausible. With that, everything in the trace above exists.

So the whole of what Kubernetes asks a runtime is answered. pause, resume and update came later anyway — the freezer in 0.12.0, and update in 0.13.0 because an in-place pod resize does send it (update --resources=- <id>, the linux.resources document on stdin), and the CRI test now asks for one. What is left is events, a metrics stream, which nothing has asked for.

Getting the binary onto the node took two corrections that have nothing to do with Kubernetes, and both would have looked like runtime bugs:

The script removes the k3s it installed unless --keep is passed (or KEEP=1), because that node is the E2E runner. /etc/nexcage/config.json, which routes everything to crun, is written only when absent and removed again by the run that wrote it. Any nexcage on that node run without --config reads that file, and a default build has no crun backend. On a node that already had k3s, the containerd template with the nexcage runtime (pointing at /nexcage-traced, which is removed) and RuntimeClass/nexcage are left in place.

k8s_e2e.yml runs the script on the E2E node (#312) on release tags, on demand, and on a pull request that changes tests/k8s/ or the node files below, or scripts/build_deb_local.sh. It builds the .deb from the commit under test, as the release does, and installs it on the node with apt (#380), then removes it. Its node job never runs a fork’s pull request, because the runner user there has sudo for everything.

The script installs what a release ships for a node (#313): packaging/config/config.oci.example.json, and the k3s template and RuntimeClass in deploy/kubernetes/node/, the files INSTALL.md tells an administrator to install. A run checks those files and not a copy of them. The only difference is that the template’s BinaryName points at the trace wrapper. The containerd and CRI-O examples in the same directory are the configurations stage 4 verified, since set for a systemd node’s cgroup driver, and no workflow installs them.

Then it does the same for isolation profiles (#315). It adds the two handlers INSTALL.md describes, nexcage-hardened and nexcage-small, whose BinaryName runs nexcage as nexcage@<profile>, and a RuntimeClass for each. It then runs one pod under each, and reads from inside each pod:

and from the host, that each pod’s container and sandbox are in its pod’s cgroup under kubepods.slice (/kubepods/ with the cgroupfs driver), as for the first pod.

A third pod under nexcage-hardened without hostUsers: false must stay uncreated, with “requires a user namespace” in its events.

critest: what Kubernetes checks of a runtime

crun_build.yml runs critest, the CRI validation suite from cri-tools (v1.37.0, 142 specs). It runs against containerd with nexcage as the runtime handler, routing * → crun, and again through the image’s own crun for reference (#311). Each run takes under two minutes. tests/cri/critest-known-failures names every spec that fails, with the reason. A failure the list does not name turns the job red, and so does a named spec that passes.

On the CI host, 110 specs pass, 10 fail and 22 are skipped, spec for spec the same as through crun. None of the ten is nexcage’s:

The 22 skipped specs:

Specs Why skipped
Benchmarks (5) Run only with -benchmark
NRI (13) Need -nri-socket
Image ID consistency (2), image volume digest (1) containerd’s image handling, not the runtime’s
User namespaces with idmapped mounts (1) The filesystem under /run/containerd in the CI container cannot be idmapped

critest found one defect, in its first run: #361. The attach spec’s cleanup could not stop the pod. The container had exited, kill failed, and with --log nexcage gave containerd only “operation failed”, so containerd could not tell that the process was already gone. With the fix the spec passes, as it does through crun. On a node, the same thing happens whenever a container exits while the kubelet stops it. It is fixed in the same change that brought critest in.

To run it:

docker build --build-arg BUILD_FLAGS=-Denable-backend-crun=true -t nexcage:ci .
docker build -f tests/cri/Dockerfile -t nexcage:cri-test tests/cri
docker run --rm --privileged --cgroupns=private \
  -v "$PWD/tests/cri:/t:ro" nexcage:cri-test /t/critest.sh
This site is open source. Improve this page.