nexcage

CLI Reference

nexcage [--debug] [--log-level <level>] [--log-file <path>] [--config <path>] [--root <dir>] <command> [options]

Every command accepts --help. nexcage must run as root on the Proxmox VE host: it calls pct, pvesh, pvesm and pveversion, and writes state under /run/nexcage.

Global options

These work before or after the command, and in either spelling: --root <dir> and --root=<dir> mean the same thing. Both are needed because an engine picks one without asking — containerd sends the two-word form, CRI-O sends --root= for create (through conmon) and the two-word form for everything after it. For exec, they work up to the command it runs: that command’s words are its own, and an argument there keeps its =.

Option Effect
--debug Log level debug, plus startup and system information
--log-level <level> trace, debug, info (default), warn, error, fatal
--log-file <path> Also write to <path> (and to stderr) a startup line naming the version, a line when the command starts with its words — for exec, nexcage’s own and only the number of the command’s, which can carry the variables it is given — and two when it succeeds; with --debug, system information and the command’s environment (debug mode, log file, timestamp) too. The command’s own log lines, including the ones that explain a failure, are not written there: they go to stderr, or to --log. The file is opened as --log’s is: appended to, and not opened, with a warning, when it is a symlink or a FIFO that nothing reads
--config <path> Read configuration from <path> only. A missing or unparsable file is an error (exit 1), and so is a directory, or a file that someone other than root or the user running nexcage could have written, as described below the table. A pipe is read too, once: see below
--log <path> The runtime’s own log goes to <path> instead of stderr, as an OCI runtime’s does. A container engine reads it back to report why the runtime failed. The file is appended to, so calls that share it keep each other’s lines; if its last component is a symlink, or it is a FIFO that nothing reads, it is not opened and the log goes to stderr with a warning
--log-format <text\|json> json writes one object per line — level, msg and an RFC 3339 time — which is what an engine parses
--systemd-cgroup create on the crun backend has systemd place the container, as crun and runc do: libcrun asks systemd, over D-Bus, for a scope under the slice the bundle’s cgroupsPath names (<slice>:<prefix>:<name>). containerd sends it for a handler with SystemdCgroup = true; CRI-O and podman send it whenever their cgroup manager is systemd, their default. It has to match the kubelet’s cgroup driver: without it, a cgroupsPath in that form becomes a cgroup of that very name at the root of the hierarchy, outside the pod’s. Where systemd cannot be reached, create fails with libcrun’s cannot open sd-bus, as crun’s does. Later commands follow the manager the container was created with; delete --force under it leaves /sys/fs/cgroup/<id> alone (see delete). On the Proxmox LXC backend it changes nothing: Proxmox puts the container’s cgroup at /lxc/<vmid>
--rootless <value> Accepted and ignored, as crun does. go-runc sends --rootless=<true\|false> when its caller sets one, and to runc it means “ignore cgroup permission errors”; libcrun decides that for itself, and as root it ignores none
--root <dir> Keep per-container state under <dir> instead of /run/nexcage. An OCI runtime takes this from its caller: containerd gives each namespace its own directory, so two callers on one host do not see each other’s containers. Must be absolute; it is normalised first, so /run/nexcage/ is the default. The crun backend’s state is <dir>/<id>, libcrun’s own; the Proxmox LXC backend’s is <dir>/.proxmox-lxc/<name>, so the two never share a directory. It used to be <dir>/<name>, which a build with crun took for libcrun’s; a record an older version left there is not read, and its container takes --runtime lxc once, on a command that changes it (start, stop, pause, resume, rollback), which writes the record again, as described under Backend selection. Until then a plain delete --force of it exits 0 and leaves the container in place
--profile <name> Create the container under the isolation profile <name> from the configuration file; see Isolation profiles. Only create reads it, and run refuses it
--version The same output as the version command. CRI-O asks a runtime its version this way before it will use one

An option no command takes is refused with exit 2, naming it, and so is a value option with nothing after it. That includes runc options nexcage does not implement, such as --no-pivot or --preserve-fds: dropping one would run the container differently from what was asked.

Logs go to stderr. stdout carries only command output (list, state, help text).

A log message often carries text nexcage did not write, such as what pct printed, so in a text log line and in the error echoed to stderr each control character but tab is written as an escape: \n, \r, or \xNN for each byte, the C1 controls U+0080 to U+009F and the line separators U+2028 and U+2029 included, and so is any byte that is not valid UTF-8. A backslash is written \\. One message is always one line, and nothing in it acts on the terminal. A json line carries the message as a JSON string, with those characters escaped there too (\n, \u009b); a byte that is not valid UTF-8 is left as it is.

Without --config, configuration comes from the first of /etc/nexcage/config.json, /etc/nexcage/nexcage.json that exists; see the README for the keys. Nothing in the working directory is read: up to 0.17.0 ./config.json came first, and podman runs the runtime in its caller’s directory, so a config.json another user left in /tmp became root’s configuration — its profiles, unprivileged: false, a log_file of their choosing. Name such a file with --config instead.

A file that declares ociVersion is not a configuration but an OCI runtime spec, the config.json a bundle holds. Such a file is skipped in the search, whatever its size, and refused when named with --config. An error names the file that has it, and health names the file read and any spec it skipped.

A configuration file, named with --config or found in /etc/nexcage, is refused with exit 1, and a message naming the file and the reason, when:

A pipe is read as 0.17.0 read it: --config <(...), or --config /dev/stdin fed by one. nexcage reads it once, up to the 1 MiB a configuration may hold, waiting for what the writer sends, and looks for ociVersion in what it read. A pipe is its maker’s: ... | sudo nexcage --config /dev/stdin hands root a pipe of the user’s, which is refused as any file of theirs is. A FIFO in a file’s place is read the same way, so one nothing writes to reads as empty, at once, and is refused as not JSON rather than waited on. A command that acts on a container (create, start, exec…), and features without --runtime, reads the configuration a second time and finds the pipe empty, as in 0.17.0: give those a file.

Who owns a configuration file is judged by the uids nexcage’s user namespace sees. Rootless podman runs the runtime in a namespace of its own, where a file host root owns shows as uid 65534, the kernel’s overflow uid; such a file is refused there, and the message says what that uid means.

--root is read before any command touches state, wherever it appears on the command line.

Clusters

A Proxmox VE cluster has more than one node, and pct only ever sees the one it runs on. nexcage asks the cluster instead — /cluster/resources — so a container is found wherever it lives:

$ nexcage list
ID    IMAGE     COMMAND  CREATED   STATUS   BACKEND      NODE        NAMES
100   unknown   pct      unknown   running  proxmox-lxc  prox-home   web-1
200   unknown   pct      unknown   running  proxmox-lxc  titan       web-2

$ nexcage start web-2        # on titan, from here

That start needs nexcage’s record of web-2 on this host, which create writes on the host that runs it — --node titan included. A container created from another host, or by hand, takes --runtime lxc here the first time; that command writes the record here, and later ones need no --runtime.

start, stop, delete and state work on any node: for a container on this host they run pct, and for one elsewhere they go through that node’s API. The state of a container on another node carries it as an annotation:

  "pid": 0,
  "annotations": { "io.cageforge.nexcage.node": "titan" }

Four things stay local. exec, kill, pause and resume are refused with the node’s name rather than answered wrongly; the pid in state is 0, and the io.cageforge.nexcage.node annotation names the node:

  why
exec pct exec attaches to a container on the host it runs on, and the API has no exec
kill a signal goes to the container’s init from the host, and the API has no call for that — stop and delete do work from here
pause, resume freezing writes the container’s cgroup on the host it runs on, and the API has no call for that
the pid in state an init’s PID belongs to its node’s process table; reporting it here would name whatever holds that number on this host

paused is local too, and is not refused: state and list read it from this host’s cgroup freezer, so a frozen container on another node shows as running.

create --node <name> makes the container on another node:

nexcage create --name web-2 --node titan local:vztmpl/debian-13-standard_13.0-1_amd64.tar.zst

The name is checked across the whole cluster first, because every other command resolves a name and two containers sharing one would make that ambiguous. Then the template is checked on the target node, before a VMID is taken: a storage called local is a different directory on every node unless it is shared, so a volid that exists here may simply not be there. The message names the node and the storage rather than leaving the API to say “volume does not exist”. The template has to be a <storage>:vztmpl/<file> volid in the characters Proxmox names templates with, as for rmi; anything else is a usage error (exit 2) before that node is asked.

Two things --node will not do, and refuses rather than half-does:

  why
an OCI bundle (--bundle) its rootfs is packed into a private file under $TMPDIR on this host, which the other node cannot read
a registry image (docker.io/...) the pull lands on this host’s storage, which the other node cannot read unless that storage is shared, and then only by its template name

A rootfs on a ZFS storage goes through: proxmox.storage reaches that node’s API as --rootfs <storage>:<size>, and Proxmox makes the volume there, so name a storage the target node has.

--node naming the host nexcage is running on is not “another node”: it takes the ordinary local path, pct and all.

The name goes into the API paths, /nodes/<name>/..., and pvesh hands the path of a call for another node to that node’s shell unquoted, so create, pull, images and rmi take a --node only in the form Proxmox gives a node name: letters, digits and -, a letter or digit first and last. Anything else is a usage error (exit 2) before Proxmox is asked.

A host that is not in a cluster answers /cluster/resources with its own containers, so nothing changes there. A cluster listing that cannot be read is an error (exit 1) naming the cause — pvesh failing, its output not being JSON, or more of it than nexcage reads (64 MiB, room for tens of thousands of guests). It used to be a quiet fallback to pct, which sees this host alone: a container on another node was then “not found”, list was short without saying so, and create’s check that a name is free across the cluster passed a name another node had. images likewise fails, naming the node, when a node cannot be asked for its storages; images --node <name> lists the nodes that answer one at a time.

The cluster’s listing is a cache, refreshed by pvestatd every few seconds, and nexcage is built around that in two places. A container created a moment ago is not in it yet, so a name missing there means “keep looking on this host” rather than “no such container” — otherwise create followed straight away by start fails. And for a container on this host, pct is the current answer while the cluster’s copy can be seconds stale, so list, state and every lookup take the status and the name from pct and keep from the cluster the one thing only it knows: which node a container elsewhere is on.

A container the cluster has no report for has no name in its listing. Proxmox takes a container’s name from its node’s reports, so one made on another node a few seconds ago has none yet, and nor does any container of a node that has been down for five minutes. list shows - for its name. A name that matches nothing while such a container is listed may still be that one, so it is not “no such container”: the command exits 1 naming the unnamed CT and its node, does nothing, and can be tried again once the cluster names it. create is the exception, or a node that is down would stop every create; two creates of one name on different nodes seconds apart can therefore both succeed, and every later lookup of the name refuses it.

Which container a name means. A name is looked for before a VMID, so a container named 1000 is the one 1000 finds even when another container has VMID 1000. A VMID finds its container only with --runtime lxc, and only when no container has that name: without --runtime the command reached the Proxmox backend through nexcage’s record of a container with that name, which says nothing about a VMID. A name that more than one container has — two nodes’ creates racing, or a duplicate made outside nexcage — is refused with exit 1 and the containers listed, never resolved to whichever the listing put first; address one by its VMID, with --runtime lxc. create checks both the cluster’s listing and this host’s pct list for the name, and refuses when the cluster cannot be asked.

A VMID can change hands. Proxmox gives a destroyed container’s VMID to the next container created, so right before every command that acts on the container it found — start, stop, delete (and the stop --force does first), kill, exec, pause, resume, update, snapshot, delsnapshot and rollback — nexcage reads the container’s hostname from its config again — pct config, or the node’s API — and refuses, touching nothing, when the VMID now belongs to a container with another name. exec, pause and resume check on this node only, the one place they act.

Backend selection

create and run send a new container to the backend chosen by the routing rules in the config file, Proxmox LXC by default; features asks that backend too. Every other command — start, stop, delete, kill, exec, state, pause, resume, update, ps and the snapshot commands — goes to the backend that has a record of the container: the crun backend when libcrun’s state for it is in <root>/<id> (/run/crun/<id> without --root) — its status file, or, from a create killed half way, the copy of the bundle’s config.json libcrun writes first, with no state.json beside it — and Proxmox LXC when nexcage’s own state.json for it exists, written when nexcage creates the container, in /run/nexcage/<name> (<root>/.proxmox-lxc/<name> under --root). A rule changed after create does not move a container (since 0.17.0, #372).

An id neither backend has a record of does not exist. The command exits 1 saying so, without asking Proxmox anything, and delete --force exits 0, as it does with crun and runc — a container engine cleans up that way. Such an id used to go to the Proxmox backend, which matched it against every container in the cluster by name or VMID, so a second delete --force of a crun container named like a VMID destroyed that Proxmox container. A Proxmox container nexcage did not create, one addressed by its VMID, and one whose record a reboot cleared from /run are reached with --runtime lxc; a command that changes one (start, stop, pause, resume, rollback) writes the record again under its name, so the next command needs no --runtime.

A record is of a name, not a VMID. Through it, an id finds the container with that name and nothing else: a record of 250 whose container was destroyed outside nexcage does not reach whatever container has VMID 250 now. Such a record — its container destroyed from the GUI or with pct, or renamed — is forgotten by delete --force, which then exits 0 and does nothing else; any other command exits 1 saying no container has that name. A cluster that cannot be asked is an error, never taken for “no such container”, so a record is not forgotten on a failed listing.

One id, one backend. create and run refuse, with exit 1, an id the other backend already has a record of, since every later command would reach only one of the two containers. In a binary built without crun, libcrun’s state under /run/crun is another runtime’s container and holds nothing up. For the same reason a command run with --runtime lxc writes no record of a Proxmox container whose name libcrun has: it says so, and that container is reached by its VMID, with --runtime lxc, from then on. By the name, --runtime lxc is refused, as it is for any id the crun backend has.

--runtime <lxc|crun> overrides that for one command, before or after the command name; --runtime lxc for a container the crun backend has exits 2.

A routing rule’s pattern is a regular expression only when it starts with ^ or ends with $; anything else is matched as a shell-style wildcard. So ^(kube-ovn-.*|cilium-.*)$ is a regex, web-* is a wildcard, and .* is neither a catch-all nor an error — as a wildcard it means a literal dot followed by anything, and matches nothing whose name does not begin with a dot. Write * or ^.*$ for a catch-all.

A container engine never passes --runtime, so driving nexcage from one means routing to the OCI backend in the configuration file:

{ "runtime": { "routing": [ { "pattern": "*", "runtime": "crun" } ] } }

Isolation profiles

A profile is a name in the configuration file for what a container on this node may not exceed (since 0.17.0; ADR-005). create applies it to the bundle, and only create: the container keeps what it was made with, and every later command finds it without the profile. A profile on crun is described first. A profile on Proxmox LXC is for the command line, and is described after it.

{ "profiles": {
    "hardened": { "runtime": "crun", "crun": {
      "user_namespace": "require",
      "seccomp": "require",
      "capabilities": { "drop": ["CAP_NET_RAW", "CAP_MKNOD"] },
      "limits": { "memory": "512M", "pids": 256 } } } } }
Key What it does to the bundle
runtime The backend: crun, or lxc for a profile on Proxmox LXC. A profile takes only its own backend’s parameters: an lxc section under crun, or a crun section under lxc, refuses the file
crun.user_namespace: "require" Refuses a bundle without a user namespace with uid and gid mappings, or whose mappings include host uid or gid 0. A user namespace joined by path, as a pod’s containers join the sandbox’s (/proc/<pid>/ns/user, or CRI-O’s bind mount), must not be nexcage’s own and must leave host uid and gid 0 out: nexcage reads its maps from inside it, and refuses one it cannot read. A bundle with the annotation run.oci.keep_original_groups (other than "0") is refused too: libcrun would leave the container nexcage’s own supplementary groups, which are host gids. In Kubernetes, hostUsers: false on the pod
crun.seccomp: "require" Refuses a bundle without linux.seccomp, or with a filter that refuses no system call by itself: one whose default action and every rule are SCMP_ACT_ALLOW, SCMP_ACT_LOG, SCMP_ACT_NOTIFY or SCMP_ACT_TRACE. A notifier at listenerPath decides in user space, so it does not count. Which calls a filter refuses is not judged, nor whether a rule’s args conditions can ever be met. A bundle with the annotation run.oci.seccomp_bpf_data is refused too: libcrun installs that precompiled program in place of linux.seccomp, and nexcage cannot judge it. In Kubernetes, securityContext.seccompProfile.type: RuntimeDefault
crun.capabilities.drop Removes these from every capability set of the process. Spelt as the kernel spells them, CAP_NET_RAW: a name the kernel does not have, such as CAP_SYSADMIN, refuses the file, since it would drop nothing
crun.limits.memory, crun.limits.pids Lowers the bundle’s limit to this, and sets it when the bundle has none. Memory takes bytes or 64M, 1G. The bundle’s memory.swap, which is memory plus swap, is lowered by the same amount, so the bundle gets no more swap than it asked for. containerd sets it equal to the limit, which means none. memory.max, memory.high and pids.max under linux.resources.unified, which libcrun writes last, are lowered the same way; unified’s memory.swap.max stays the bundle’s, as its memory.swap does. A unified value that is not plain decimal digits, such as max, 32M or one with a newline, becomes the profile’s, even where the kernel would have read it as less

A profile only takes away: it never adds a capability, nor raises a limit the bundle wrote as a plain number. One written otherwise becomes the profile’s. It holds for exec too. An engine builds the process for exec --process from its own copy of the spec, from before the profile narrowed it, so in a container made under a profile every capability set of that process is cut to the container’s bounding set.

It holds for update as well, for the same reason: containerd builds an in-place resize from its copy of the spec and sends the pod’s own limit. create records the profile’s caps in the container’s config, as the annotations io.cageforge.nexcage.profile.memory and .pids. An update that would take memory.limit, memory.swap or pids.limit (or the cgroup v2 files memory.max, memory.swap.max and pids.max under unified) above a cap, or to no limit (-1, max), exits 2 naming the cap and changes nothing; so does a memory.limit of 0, which libcrun writes as a limit of zero bytes. A pids limit of 0, which libcrun reads as “leave it”, passes. A resize within the caps applies as it always did; the caps are the profile’s, not the limits the container was made with, so a pod that asked for less can be resized up to them. A pod whose declared memory limit is above the cap cannot be updated in place at all, since containerd rebuilds every update from the pod’s spec and so resends that limit: an in-place resize is refused, and so is the kubelet CPU manager’s cpuset update (static policy), which leaves the container on the CPUs it had. Declare limits within the cap. memory.swap, which is memory plus swap, is held to the memory cap too: on a node with swap (NodeSwap, LimitedSwap), a pod whose memory plus swap is above the cap is created, but refused every update in the same way. A container made by an earlier nexcage carries no recorded caps and is not held to them.

create refuses a bundle that libcrun would read otherwise than nexcage checked it, with a profile or without, and so does update its resources document and exec --process into a container made under a profile its process spec. The message names where in the document:

A refusal quotes the bundle’s keys and strings escaped as in a JSON string: a NUL is \u0000, a newline \n. That is how the message an engine decodes from a json log line reads; on stderr and in a text log line, where every backslash is written \\, the same key reads process\\u0000.

Without a profile, a bundle carrying the annotation io.cageforge.nexcage.profile, or one under it such as its recorded caps, is refused.

It is named by the program name, nexcage@<profile>, or by --profile <profile>. The program name is how a container engine names one: containerd runs a runtime’s BinaryName, and conmon a CRI-O handler’s runtime_path, as argv[0], and neither has a field for an extra argument. A symlink is enough:

ln -s /usr/bin/nexcage /usr/local/bin/nexcage@hardened

A program name with no @ names no profile, so the release files’ names are not read as one.

A profile on Proxmox LXC, for the command line

A profile can name the Proxmox LXC backend instead (#394). It is a named set of what the proxmox and network sections say, plus the cores and memory a container is sized by, for the containers create and run make under it. It is for a person or a script, which name it with --profile.

{ "proxmox": { "storage": "local-lvm" },
  "profiles": {
    "pg": { "runtime": "lxc", "lxc": {
      "storage": "local-zfs", "rootfs_size_gb": 32, "bridge": "vmbr1",
      "cores": 4, "memory": "8G", "unprivileged": true } } } }

nexcage run --profile pg --name db-1 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst
Key What it does
lxc.storage, lxc.rootfs_size_gb, lxc.ostype As proxmox.storage, proxmox.rootfs_size_gb and proxmox.ostype, for this profile’s containers
lxc.bridge As network.bridge
lxc.unprivileged Only true, which makes the container unprivileged even where proxmox.unprivileged is false. false refuses the file: a profile only narrows, and cannot make a container privileged
lxc.cores The cores the container sees, 1 to 8192, when the command does not name fewer
lxc.memory Its memory, in bytes or 512M, 8G, when the command does not name less. Proxmox gives a container at least 16M, and less refuses the file

What the profile leaves out comes from the file’s sections, as for a container made without a profile.

Exit status

Code Meaning
0 Success
1 The operation failed: container not found, pct error, not a Proxmox VE host, invalid configuration file, backend not built or not implemented
2 Invalid usage: unknown command, missing name or image, unknown --runtime, unusable OCI bundle

A failure prints one line such as nexcage: start: not found after the log line that explains it.

exec is the exception: it exits with the status of the command it ran, the way runc exec does, so 1 and 2 from that command mean whatever the command means by them.

On the Proxmox LXC backend the tools nexcage runs (pct, pvesh, pvesm, tar; not exec’s pct exec) get SIGTERM when nexcage dies, and pct passes it on to its task. A command whose nexcage is killed — by a caller’s timeout, say — is cut short rather than finishing behind its back. A pct create cut short is removed by Proxmox, but stop, delete and pull stop where they were, and a pct destroy cut short can leave a container half removed. Run the command again (delete --force for a delete) to finish it. Work on another node — create --node, or any command on a container that lives there — is not cut short: pvesh hands it over ssh to that node’s pvesh, which a SIGTERM here does not reach.

Commands

create

nexcage create --name <name> [--storage <name>] [pct options] <image>
nexcage create <container-id> --bundle <dir> [--console-socket <path>] [--pid-file <path>]

Creates a container named <name>; the name becomes its hostname and must be unique in the cluster. The VMID comes from pvesh get /cluster/nextid. `–image

` is accepted in place of the positional image. `` can be: | Form | Example | Notes | |---|---|---| | Proxmox template | `local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst` | Any `:vztmpl/…` volume | | Template file name | `debian-12-standard_12.7-1_amd64.tar.zst` | Looked up as `local:vztmpl/` | | OCI registry reference | `docker.io/library/redis:7` | Proxmox VE 9.1+ only. Pulled onto `--storage` (default `local`) through `oci-registry-pull` into a file named for the whole reference, as [`pull`](#pull) names it, and created from exactly that file. A file already there under that name is used rather than pulled again; no other template is, whatever its name | | OCI bundle directory | `/var/lib/nexcage/bundles/web` | Any absolute path, containing `config.json` and `rootfs/`; a relative path is a usage error (exit 2) | `--console-socket ` is where the runtime sends the master end of the container's pty, over a Unix socket with `SCM_RIGHTS`, as the runtime-spec requires. On the crun backend, a bundle whose `config.json` sets `process.terminal` cannot be created without it. `--pid-file ` is where the container process's pid is written. Both need a backend that leaves a process running after `create`, which means `--runtime crun`. On the Proxmox LXC backend they fail with exit 1 and say so: `pct create` starts nothing, so there is no pty to hand over and no pid to write. They are refused rather than ignored — a caller that passes `--console-socket` and receives no file descriptor waits for one that is never coming. With `--bundle ` the first positional word is the **container id**, not the image, which is the form the runtime-spec defines and a container engine sends: `nexcage create web-1 --bundle /var/lib/nexcage/bundles/web`. Without `--bundle` the positional word is still the image, so the older form keeps working. The container gets `eth0` on `network.bridge` with DHCP, 512 MiB of memory and one core unless the options below or the OCI bundle set them. Its root filesystem goes to `proxmox.storage` (`:`) when that is configured. It is unprivileged unless `proxmox.unprivileged` is `false`; images from a registry always run unprivileged. For an OCI bundle, nexcage packs `rootfs/` with tar into an archive, 0600, in a directory of its own that only root can enter, `nexcage-bundle--/` under `$TMPDIR` or else `/var/tmp`. It gives `pct create` that file, which Proxmox takes from root, and deletes it and its directory afterwards, also when SIGINT, SIGTERM or SIGHUP ends nexcage first (a SIGKILL leaves them, still readable by root alone). The archive is never on a Proxmox storage, so it is not a template anyone else can see or use. `$TMPDIR` has to be an absolute path to a directory owned by root (or by the user nexcage runs as), and sticky if its group or anyone else can write to it, as `/var/tmp` is; another is refused before anything is packed. A symlink in the path is resolved once, and the directory it leads to is the one checked and used; the directories above it are not checked. The archive's path is 48 characters longer than that directory's and the name together, and `pct create` takes one of at most 255: under `/var/tmp` a name of up to 199 characters fits, and a longer path is refused (exit 2) before anything is packed. The rootfs is used as it is, so it must boot as a system container: an image without an init will not start. The bundle's namespaces turn on no Proxmox features: LXC gives the container its own namespaces, and a user namespace when it is unprivileged. Nesting and keyctl, which loosen the container's AppArmor profile and seccomp filter, are the host administrator's to give with `pct set --features`. Proxmox confines every container with its own AppArmor profile, capability set, seccomp filter and device rules, and pct cannot take a bundle's instead. `process.capabilities`, `process.noNewPrivileges`, `process.apparmorProfile`, `process.selinuxLabel`, `process.rlimits`, `root.readonly`, `linux.seccomp`, `linux.maskedPaths`, `linux.readonlyPaths`, `linux.devices` and `linux.resources.devices` are therefore not applied: create names the ones a bundle sets in one warning and goes ahead. The warning is on stderr, and in the `--log` file as well when there is one; `--log-level error` leaves it out. A bundle whose confinement has to hold needs `--runtime crun`. The bundle's bind mounts become the container's mount points `mp0`, `mp1`, ... through one `pct set` after the create, which checks each as Proxmox's API does. A mount's source and destination must be absolute, normalised paths (no empty, `.` or `..` component, no trailing `/`) without control characters (ASCII's, or the C1 ones from U+0080 to U+009F), the line and paragraph separators U+2028 and U+2029, `,` or `=`; the destination cannot be `/`, and a source under `/dev/` is refused, since Proxmox would mount the filesystem on that device rather than bind it. Proxmox binds directories only: a source that is not a directory when the container starts makes the start fail. The kernel filesystems a runtime-spec bundle mounts at `/proc`, `/sys`, `/sys/fs/cgroup`, `/dev`, `/dev/pts`, `/dev/shm` and `/dev/mqueue` are the container's own, set up by LXC and its init, and are left out with their options: the warning above names each such mount that has options (a `noexec` on `/dev/shm`, say). Any other mount that is not a bind mount is refused. A container has 256 mount points, `mp0` to `mp255`, so a bundle with more bind mounts than that is refused too. A bind mount's `ro` becomes `ro=1`, which Proxmox checks took effect, and `nosuid`, `nodev`, `noexec` and `noatime` become its `mountoptions`. Proxmox binds non-recursively, so the container sees the source without the mounts below it: `rbind` gives no more than `bind`, and the recursive forms (`rro`, `rnosuid`, ...) mean the same as the plain ones. `rw` and the `private` and `slave` propagation modes need nothing, since LXC never lets a container's mounts propagate to the host. Any other option (`rshared`, `strictatime`, `suid`, ...) is refused rather than dropped. A refusal is exit 2, before anything is packed or created, and quotes the mount's path or option escaped as in a JSON string, as the crun backend's refusals quote a bundle: U+0085 is `\u0085`, and `\\u0085` on stderr. What a bind mount can reach is in [BACKENDS.md](/architecture/BACKENDS.html#a-container-from-an-oci-bundle). The bundle's `linux.resources.memory.limit`, in bytes, becomes `--memory`. `-1`, the runtime-spec's "no limit", leaves `--memory` out, so pct's default of 512 MiB applies: Proxmox has no unlimited memory for a container. `linux.resources.cpu.shares` becomes `--cores`, 1024 shares to a core, up to the kernel's maximum of 262144 (256 cores). Nothing else under `linux.resources` reaches the container — `memory.swap`, `cpu.quota` and `cpu.period`, `pids.limit`, `unified` and the rest — and create names each one it leaves out in a warning, as it does unapplied confinement, and goes ahead; `--memory-swap`, `--cpu-quota` and `--cpu-period` give pct a swap and a CPU limit. Another negative limit, a limit below the 16 MiB Proxmox gives a container at least, more shares than that, or a value that is not an integer is refused (exit 2) before anything is packed. So is a `config.json` whose `mounts` is not a list, or whose `linux`, `linux.resources`, `memory` or `cpu` is not an object, or that holds a mount, argument or environment entry of the wrong type: such a part used to be skipped, with the mounts or limits it held. nexcage records the container as soon as `pct create` has made it. A create that fails after that point — the `pct set` that applies the bundle's mounts refused, the record not written — destroys the container it made, and that record, before it exits 1, as `runc create` does, so the same name can be created again. It checks the VMID's hostname first, as `delete` does. Should that check or the `pct destroy` fail, create says so, naming the CT, and keeps the record: `delete --force `, which is how a container engine cleans up a failed create, then destroys it. If nexcage itself is killed while `pct create` runs, pct gets `SIGTERM` and aborts the create; Proxmox removes what it had made. #### pct options What `pct create` is usually given, on the Proxmox LXC backend. Each maps onto one pct option, through `pct create` here and the node's API with `--node`, so `pct config ` shows it afterwards. Nothing given means pct's default (or nexcage's, for memory, cores and the network). | Option | pct | Notes | |---|---|---| | `--memory ` | `memory` | As for `update`: bytes, or with K, M or G. MiB, rounded up; at least 16 MiB | | `--memory-swap ` | `swap` | As for `update`, runc's meaning: memory plus swap. pct's `swap` is the difference, so it cannot be less than the memory | | `--cpu-quota `, `--cpu-period ` | `cpulimit` | Quota over period, in cores; the period defaults to 100000. At least a hundredth of a core, as for `update` | | `--cpu-share ` | `cpuunits` | cgroup v1 shares converted to the v2 weight, as runc converts them; at least 1 | | `--cores ` | `cores` | The cores the container sees | | `--ip ` | `net0` `ip=` | `dhcp` by default; `manual` leaves it unset | | `--gw
` | `net0` `gw=` | | | `--vlan ` | `net0` `tag=` | 1 to 4094 | | `--firewall` | `net0` `firewall=1` | | | `--onboot` | `onboot 1` | | | `--tags <a;b>` | `tags` | | | `--mp ` | `mp0`, `mp1`, ... | pct's own syntax, e.g. `local-lvm:8,mp=/data` (a new 8 GiB volume) or `/srv/data,mp=/data` (a bind mount). Each `--mp` takes the next index | A limit Proxmox has no setting for (`--pids-limit`, `--cpuset-cpus`, ...) is refused by name, as `update` refuses it. `--ip` and `--gw` cannot contain `,` or `=`, which would add a `net0` key nobody asked for (exit 2). `--mp` with an OCI bundle is refused: the bundle's mounts take the `mp` entries. On the crun backend every one of these is refused; it takes the limits, network and mounts from the bundle's `config.json`. ```bash nexcage create --name web-1 --memory 2G --cores 2 --ip 10.0.0.5/24 --gw 10.0.0.1 \ --vlan 20 --onboot --tags 'web;prod' --mp local-lvm:8,mp=/srv/data \ local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst ``` ### run ```bash nexcage run --name ``` `create` followed by `start`, on this host. If `start` fails, the container `create` made is destroyed again before `run` exits 1, as `runc run` does, so nothing is left under the name. `run` starts and destroys it by the VMID `create` made, never by the name, which another container may have taken in the meantime; and as `stop` and `delete` do, it first checks that the VMID's hostname is still that name. A VMID that went to another container while `pct start` ran is left alone, and so is the record, which `delete --force ` then clears. `--node` is refused (exit 1): to make the container on another node, `create --node`, then `start`. `--console-socket` and `--pid-file` are refused as `create` refuses them, since `pct start` runs the container's init itself; and the crun backend does not implement `run` (exit 1). ``, `--bundle` and `--storage` work as for `create`, and so does `--profile` with [a profile on Proxmox LXC](#a-profile-on-proxmox-lxc-for-the-command-line). ### start ```bash nexcage start # or: nexcage start --name ``` ### stop ```bash nexcage stop ``` A clean shutdown (`pct shutdown --timeout 60 --forceStop 1`): the container gets 60 seconds, then it is stopped forcibly. ### delete ```bash nexcage delete [--force] ``` Destroys the container with `pct destroy` and removes its state directory. A running container must be stopped first, unless `--force` is given: that shuts it down and then destroys it, as `runc delete --force` does. A container engine sends `--force` once it has given up waiting for a clean shutdown. `delete --force` of a container no backend has a record of exits 0, and so is safe to repeat; plain `delete` of one exits 1 (see [Backend selection](#backend-selection)). So does `delete --force` of a container nexcage has a record of but no node has any more: it forgets the record and touches nothing else. Not while the cluster lists a container it has no name for, which may be that one (see [Clusters](#clusters)): `delete` then exits 1 and keeps the record. If a node stays down and the container is known to be gone, its record is the directory `/run/nexcage/` (`/.proxmox-lxc/` under `--root`), which can be removed by hand. On the crun backend, `--force` also cleans up after a `create` that was killed half way: libcrun's state directory is there without its status file, and the cgroup create had made would otherwise stay behind, one per attempt. Such an id is the crun backend's (see [Backend selection](#backend-selection)), so an engine's `delete --force`, which names no backend, reaches libcrun. nexcage removes that cgroup only when it is the default one, `/sys/fs/cgroup/` (the config libcrun kept names no `cgroupsPath`, and `delete` was not given `--systemd-cgroup`, under which libcrun puts a container in a systemd scope and makes nothing at the root), and only with `rmdir`: a cgroup that still has processes or child cgroups is left in place and named in a warning. ### kill ```bash nexcage kill [SIGNAL] nexcage kill [-s|--signal SIGNAL] [--all] ``` Sends `SIGNAL` to the container's init process from the host, as an OCI runtime does: nexcage reads the init's host PID from `pct status --verbose`, opens a pidfd for it, checks that the process is in the container's cgroup (under `/sys/fs/cgroup/lxc//`) and sends the signal through the pidfd. A PID that is not the container's any more — the init exited after `pct status` read it, and the PID went to another process — is not signalled; the container is reported as not running (exit 1). A cgroup in that tree that cannot be read — a threaded cgroup made inside the container, one removed while nexcage looks — is passed over as one without the init, rather than failing the kill of a live container. `SIGNAL` is a name in any case, with or without `SIG` (`TERM`, `SIGKILL`, `usr1`), or a number from 1 to 64 in decimal digits and nothing else (`1_5` is refused, as runc refuses it); the default is `SIGTERM`. On `--runtime crun` libcrun is given the number nexcage read, not the text, so both backends send the same signal for the same input. An unknown signal is a usage error (exit 2); a container that is not running is an error (exit 1). `--all` means every process in the container, not only its init. On `--runtime crun` it goes to `libcrun_container_killall`, which walks the container's cgroup — it needs one it can read, and fails with `read from file 'cgroup.procs': Operation not supported` where the cgroup is not delegated. On the Proxmox LXC backend nexcage can only signal the init from the host, and says so when `--all` is given rather than implying it did more. The kernel delivers a signal from the host to a container's init only if init handles it, except `SIGKILL` and `SIGSTOP`. `SIGKILL` always stops the container; what `SIGTERM` does depends on the init system. Use `stop` for a clean shutdown. ### exec ```bash nexcage exec [args...] nexcage exec -- [args...] nexcage exec --process ``` | Option | Effect | |---|---| | `--process ` | An OCI process spec: the args, the user, the environment, whether there is a terminal. This is how a container engine sends an exec — containerd and CRI-O write the file and name it — and it takes no command of its own | | `-t`, `--tty` | The command gets a terminal | | `-d`, `--detach` | Return once the command is started, rather than waiting for it | | `--cwd ` | Working directory inside the container (`--workdir` is the same flag) | | `--user <uid[:gid]>` | Identity to run as. On the crun backend an unparsable value is an error rather than a silent root | | `--console-socket `, `--pid-file ` | Where to send the master end of the pty and where to write the pid; crun backend only | Runs `` inside a running container and exits with its status: `nexcage exec web-1 false` exits 1, and `nexcage exec web-1 sh -c 'exit 7'` exits 7. A caller that watches the status — containerd, a readiness probe — reads the command's result rather than "the runtime succeeded", which is why this command does not follow the exit table above. The command starts at the first word after `` that is neither an option nor an option's value, or after `--`. nexcage reads its own options — `--config`, `--root`, `--log`, `--debug` and the rest — anywhere before it and none after: every word of the command is passed through untouched, `=` included, so `nexcage exec web-1 grep --config x` runs `grep --config x`, as `runc exec` would. `--` is needed only when the command itself starts with a dash. Before the command, `--option=value` is one word: `exec --config=/etc/nexcage/alt.json web-1 id` reads that file as nexcage's configuration, and `id` is the command. Written that way, an option `exec` does not have, or one that takes no value, is a usage error (exit 2) rather than guessed at: `exec web-1 --tty=false sh` asked for a terminal in 0.17.0, and took `false sh` for the command. stdin, stdout and stderr are connected straight to the process in the container; nexcage does not buffer the output. On the Proxmox LXC backend this runs `pct exec --keep-env 0 -- `, so the command has to exist in the container's image — an image without a shell has no `sh` for `exec` to call. A container that is not running is an error (exit 1), as is one that does not exist; `exec` without a command is a usage error (exit 2). The command starts with LXC's clean environment there: `PATH` set to `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`, `container=lxc` and the container's own `lxc.environment`. The environment nexcage was run with does not reach it — `pct exec` would otherwise hand every variable of the caller's shell or CI job, tokens and keys included, to a binary from the image. `TERM` is not passed either, so a `--tty` session starts without it. To give the command a variable, say so on its command line: `nexcage exec web-1 -- env TERM=xterm FOO=1 `. `--process`, `--detach`, `--user`, `--cwd`, `--console-socket` and `--pid-file` are refused there rather than ignored (exit 1): `pct exec` takes a command, runs it as root in a directory nexcage does not choose, and returns when it ends, so there is no identity or directory to apply, nothing to detach from and no pty or pid to hand back. Same rule as `--console-socket` on `create`. `--tty` is honoured when nexcage runs on a terminal: `pct exec` runs `lxc-attach`, which gives the command a terminal whenever one of its standard descriptors is one. With none, `--tty` is refused. With `--process` even crun takes the terminal, working directory and user from the file, not from `--tty`, `--cwd` and `--user`. On the crun backend both forms go to `libcrun_container_exec_process_file`, which takes the path of a file holding the process spec. `--process` hands the caller's file over as it is, unless the container was made under an isolation profile: then every capability set of the process is cut to the container's bounding set, and libcrun is handed that copy, written to the container's state directory and removed afterwards (see [Isolation profiles](#isolation-profiles)). A command typed on the command line is written to a temporary spec there, which nexcage removes afterwards too. That spec carries a default `PATH`, because a process with no `PATH` cannot find `ls` and nothing would say why. It carries no other environment: nexcage has no `--env`. ### list ```bash nexcage list ``` Tab-separated columns `ID IMAGE COMMAND CREATED STATUS BACKEND NODE NAMES`. `ID` is the VMID, `NODE` the cluster node the container is on, and `NAMES` the container name. Only Proxmox LXC containers are listed; the crun backend's are not. Containers on the other nodes of the cluster are listed too, which `pct list` on one host cannot do. `STATUS` is what `pct` says, except that a frozen container is `paused` — the same answer [`state`](#state) gives, read from the same cgroup freezer. When the listing cannot be read at all, `list` fails (exit 1) instead of printing an empty table. ### state ```bash nexcage state nexcage --runtime lxc state ``` A VMID names a Proxmox LXC container only with `--runtime lxc`; without it, an id is the name of a container nexcage has a record of (see [Backend selection](#backend-selection)). Prints OCI runtime state JSON: ```json { "ociVersion": "1.3.0", "id": "web-1", "status": "running", "pid": 48213, "bundle": "/var/lib/nexcage/bundles/web", "annotations": { "io.cageforge.nexcage.node": "prox-home" } } ``` On `--runtime crun` the state comes from libcrun, which writes it itself, so the output is what `crun state` gives for the same container — the same fields in the same order, including `rootfs`, `created`, `systemd-scope` and `owner`, which the Proxmox LXC backend has no way to report. A container libcrun does not know is an error (exit 1) carrying libcrun's own message. On the Proxmox LXC backend nexcage composes the state itself, because `pct` has no such call: `status` is `created` for a container not yet started through nexcage, and otherwise `running` or `stopped` as pct reports it — or `paused`, which pct cannot report: that one is read from the container's cgroup freezer. `pid` is the host PID of the container's init while it runs or is paused on this host, and 0 otherwise; `annotations` names the node it is on as `io.cageforge.nexcage.node`, and is `{}` when the node is not known. `bundle` is the directory the container was created from, and `null` for one created from a template or a registry image; `start` and `stop` keep it. A container that does not exist is an error (exit 1). ### images ```bash nexcage images [--node ] ``` The container templates the cluster can create from, with the node and storage each one is on: ``` NODE STORAGE SHARED SIZE TEMPLATE prox-home local no 129M local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst prox-home shared-rdma yes 31M shared-rdma:vztmpl/docker.io_library_redis_7-a0ed741ac6ba1c57.tar titan local no 98M local:vztmpl/alpine-3.22-default_20250617_amd64.tar.xz ``` A template has to be readable by the node the container is created on, which is what `create --node` checks, so this is where to look before placing a container elsewhere. **A storage marked `SHARED` carries the same files on every node** and is listed once rather than once per node — counting a template twice is worse than not showing it. A storage that cannot be listed — an NFS mount that is offline, say — is left out with a warning that names it, rather than hiding every other template. Proxmox LXC only: the crun backend is handed a bundle whose rootfs is already there, so it has nothing to list. ### pull ```bash nexcage pull [--node ] [--storage ] [--filename ] ``` Fetches an OCI image from a registry into a Proxmox storage, where it becomes a container template, and prints the volid — which is what `create` takes: ```bash $ nexcage pull docker.io/library/redis:7 --storage shared-rdma shared-rdma:vztmpl/docker.io_library_redis_7-a0ed741ac6ba1c57.tar $ nexcage create --name r1 --node titan shared-rdma:vztmpl/docker.io_library_redis_7-a0ed741ac6ba1c57.tar ``` That pair is the point. `create` pulls too when given a registry reference, but only on this host, onto `local` unless `--storage` names another, and `create --node` refuses a registry reference whatever `--storage` says. Pulling onto a shared storage with `pull`, then creating from the volid it prints, is how `create --node` gets a registry image. **The file is named for the whole reference**: registry, namespace, name and tag, with every character a file name cannot have as `_`, then `-` and 16 hex digits of the reference's SHA-256, plus the `.tar` Proxmox adds. The digest tells apart references that read the same once `/` and `:` are `_`. Past 110 characters the reference is cut, and the digest still covers all of it: Proxmox names the pull's task log for an ID that holds the node and the file, and that has to fit in one file name on any node. Proxmox's own name kept only the last path component and the tag, so `docker.io/library/redis:7` and `ghcr.io/someone/redis:7` were both `redis_7.tar`. nexcage prints the volid once the storage holds exactly that file, and fails otherwise — a storage that cannot be listed is an error, and a file someone else writes during the pull is not taken for it. `create` from a registry reference uses the same name, and uses a template already on the storage only when it is that exact file. Proxmox takes a reference that ends in a tag. One pinned by a digest (`...@sha256:...`) or with no tag it refuses ("reference: value does not match the regex pattern"), and nexcage exits 1 with that message. A reference pulled there before is not pulled again: Proxmox does not write over a file, and the volid of the one there is printed. To fetch it afresh, `rmi` it first. Templates an earlier nexcage pulled have Proxmox's short names (`redis_7.tar`): `create` pulls the image once more rather than guess what they hold, and `rmi` removes the old file. `--storage` has to be a Proxmox storage ID — a letter, then letters, digits, `-`, `_` or `.` — for `pull` and for `create`, or it is a usage error (exit 2) before Proxmox is asked anything: it goes into pvesh paths, and for another node pvesh passes the path to that node's shell. `--filename ` names the file instead, before the `.tar` Proxmox adds. It takes letters, digits, `.`, `_` and `-`, not first a `.` or `-`, not last a `.` and without `..`, so that Proxmox keeps it as it is and `rmi` and `create --node` take the volid `pull` prints (a last `.` and the `.tar` Proxmox adds would make `..`), and at most 127 characters, which leaves room for the task's log; anything else is a usage error (exit 2). So is a name ending in `-` and 16 lower-case hex digits, the shape of a name nexcage derives: `create` would take that file for the reference the name was derived from, whatever image was pulled into it. A file of that name already on the storage is an error (exit 1) rather than an answer: the name says nothing about which image it holds. Whoever can write templates to the storage (`Datastore.AllocateTemplate`) can put a file under any name, this one included, and Proxmox's upload writes over an existing file. Those users are trusted with every image created from that storage. Needs **Proxmox VE 9.1 or later** on the target node, which is where `oci-registry-pull` arrived. nexcage checks the version of the host it runs on, not of the `--node` it pulls on: an older host is told which version it needs rather than left with an API error. The endpoint takes **no credentials**. Proxmox pulls with `skopeo copy` as root, and skopeo reads root's auth file on the node that pulls, so a private registry needs `skopeo login --authfile /root/.config/containers/auth.json ` on that node ([INSTALL.md](/INSTALL.html#images-from-a-private-registry)). A refused login is reported with the registry's reason and that line. ### rmi ```bash nexcage rmi
This site is open source. Improve this page.