nexcage

Troubleshooting Guide

Overview

This document contains detailed instructions for diagnosing and resolving common problems that occur when working with Nexcage. Use debug logging for detailed problem analysis.

Quick Diagnostics

1. Basic System Check

# Check version and basic functionality
./nexcage --version
./nexcage --help

# List containers
./nexcage list

2. Enable Debug Mode

# Full diagnostics go to stderr
./nexcage --debug <command> 2> /tmp/nexcage-debug.log

The file named with --log-file (or NEXCAGE_LOG_FILE, or runtime.log_path or log_file in the configuration) holds only the startup line, Starting command, the details --debug adds and, on success, the completed in and completed successfully lines. It never holds an ERROR or WARN line, and a failed command adds nothing to it after Starting command and the --debug lines that follow it.

Common Problems and Solutions

1. Container Creation Issues

Problem: “Container not created”

# Diagnostics
./nexcage --debug create --name test-container --image ubuntu:20.04 2> /tmp/create-debug.log

# Log analysis
grep -E "(ERROR|WARN|Failed|^nexcage:)" /tmp/create-debug.log

Possible Causes:

• Missing OCI bundle
• Incorrect image path
• Permission problems
• Missing dependencies

Solutions:

# 1. Check OCI bundle
ls -la /path/to/oci-bundle/
ls -la /path/to/oci-bundle/config.json

# 2. Check permissions: nexcage needs root on the Proxmox host
id -u    # must print 0

# 3. Check dependencies: nexcage runs pct, pvesh, pvesm, pveversion and hostname, and `tar --zstd` to pack an OCI bundle
which pct pvesh pvesm pveversion hostname tar zstd
# the binary also needs the shared libraries libcrun is built against (the .deb pulls them): libsystemd, libcap, libseccomp, libjson-c
ldd ./nexcage | grep 'not found'

Problem: “Image not found”

# Check available images
./nexcage --debug images

# Check LXC templates
pveam list
pveam available

Solutions:

# Download template
pveam download local ubuntu-20.04-standard_20.04-1_amd64.tar.zst

# Check after download
pveam list | grep ubuntu

2. Container Startup Issues

Problem: “Container not starting”

# Startup diagnostics
./nexcage --debug start --name test-container

# Check status
./nexcage --debug list

Possible Causes:

• Container doesn’t exist
• Configuration problems
• Missing resources
• Network problems

Solutions:

# 1. Check container existence
pct list | grep test-container

# 2. Check configuration
pct config 100  # replace 100 with container VMID

# 3. Check container logs
pct logs 100

# 4. Restart with detailed logging
pct start 100 --debug

Problem: “Network configuration error”

# Network diagnostics
./nexcage --debug create --name net-test --image ubuntu:20.04 2> /tmp/network-debug.log

Solutions:

# 1. Check network interfaces
ip link show
brctl show

# 2. Check Proxmox configuration
cat /etc/pve/lxc/100.conf | grep -i net

# 3. Create bridge if needed
sudo ip link add name vmbr1 type bridge
sudo ip link set vmbr1 up

3. Performance Issues

Problem: “Slow command execution”

# Performance measurement
./nexcage --debug --log-file /tmp/perf.log list
./nexcage --debug --log-file /tmp/perf.log create --name perf-test --image ubuntu:20.04

# Analyze execution time
grep "completed in" /tmp/perf.log

Optimization:

# 1. Check resource usage
htop
iostat -x 1

# 2. Check disk space
df -h
du -sh /var/lib/lxc/

# 3. Clear cache
sudo sync
sudo echo 3 > /proc/sys/vm/drop_caches

Problem: “Memory issues”

# Debug log; nexcage has no memory tracking (NEXCAGE_MEMORY_TRACKING is ignored)
./nexcage --debug --log-file /tmp/memory.log list

Solutions:

# 1. Check memory usage
free -h
cat /proc/meminfo

# 2. Configure swap
sudo swapon --show
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

4. Logging Issues

Problem: “Logs not created”

# Check permissions
ls -la /tmp/nexcage-debug.log
ls -la /var/log/nexcage/

# Create log directory
sudo mkdir -p /var/log/nexcage
sudo chown $USER:$USER /var/log/nexcage

Problem: “Permission denied”

# nexcage needs root on the Proxmox host
id -u    # must print 0

5. OCI Bundle Issues

Problem: “Invalid OCI bundle”

# Diagnose bundle structure
ls -la /path/to/bundle/
ls -la /path/to/bundle/rootfs/
cat /path/to/bundle/config.json | jq .

Solutions:

# 1. A bundle can be any absolute path; it needs config.json and rootfs/
mkdir -p /var/lib/nexcage/bundles/test/rootfs
echo '{"ociVersion":"1.0.0","process":{"args":["/sbin/init"]},"root":{"path":"rootfs"}}' > /var/lib/nexcage/bundles/test/config.json
# rootfs/ has to boot as a system container, e.g. an extracted template
tar --zstd -xpf /var/lib/vz/template/cache/debian-12-standard_12.7-1_amd64.tar.zst -C /var/lib/nexcage/bundles/test/rootfs

# 2. Test with the bundle
./nexcage --debug create --name test /var/lib/nexcage/bundles/test

Problem: “Rootfs not found”

# Check rootfs
ls -la /path/to/bundle/rootfs/
file /path/to/bundle/rootfs/

Solutions:

# Create rootfs
mkdir -p /path/to/bundle/rootfs
# Copy files or extract archive

Specific Scenarios

1. Proxmox Problem Diagnostics

Check Proxmox Connection

# nexcage uses no API connection or token: it runs pct and pvesh on the host
pvesh get /version
pct list

VMID Problems

# The VMID nexcage takes next
pvesh get /cluster/nextid

# Check conflicts: a name must be unique across the cluster
./nexcage list | grep "test-container"

2. Network Problem Diagnostics

Check Network Configuration

# List network interfaces
ip link show
brctl show

# Check routes
ip route show

DNS Problems

# DNS test
nslookup google.com
dig google.com

# Check /etc/resolv.conf
cat /etc/resolv.conf

3. Resource Problem Diagnostics

Check CPU

# CPU load
top -n 1
htop

# CPU information
lscpu
cat /proc/cpuinfo

Check Disk Space

# Disk usage
df -h
du -sh /var/lib/lxc/*

# Check inodes
df -i

Diagnostic Automation

1. Diagnostic Script

#!/bin/bash
# nexcage-diagnostics.sh

echo "=== nexcage Diagnostics ==="
echo "Date: $(date)"
echo "User: $(whoami)"
echo ""

echo "=== System Information ==="
uname -a
lsb_release -a
echo ""

echo "=== Memory Usage ==="
free -h
echo ""

echo "=== Disk Usage ==="
df -h
echo ""

echo "=== LXC Status ==="
pct list
echo ""

echo "=== Network Interfaces ==="
ip link show
echo ""

echo "=== nexcage Version ==="
./nexcage --version
echo ""

echo "=== Test Command ==="
./nexcage --debug --log-file /tmp/diagnostics.log list
echo "Log written to /tmp/diagnostics.log"

2. Log Monitoring

#!/bin/bash
# monitor-nexcage.sh

# The runtime log: pass it to every nexcage invocation as --log "$LOG_FILE".
# It is not the file runtime.log_path names in the configuration: that is the
# --log-file log, and it never holds an ERROR line.
# With --log-format json, grep for '"level":"error"' instead of ERROR. The
# final 'nexcage: <command>: ...' line goes to stderr only.
LOG_FILE="/var/log/nexcage/oci-runtime.log"
ALERT_EMAIL="admin@example.com"

# Monitor errors
tail -f "$LOG_FILE" | grep --line-buffered "ERROR" | while read line; do
    echo "ERROR detected: $line" | mail -s "nexcage Error Alert" "$ALERT_EMAIL"
done

3. Automatic Log Cleanup

#!/bin/bash
# cleanup-logs.sh

LOG_DIR="/var/log/nexcage"
MAX_SIZE="100M"
MAX_AGE="7"

# Cleanup by size
find "$LOG_DIR" -name "*.log" -size +$MAX_SIZE -delete

# Cleanup by age
find "$LOG_DIR" -name "*.log" -mtime +$MAX_AGE -delete

# Log rotation
for log in "$LOG_DIR"/*.log; do
    if [ -f "$log" ]; then
        mv "$log" "$log.$(date +%Y%m%d)"
        gzip "$log.$(date +%Y%m%d)"
    fi
done

Monitoring System Integration

1. Prometheus Metrics

# Export metrics from logs
# The --log-file log (runtime.log_path in the configuration): only it has the
# 'completed in' lines, never the --log runtime log
grep "completed in" /var/log/nexcage/production.log | \
  LC_ALL=C awk -v q="'" '{c=$5; gsub(q, "", c); print "nexcage_command_duration_seconds{command=\"" c "\"} " $8/1000}' > /var/lib/prometheus/nexcage.prom

2. Grafana Dashboard

{
  "dashboard": {
    "title": "nexcage Performance",
    "panels": [
      {
        "title": "Command Duration",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(nexcage_command_duration_seconds[5m])",
            "legendFormat": ""
          }
        ]
      }
    ]
  }
}

3. ELK Stack

# filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nexcage/*.log
  fields:
    service: nexcage
    environment: production
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

Security Recommendations

1. Log Protection

# Set proper permissions
chmod 640 /var/log/nexcage/*.log
chown root:nexcage /var/log/nexcage/*.log

# Encrypt sensitive logs
gpg --symmetric --cipher-algo AES256 /var/log/nexcage/sensitive.log

2. Sensitive Data Filtering

# Remove passwords from logs
sed -i 's/password=[^[:space:]]*/password=***/g' /var/log/nexcage/*.log

# Remove tokens
sed -i 's/token=[^[:space:]]*/token=***/g' /var/log/nexcage/*.log

Testing and Validation

Configuration Priority System Testing

The configuration priority system has been extensively tested to ensure proper behavior:

Test Results Summary

Priority Level	Test Method	Result	Notes
Command Line	--debug --log-file /tmp/override.log	✅ PASS	Overrides all other settings
Environment	NEXCAGE_LOG_FILE=/tmp/env-test.log	✅ PASS	Partially overrides config file
Config File	config.json with debug settings	✅ PASS	Overrides defaults
Defaults	No configuration provided	✅ PASS	Uses system defaults

Detailed Test Scenarios

Scenario 1: Configuration File Priority

# Test with config.json containing:
# {
#   "log_level": "debug",
#   "log_file": "/tmp/nexcage-logs/nexcage.log"
# }
./nexcage --config config.json list

• Expected: DEBUG mode enabled, logs to config file path
• Actual: ✅ DEBUG mode enabled, logs written to /tmp/nexcage-logs/nexcage.log
• Performance: 6ms execution time
• Status: ✅ PASS

Scenario 2: Command Line Override

# Override config file settings
./nexcage --debug --log-file /tmp/override.log list

• Expected: Command line args override config file
• Actual: ✅ Log file changed to /tmp/override.log, DEBUG mode enabled
• Performance: 6ms execution time
• Status: ✅ PASS

Scenario 3: Environment Variable Override

# Override via environment variables
NEXCAGE_LOG_FILE=/tmp/env-test.log NEXCAGE_LOG_LEVEL=warn ./nexcage list

• Expected: Environment vars override config file
• Actual: ✅ Log file changed to /tmp/env-test.log, DEBUG mode still enabled
• Performance: 5ms execution time
• Status: ✅ PASS

Performance Metrics

Test Case	Execution Time	Memory Usage	Log File Size	Status
Config file	6ms	Normal	~500 bytes	✅ Pass
Command line	6ms	Normal	~500 bytes	✅ Pass
Environment	5ms	Normal	~500 bytes	✅ Pass

Known Issues

1. File Permissions: Log file creation may fail in restricted directories
   • Impact: Logging to file disabled
   • Workaround: Use accessible directories like /tmp/
   • Fix: Since 0.8.0 nexcage prints a warning and logs to stderr only; the command still runs

Validation Checklist

• Configuration file loading works correctly
• Environment variable override functions properly
• Command line argument override works as expected
• Log file creation and writing successful
• DEBUG mode logging includes system information
• Performance tracking measures execution time
• Log format is consistent and readable
• Memory management handles allocations properly
• Error handling works for missing files
• Priority system follows correct order

Summary

This guide provides a comprehensive approach to diagnosing and resolving nexcage problems. Use debug logging as the primary tool for problem analysis and always preserve logs for further analysis.

Key Principles:

1. Always use debug mode for diagnostics
2. Preserve logs for analysis
3. Monitor system performance
4. Regularly clean old logs
5. Protect sensitive data in logs

This site is open source. Improve this page.