Proxmox VE
Run VMs and containers on Proxmox VE from the shell: qm, pct, pvesm, pvecm, vzdump, cloud-init templates, SDN and the API, plus fixes for locked guests and lost quorum.
On this page
Cheatsheet#
| Task | Command |
|---|---|
| Guests on this node | qm list, pct list |
| Every guest in the cluster | pvesh get /cluster/resources --type vm |
| Guest configuration | qm config 100, pct config 200 |
| Start, graceful stop, hard stop | qm start 100, qm shutdown 100 --timeout 120, qm stop 100 |
| Serial console of a VM | qm terminal 100 |
| Shell in a container | pct enter 200 |
| Run a command in a container | pct exec 200 -- systemctl status sshd |
| Full clone from a template | qm clone 9000 100 --name my-app --full |
| Grow a disk | qm disk resize 100 scsi0 +20G |
| Snapshot and roll back | qm snapshot 100 pre-upgrade, qm rollback 100 pre-upgrade |
| Live migrate | qm migrate 100 node2 --online |
| Move a disk to other storage | qm disk move 100 scsi0 ceph --delete 1 |
| Clear a stale lock | qm unlock 100, pct unlock 200 |
| Storage usage | pvesm status |
| What a storage holds | pvesm list local-lvm --vmid 100 |
| Back up one guest now | vzdump 100 --storage pbs --mode snapshot |
| Restore a VM | qmrestore /var/lib/vz/dump/vzdump-qemu-100-*.vma.zst 101 --storage local-lvm |
| Cluster health | pvecm status |
| HA resource state | ha-manager status |
| Node tasks and their result | pvesh get /nodes/$(hostname)/tasks --limit 20 |
| Package versions | pveversion -v |
| Apply network changes without reboot | ifreload -a |
Behaviour below is Proxmox VE 9.x (Debian 13, QEMU 10, LXC 6) unless a version is given. Reference: the Proxmox VE Administration Guide.
The model#
Proxmox VE is Debian with a cluster filesystem, an API and a set of Perl daemons on top of QEMU/KVM and LXC. Everything the CLI or web UI does is an API call, and every API call reads or writes plain-text configuration under /etc/pve. That directory is pmxcfs, a FUSE filesystem replicated to every node through Corosync. Guest configs are /etc/pve/qemu-server/<vmid>.conf and /etc/pve/lxc/<vmid>.conf, storage is /etc/pve/storage.cfg, users and ACLs are /etc/pve/user.cfg. The whole tree is limited to a few tens of megabytes and becomes read-only when the node loses quorum, which is why a broken cluster refuses to start guests.
VMIDs are cluster-wide integers (100 upward by default); a VM and a container cannot share one. Each guest belongs to one node at a time, so qm config 100 only works on the node that owns VM 100. Cluster-wide queries go through /cluster/resources.
qm manages QEMU VMs, pct manages LXC containers, pvesm manages storage, pvecm the cluster, ha-manager high availability, vzdump backups, pveum users, and pvesh is a shell over the whole API. Long-running operations (clone, migrate, backup) return a task UPID; the CLI blocks until it finishes, the API returns immediately.
VMs with qm#
qm create 100 --name my-app --memory 4096 --cores 2 --cpu host \
--scsihw virtio-scsi-single --scsi0 local-lvm:32,discard=on,ssd=1,iothread=1 \
--net0 virtio,bridge=vmbr0,tag=20 --ostype l26 \
--ide2 local:iso/debian-13-netinst.iso,media=cdrom --boot order=scsi0;ide2 \
--agent enabled=1,fstrim_cloned_disks=1
qm start 100
qm status 100 --verbose # includes balloon, disk and net counters
qm pending 100 # config changes waiting for a restart
qm set 100 --memory 8192 --delete ide2 # change one option, remove another
qm showcmd 100 --pretty # the exact QEMU command line that will run
qm guest cmd 100 ping # needs qemu-guest-agent in the guest
qm guest exec 100 -- cat /etc/hostname
qm monitor 100 # interactive QEMU monitor
qm shutdown 100 --timeout 300 --forceStop 1 # ACPI shutdown, kill after 300 s
qm destroy 100 --purge # deletes disks; --purge also removes it from backup jobs, replication and HA--scsihw virtio-scsi-single with iothread=1 per disk gives each disk its own I/O thread and is the recommended layout. --cpu host exposes the host CPU model, which is fastest but blocks migration to a node with a different CPU; use x86-64-v2-AES or x86-64-v3 in mixed clusters. discard=on lets the guest’s fstrim return space to thin storage. --agent enabled=1 makes qm shutdown, snapshots with RAM and IP reporting reliable; install qemu-guest-agent in the guest.
Clone, template, disk#
qm template 9000 # converts the VM to a template; irreversible, disks become base images
qm clone 9000 100 --name my-app --full --storage local-lvm # --full copies disks; without it a linked clone shares the base
qm clone 100 101 --snapname pre-upgrade # clone from a snapshot
qm disk resize 100 scsi0 +20G # grows only; the guest must then grow its partition and filesystem
qm disk move 100 scsi0 ceph --delete 1 # copy to another storage, remove the source when done
qm disk import 100 /var/lib/vz/images/disk.qcow2 local-lvm --format raw # lands as "unused0"; attach with qm set --scsi1 local-lvm:vm-100-disk-1
qm set 100 --scsi0 local-lvm:0,import-from=/var/lib/vz/images/disk.qcow2 # import and attach in one step
qm disk rescan --vmid 100 # find volumes on storage that the config does not reference
qm disk unlink 100 --idlist unused0 # delete an unused diskLinked clones need a storage that supports base images (qcow2 directories, ZFS, LVM-thin, Ceph) and cannot be migrated away from the base’s storage. Full clones are independent.
Snapshots#
qm snapshot 100 pre-upgrade --description "before 13 upgrade" --vmstate 1 # --vmstate saves RAM too; the guest pauses briefly
qm listsnapshot 100
qm rollback 100 pre-upgrade --start 1 # discards all changes since the snapshot; --start boots it afterwards
qm delsnapshot 100 pre-upgradeSnapshots require a storage that supports them: qcow2 on a directory, ZFS, LVM-thin, Ceph RBD. Raw images on plain LVM or NFS can only be snapshotted through volume chains, which arrived in PVE 9. A long-lived snapshot on qcow2 slows writes and on ZFS holds space forever; delete them once the change is proven.
Migration#
qm migrate 100 node2 --online # live; shared storage
qm migrate 100 node2 --online --with-local-disks # live with local disks copied over the migration network
qm migrate 100 node2 --targetstorage ceph # map local storage to a different target storage
qm remote-migrate 100 100 'apitoken=PVEAPIToken=root@pam!migrate=<uuid>,host=pve.example.com' \
--target-bridge vmbr0 --target-storage ceph --online # cross-cluster; still marked experimentalOnline migration needs the same CPU flags on both nodes (hence a portable --cpu model), matching bridge names and, for local disks, enough bandwidth on the migration network set in /etc/pve/datacenter.cfg (migration: type=insecure,network=192.0.2.0/24 to move bulk data off SSH). Offline migration is a config move plus a storage copy and works with anything.
Containers with pct#
pveam update
pveam available --section system | grep debian
pveam download local debian-13-standard_13.0-1_amd64.tar.zst
pct create 200 local:vztmpl/debian-13-standard_13.0-1_amd64.tar.zst \
--hostname my-app --memory 1024 --swap 512 --cores 2 \
--rootfs local-lvm:8 --net0 name=eth0,bridge=vmbr0,ip=dhcp \
--unprivileged 1 --features nesting=1 --ostype debian --start 1
pct enter 200 # root shell, no password
pct exec 200 -- apt-get update
pct push 200 ./app.conf /etc/app.conf --perms 0644
pct pull 200 /var/log/app.log ./app.log
pct set 200 --memory 2048 --cores 4 # applied live
pct resize 200 rootfs +8G # grows the filesystem too
pct snapshot 200 before-change; pct rollback 200 before-change
pct migrate 200 node2 --restart # containers cannot live-migrate; --restart stops, moves, starts
pct fstrim 200 # return free space to thin storage
pct destroy 200 --purgeUnprivileged is the default and the right choice. The container’s UID 0 maps to host UID 100000, so a compromise of the container’s root is an unprivileged user on the host. Privileged containers (--unprivileged 0) run as real root confined only by AppArmor and seccomp; the LXC upstream calls them unsafe, and they are only needed for things like NFS server or kernel-module-adjacent software.
--features opens holes on purpose: nesting=1 exposes /proc and /sys writable enough for systemd and Docker to run inside; keyctl=1 is needed by systemd in some distributions; fuse=1 allows FUSE mounts (avoid, it breaks snapshot-mode backups); mount=nfs;cifs allows those mounts and requires a privileged container; mknod=1 allows device node creation. Only root@pam may set most of them.
Bind mounts and devices#
pct set 200 --mp0 /srv/media,mp=/media # bind mount; not backed up, not snapshotted, not migratable
pct set 200 --mp1 local-lvm:20,mp=/var/lib/data,backup=1 # storage-backed mount point; included in backups
pct set 200 --dev0 /dev/ttyUSB0,uid=0,gid=20 # device pass-through with ownership inside the container (PVE 8+)In an unprivileged container a bind mount’s host files appear as nobody:nogroup unless their host UID lies in the container’s mapped range. Either chown 100000:100000 the host directory (host UID 100000 is container root), or map a specific host UID into the container in /etc/pve/lxc/200.conf:
# map container UID 0-999 to host 100000-100999, container 1000 to host 1000, then the rest
lxc.idmap: u 0 100000 1000
lxc.idmap: g 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: g 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 1001 101001 64535The host user running the mapped ID must be allowed in /etc/subuid and /etc/subgid (root:1000:1). Never bind mount /, /etc, /var or another guest’s storage; the container gets whatever permissions the mapping grants.
Docker inside LXC works with nesting=1 and an unprivileged container on ZFS or a directory storage, but the Proxmox documentation recommends a VM for container workloads because kernel and cgroup changes on the host regularly break it.
Storage and pvesm#
Storage is defined once in /etc/pve/storage.cfg and visible on every node; the nodes property restricts a definition to a subset. A volume is addressed as <storage>:<volume>, for example local-lvm:vm-100-disk-0 or local:iso/debian.iso. Each storage advertises content types: images (VM disks), rootdir (container disks), vztmpl, iso, backup, snippets (cloud-init and hook scripts), import.
| Type | Level | Shared | Snapshots | Thin | Notes |
|---|---|---|---|---|---|
dir | file | no | qcow2 only | qcow2 only | Any mounted path; the default local is /var/lib/vz |
nfs, cifs | file | yes | qcow2 only | qcow2 only | Mounted by PVE on every node |
lvmthin | block | no | yes | yes | Default local-lvm; fast, local only |
lvm | block | yes with shared LUN | volume chains (PVE 9) | no | Classic shared SAN layout |
zfspool | block | no | yes | yes | Native snapshots, replication with pvesr |
rbd | block | yes | yes | yes | Ceph; the usual choice for HA clusters |
cephfs | file | yes | no | n/a | ISOs, templates, backups |
pbs | backup | yes | n/a | dedup | Proxmox Backup Server target |
iscsi | block | yes | no | no | Usually a base for lvm on top |
pvesm status # usage of every storage from this node
pvesm status --content images # only storages that can hold VM disks
pvesm list local-lvm # every volume; --vmid 100 filters
pvesm path local-lvm:vm-100-disk-0 # the device or file behind a volume
pvesm add nfs nas --server nas.example.com --export /export/pve --content backup,iso --options vers=4.2
pvesm add dir bulk --path /mnt/bulk --content images,backup --is_mountpoint 1 # refuse to write if not mounted
pvesm add pbs pbs --server pbs.example.com --datastore main --username backup@pbs \
--password "$PBS_PASSWORD" --fingerprint "$PBS_FINGERPRINT"
pvesm set nas --disable 1 # take a storage offline without removing the definition
pvesm scan nfs nas.example.com # list exports before adding
pvesm alloc local-lvm 100 vm-100-disk-9 16G --format raw # create a volume by hand
pvesm free local-lvm:vm-100-disk-9 # deletes the volume
pvesm prune-backups nas --keep-last 3 --keep-weekly 4 --dry-run 1
pvesm remove nas # removes the definition only; data staysis_mountpoint 1 on a directory storage matters: without it, a failed mount makes PVE happily write disks into the empty mount point on the root filesystem until it fills.
Networking#
The host network is plain Debian ifupdown2 in /etc/network/interfaces. Guests attach to Linux bridges named vmbr0 to vmbr4094. Editing the file through the UI writes /etc/network/interfaces.new and applies on the next ifreload -a or reboot; editing by hand and running ifreload -a applies immediately.
auto lo
iface lo inet loopback
iface eno1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
bond-miimon 100
auto vmbr0
iface vmbr0 inet static
address 192.0.2.10/24
gateway 192.0.2.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
auto vmbr0.30
iface vmbr0.30 inet static
address 198.51.100.10/24 # host address on VLAN 30 via the VLAN-aware bridge
source /etc/network/interfaces.d/*A VLAN-aware bridge carries every VLAN in bridge-vids as tagged traffic and lets each guest pick one with tag= on its NIC (--net0 virtio,bridge=vmbr0,tag=20), or several with trunks=20;30. The alternative is one bridge per VLAN (vmbr20 on bond0.20), which is easier for firewall rules but scales badly. The host itself reaches a VLAN through a vmbr0.<id> sub-interface.
ifreload -a # apply /etc/network/interfaces; ifquery --check -a shows drift first
bridge vlan show # VLAN membership per bridge port
bridge fdb show br vmbr0 | grep -v permanent # learned MACs; a guest missing here is not sending
qm config 100 | grep ^net # tap interface names are tap<vmid>i<n>The built-in firewall (pve-firewall) has cluster, node and guest levels in /etc/pve/firewall/; enable it per guest NIC with firewall=1 on net0 and the [OPTIONS] enable: 1 block in /etc/pve/firewall/100.fw. See iproute2 for the underlying bridge and VLAN tools.
SDN#
SDN adds a cluster-wide layer on top of the per-node bridges: a zone is an isolation technology, a VNet is a bridge created in a zone on every node, a subnet attaches IPAM, DHCP and gateway settings to a VNet. Definitions live in /etc/pve/sdn, and nothing changes on any node until you apply, at which point /etc/network/interfaces.d/sdn is regenerated and reloaded on every node. /etc/network/interfaces must end with source /etc/network/interfaces.d/*.
| Zone | Use |
|---|---|
simple | Isolated bridge per node; optional NAT and DHCP through dnsmasq; no cross-node traffic |
vlan | One VNet per 802.1Q VLAN on an existing bridge; the common case |
qinq | Stacked VLANs; reduce guest MTU by 4 |
vxlan | Overlay over routed underlay; reduce MTU by 50; no encryption, so tunnel between sites |
evpn | VXLAN with BGP control plane and anycast gateway; needs frr-pythontools |
pvesh create /cluster/sdn/zones --type vlan --zone prod --bridge vmbr0
pvesh create /cluster/sdn/vnets --vnet app --zone prod --tag 20 --alias "App tier"
pvesh create /cluster/sdn/subnets --vnet app --subnet 10.20.0.0/24 --gateway 10.20.0.1 --type subnet
pvesh set /cluster/sdn # apply pending changes on every node
pvesh get /cluster/sdn/vnets --output-format json-pretty
qm set 100 --net0 virtio,bridge=app # the VNet name is the bridge nameCloud-init templates#
The fastest way to a new VM is a template built from a distribution cloud image with a cloud-init drive. PVE generates a NoCloud ISO from the ci* options on each start; the guest reads user, SSH keys, network and hostname from it.
wget -q https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
qm create 9000 --name debian-13-template --memory 2048 --cores 2 --cpu x86-64-v2-AES \
--net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --ostype l26 --agent enabled=1
qm set 9000 --scsi0 local-lvm:0,import-from=$PWD/debian-13-generic-amd64.qcow2,discard=on,iothread=1
qm set 9000 --ide2 local-lvm:cloudinit # the cloud-init drive
qm set 9000 --boot order=scsi0
qm set 9000 --serial0 socket --vga serial0 # cloud images log to the serial console
qm set 9000 --ciuser deploy --sshkeys ~/.ssh/id_ed25519.pub --ipconfig0 ip=dhcp --ciupgrade 0
qm disk resize 9000 scsi0 20G
qm template 9000
qm clone 9000 100 --name my-app --full
qm set 100 --ipconfig0 ip=192.0.2.50/24,gw=192.0.2.1 --nameserver 192.0.2.53 --searchdomain example.com
qm set 100 --cicustom "user=local:snippets/my-app-user.yaml" # full user-data from a snippets storage; replaces ciuser/sshkeys
qm cloudinit dump 100 user # show the generated user-data
qm cloudinit pending 100 # changes not yet written to the drive
qm cloudinit update 100 # regenerate the drive without a restart
qm start 100# /var/lib/vz/snippets/my-app-user.yaml, requires "content snippets" on storage "local"
#cloud-config
hostname: my-app
users:
- name: deploy
groups: [sudo]
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- ssh-ed25519 AAAA... deploy@example.com
package_update: true
packages: [qemu-guest-agent]
runcmd:
- systemctl enable --now qemu-guest-agent--cipassword stores a hash in the config but the option is visible to anyone who can read it; prefer keys. Cloud-init only runs on first boot for most modules, so changes to --ipconfig0 after that need cloud-init clean inside the guest or a qm cloudinit update plus a reboot with the network module set to run always. Ubuntu images configure networking through netplan from the same data; nothing extra is needed.
Backups#
vzdump is the backup engine behind the datacenter backup jobs (/etc/pve/jobs.cfg). It writes .vma.zst (VMs) or .tar.zst (containers) to a storage with backup content, or streams chunks to a Proxmox Backup Server storage where they are deduplicated and can be restored per file.
| Mode | VM | Container |
|---|---|---|
snapshot (default) | QEMU dirty-bitmap backup while running; no downtime | Storage snapshot if supported, else a suspend with rsync |
suspend | Pause, back up, resume | Freeze, rsync twice, thaw |
stop | Shut down, back up, start; the only fully consistent option without an agent | Same |
vzdump 100 --storage pbs --mode snapshot --notes-template '{{guestname}} on {{node}}'
vzdump 100 --storage nas --compress zstd --zstd 4 --prune-backups keep-last=3,keep-weekly=4
vzdump --all --exclude 9000 --storage pbs --mailto ops@example.com --mailnotification failure
vzdump 200 --storage nas --exclude-path /var/cache --exclude-path '/tmp/*' # container path excludes
vzdump 100 --storage pbs --fleecing enabled=1,storage=local-lvm # buffer writes locally when the target is slow
vzdump 100 --storage nas --protected 1 # cannot be pruned until unprotected
pvesm list nas --content backup
qmrestore nas:backup/vzdump-qemu-100-2026_09_24-02_00_00.vma.zst 101 --storage local-lvm --unique 1 # new MACs
qmrestore pbs:backup/vm/100/2026-09-24T02:00:00Z 101 --storage ceph --live-restore 1 # start while restoring (PBS only)
pct restore 201 nas:backup/vzdump-lxc-200-2026_09_24-02_00_00.tar.zst --storage local-lvm --unprivileged 1
pvesm extractconfig nas:backup/vzdump-qemu-100-2026_09_24-02_00_00.vma.zst # just the config--remove 1 (the default) prunes according to the storage’s retention after a successful backup. Defaults for every option live in /etc/vzdump.conf. A guest with --agent enabled=1 gets a filesystem freeze during snapshot mode, which is what makes the backup crash-consistent for databases.
Proxmox Backup Server#
PBS stores chunks in a datastore, deduplicates across every guest and can verify, prune and sync to another PBS. From PVE it is just another storage, so the same vzdump and restore commands apply; the extra parts are file-level restore (Backup > File Restore in the UI or proxmox-backup-client) and the client for physical hosts.
# on the PBS host
proxmox-backup-manager datastore list
proxmox-backup-manager user create backup@pbs
proxmox-backup-manager acl update /datastore/main DatastoreBackup --auth-id backup@pbs
proxmox-backup-manager cert info | grep -i fingerprint # what pvesm add pbs --fingerprint wants
proxmox-backup-manager verify main --ignore-verified 0 # re-verify everything; slow
proxmox-backup-manager garbage-collection start main # frees chunks unreferenced for >24 h
# any host with proxmox-backup-client, for example the PVE host itself
export PBS_REPOSITORY='backup@pbs@pbs.example.com:main'
export PBS_PASSWORD="$PBS_PASSWORD" # or PBS_FINGERPRINT and a keyfile
proxmox-backup-client backup etc.pxar:/etc --ns hosts --backup-id "$(hostname)"
proxmox-backup-client snapshot list --ns hosts
proxmox-backup-client restore host/pve1/2026-09-24T02:00:00Z etc.pxar /tmp/restore --ns hosts
proxmox-backup-client catalog shell host/pve1/2026-09-24T02:00:00Z etc.pxar --ns hosts # browse and pick files
proxmox-backup-client map vm/100/2026-09-24T02:00:00Z drive-scsi0.img --ns pve # exposes the disk as /dev/loopNRetention on PBS is set per datastore or namespace with prune jobs; the PVE storage’s prune-backups setting applies when vzdump --remove 1 runs. Sync jobs pull from another PBS to make an offsite copy, and remove-vanished must be set deliberately.
Clustering and quorum#
A cluster is a group of nodes sharing /etc/pve through Corosync. Each node has one vote; the cluster is quorate when more than half the votes are present. With quorum lost, pmxcfs turns read-only, guests keep running but cannot be started, migrated or edited, and HA nodes fence themselves. Two nodes therefore cannot tolerate losing either one; add a third node or a QDevice (a small corosync-qnetd daemon anywhere, including a container on another host) to break ties.
Corosync needs UDP 5405 to 5412 open between nodes, latency under about 5 ms, synchronised clocks and, ideally, its own network so that storage or backup traffic cannot starve it. Up to eight links (link0 to link7) give redundancy; a single link on the same NIC as Ceph is the classic way to lose quorum under load.
pvecm create my-cluster --link0 192.0.2.10 --link1 198.51.100.10 # on the first node
pvecm add 192.0.2.10 --link0 192.0.2.11 --link1 198.51.100.11 # on each new node; it must have no guests yet
pvecm status # Quorate: Yes/No, Expected votes, Total votes, membership
pvecm nodes
corosync-cfgtool -s # link status per ring
pvecm qdevice setup 192.0.2.5 # add a QDevice; corosync-qnetd must be installed there
pvecm delnode node3 # only after node3 is powered off and every guest moved; never boot it back into this cluster
pvecm expected 1 # DANGEROUS: tell corosync to be quorate with only this nodepvecm expected 1
This lowers the vote requirement so a lone node can write to /etc/pve. Two partitions each doing it produce two divergent configurations that Corosync cannot merge. Use it only when the other nodes are known to be off, and restore the real value (pvecm expected <n> or a restart of corosync once nodes return).
A removed node must be reinstalled before rejoining. A node that boots with the old cluster config but is no longer a member will fight over the membership; keep it off the network until it is wiped.
High availability#
HA runs the pve-ha-crm (cluster resource manager, one active per cluster) and pve-ha-lrm (local resource manager, per node) daemons. A resource in state started is restarted on failure up to max_restart times, then relocated up to max_relocate times. When a node loses quorum, its LRM stops updating the watchdog and the node resets itself after about 60 seconds; the CRM then recovers its resources on other nodes about two minutes after the failure. Requirements: at least three votes, shared storage for every HA guest’s disks, and a hardware watchdog or the softdog module (the default).
ha-manager add vm:100 --state started --max_restart 2 --max_relocate 1 --comment "app tier"
ha-manager add ct:200 --state started
ha-manager set vm:100 --state stopped # HA keeps it stopped but still relocates it after a node failure
ha-manager set vm:100 --state disabled # stop and leave alone
ha-manager set vm:100 --state ignored # temporarily take it out of HA without losing the entry
ha-manager migrate vm:100 node2 # live if possible
ha-manager relocate vm:100 node2 # stop, move, start
ha-manager status
ha-manager remove vm:100
ha-manager crm-command node-maintenance enable node2 # drain a node before patching; disable afterwardsPVE 9 replaced HA groups with rules: node affinity (ha-manager rules add node-affinity ...) restricts where a resource may run and resource affinity keeps guests together or apart. ha-manager rules list shows what applies. Start a maintenance with node-maintenance enable rather than shutting the node down so that HA migrates guests away first; a plain reboot of an HA node triggers fencing-based recovery, which is slower and not graceful.
Updating#
Proxmox publishes pve-enterprise (subscription), pve-no-subscription and pve-test repositories. On PVE 9 they are deb822 files under /etc/apt/sources.list.d/. With no subscription, disable the enterprise entries (Enabled: false in pve-enterprise.sources and ceph.sources) or apt update fails with 401.
# /etc/apt/sources.list.d/pve-no-subscription.sources
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpgapt update && apt full-upgrade # never plain "apt upgrade": it holds back kernel and pve-manager dependency changes
pveupgrade # wrapper that runs the above and reports whether a reboot is needed
pveversion -v # every PVE package version; attach to bug reports
proxmox-boot-tool kernel list # installed kernels; pin one with "kernel pin <version>" if a new one misbehaves
proxmox-boot-tool refresh # re-sync ESPs after kernel or config changes on ZFS/UEFI installs
pve8to9 --full # major-upgrade checklist; run repeatedly until clean before following the upgrade guideWithin a major version, updates are safe to apply node by node: migrate guests off, update, reboot if a new kernel landed, migrate back. Across a major version (8 to 9), follow the wiki upgrade guide exactly; the checklist script catches most of what breaks. Update every node before enabling features that need the new version cluster-wide.
The API and pvesh#
Everything is https://pve.example.com:8006/api2/json/<path>. pvesh calls the same API locally as root without authentication, which makes it the quickest way to find the path and parameters you need before scripting against the remote endpoint.
pvesh get /nodes
pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.status=="running") | "\(.node)\t\(.vmid)\t\(.name)"'
pvesh get /nodes/pve1/qemu/100/status/current --output-format json-pretty
pvesh create /nodes/pve1/qemu/100/status/start
pvesh set /nodes/pve1/qemu/100/config --memory 8192
pvesh ls /nodes/pve1/storage # children of a path
pvesh usage /nodes/pve1/qemu/100/config -v # every parameter the endpoint accepts
pvesh get /nodes/pve1/tasks/UPID:pve1:...:qmstart:100:root@pam:/statusFor remote use, create an API token. --privsep 0 gives the token the user’s permissions; --privsep 1 (default) requires separate ACLs on the token itself.
pveum user add automation@pve --comment "CI"
pveum acl modify /vms --users automation@pve --roles PVEVMAdmin
pveum acl modify /storage/local-lvm --users automation@pve --roles PVEDatastoreUser
pveum user token add automation@pve ci --privsep 0 # prints the secret once; store it, never in the repo
pveum user token permissions automation@pve ci# token stored in $PVE_TOKEN as "automation@pve!ci=<uuid>"
curl -fsS -H "Authorization: PVEAPIToken=$PVE_TOKEN" https://pve.example.com:8006/api2/json/cluster/resources?type=vm | jq '.data[] | {vmid, name, status}'
curl -fsS -X POST -H "Authorization: PVEAPIToken=$PVE_TOKEN" https://pve.example.com:8006/api2/json/nodes/pve1/qemu/100/status/shutdownPOST, PUT and DELETE return a UPID; poll /nodes/<node>/tasks/<upid>/status until status is stopped and read exitstatus. Ticket authentication (POST /access/ticket) also works for scripts but requires a CSRF token header on writes and expires after two hours. The API viewer at https://pve.example.com:8006/pve-docs/api-viewer/ documents every path. Terraform (bpg/proxmox) and Ansible (community.proxmox) both use tokens this way; see Terraform and Ansible.
Oneliners#
# Every guest in the cluster with node, status and memory
pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | [.node, .vmid, .name, .status, (.maxmem/1073741824|floor)] | @tsv' | sort -k1,1 -k2,2n
# Guests using more than 80 % of their allocated memory (needs the guest agent for VMs)
pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.status=="running" and .mem/.maxmem > 0.8) | "\(.vmid)\t\(.name)\t\(.mem*100/.maxmem|floor)%"'
# Which guests live on a storage
pvesm list ceph | awk 'NR>1 {print $NF}' | sort -n | uniq -c
# Disks on storage that no config references (candidates for cleanup, check first)
qm disk rescan --dryrun 1
# Start every stopped VM tagged "autostart" (tags come from qm set --tags)
for id in $(pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.status=="stopped" and (.tags//"" | test("autostart"))) | .vmid'); do qm start "$id"; done
# Shut down every VM on this node gracefully, 5 minutes each, then kill
for id in $(qm list | awk 'NR>1 && $3=="running" {print $1}'); do qm shutdown "$id" --timeout 300 --forceStop 1; done
# Migrate every guest off this node to node2 (containers restart)
for id in $(qm list | awk 'NR>1 {print $1}'); do qm migrate "$id" node2 --online; done; for id in $(pct list | awk 'NR>1 {print $1}'); do pct migrate "$id" node2 --restart; done
# Guest IP addresses via the agent
qm guest cmd 100 network-get-interfaces | jq -r '.[] | .name as $n | .["ip-addresses"][]? | select(.["ip-address-type"]=="ipv4") | "\($n)\t\(.["ip-address"])"'
# Which node owns VM 100
pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.vmid==100) | .node'
# Locked guests
grep -l '^lock:' /etc/pve/qemu-server/*.conf /etc/pve/lxc/*.conf 2>/dev/null
# Guests with snapshots, and the snapshot names
grep -H '^\[' /etc/pve/qemu-server/*.conf | sed 's/.*\/\([0-9]*\)\.conf:\[\(.*\)\]/\1 \2/'
# Failed tasks in the last day, cluster-wide
pvesh get /cluster/tasks --output-format json | jq -r --arg t "$(date -d '1 day ago' +%s)" '.[] | select(.starttime > ($t|tonumber) and .status != "OK" and .status != null) | "\(.starttime|todate)\t\(.node)\t\(.type)\t\(.id)\t\(.status)"'
# Backup jobs and their schedules
pvesh get /cluster/backup --output-format json | jq -r '.[] | "\(.id)\t\(.schedule)\t\(.storage)\t\(.vmid // "all")"'
# Latest backup per guest on a storage
pvesm list pbs --content backup | awk 'NR>1 {print $1}' | sort | awk -F'[/:]' '{last[$3]=$0} END {for (v in last) print last[v]}'
# Storage that is more than 85 % full
pvesm status | awk 'NR>1 && $7+0 > 85 {print $1, $7"%"}'
# ZFS ARC size and the configured limit
awk '/^size|^c_max/ {printf "%s %.1f GiB\n", $1, $3/1073741824}' /proc/spl/kstat/zfs/arcstats
# Corosync ring health
corosync-cfgtool -s && corosync-quorumtool -s
# The QEMU process for VM 100 and its threads' CPU use
top -H -p "$(cat /var/run/qemu-server/100.pid)" -bn1 | head -20
# Set a tag and a note on a guest
qm set 100 --tags prod,web --description "Owner: platform team"
# Regenerate the cloud-init drive for every clone of template 9000
for id in $(qm list | awk 'NR>1 {print $1}'); do qm config "$id" | grep -q '^ide2:.*cloudinit' && qm cloudinit update "$id"; done
# Temporarily disable HA for a guest, do work, re-enable
ha-manager set vm:100 --state ignored; qm shutdown 100; ...; ha-manager set vm:100 --state startedScripts#
Report every guest that has no backup on the given storage within the last N days.
#!/usr/bin/env bash
# usage: stale-backups.sh <storage> [days]
set -euo pipefail
storage=${1:?storage id required}
days=${2:-2}
cutoff=$(date -d "$days days ago" +%s)
declare -A latest
while read -r volid _ _ _ ctime _; do
vmid=$(printf '%s' "$volid" | sed -E 's#.*/(vm|ct)/([0-9]+)/.*#\2#; s#.*vzdump-(qemu|lxc)-([0-9]+)-.*#\2#')
[[ ${latest[$vmid]:-0} -lt $ctime ]] && latest[$vmid]=$ctime
done < <(pvesh get "/nodes/$(hostname)/storage/$storage/content" --content backup --output-format json \
| jq -r '.[] | "\(.volid) x x x \(.ctime) x"')
rc=0
while IFS=$'\t' read -r vmid name node; do
last=${latest[$vmid]:-0}
if (( last < cutoff )); then
printf '%s\t%s\t%s\tlast backup: %s\n' "$vmid" "$name" "$node" "$( (( last )) && date -d "@$last" +%F || echo never)"
rc=1
fi
done < <(pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.template != 1) | [.vmid, .name, .node] | @tsv')
exit "$rc"Drain a node for maintenance: put it in HA maintenance mode, migrate the rest, and confirm nothing is left running.
#!/usr/bin/env bash
# usage: drain-node.sh <target-node>
set -euo pipefail
target=${1:?target node required}
self=$(hostname)
ha-manager crm-command node-maintenance enable "$self"
for id in $(qm list | awk 'NR>1 && $3=="running" {print $1}'); do
if qm config "$id" | grep -q '^lock:'; then printf 'skipping locked VM %s\n' "$id" >&2; continue; fi
qm migrate "$id" "$target" --online --with-local-disks || printf 'VM %s failed\n' "$id" >&2
done
for id in $(pct list | awk 'NR>1 && $2=="running" {print $1}'); do
pct migrate "$id" "$target" --restart || printf 'CT %s failed\n' "$id" >&2
done
remaining=$( { qm list | awk 'NR>1 && $3=="running"'; pct list | awk 'NR>1 && $2=="running"'; } | wc -l)
printf '%s guests still running on %s\n' "$remaining" "$self"
(( remaining == 0 ))Create a VM from the cloud-init template with an idempotent check, for use from CI with an API token.
#!/usr/bin/env python3
"""Clone a template if the VMID does not exist yet. Needs PVE_HOST, PVE_TOKEN (user@realm!id=secret)."""
import os, sys, time, requests
host, token = os.environ["PVE_HOST"], os.environ["PVE_TOKEN"]
node, template, vmid, name, ip = "pve1", 9000, int(sys.argv[1]), sys.argv[2], sys.argv[3]
s = requests.Session()
s.headers["Authorization"] = f"PVEAPIToken={token}"
s.verify = os.environ.get("PVE_CA", True)
base = f"https://{host}:8006/api2/json"
def wait(upid):
while True:
st = s.get(f"{base}/nodes/{node}/tasks/{upid}/status", timeout=30).json()["data"]
if st["status"] == "stopped":
if st["exitstatus"] != "OK":
sys.exit(f"task failed: {st['exitstatus']}")
return
time.sleep(2)
existing = {v["vmid"] for v in s.get(f"{base}/cluster/resources", params={"type": "vm"}, timeout=30).json()["data"]}
if vmid in existing:
print(f"{vmid} exists, nothing to do"); sys.exit(0)
wait(s.post(f"{base}/nodes/{node}/qemu/{template}/clone", data={"newid": vmid, "name": name, "full": 1}, timeout=30).json()["data"])
s.put(f"{base}/nodes/{node}/qemu/{vmid}/config", data={"ipconfig0": f"ip={ip}/24,gw=192.0.2.1", "tags": "ci"}, timeout=30).raise_for_status()
wait(s.post(f"{base}/nodes/{node}/qemu/{vmid}/status/start", timeout=30).json()["data"])
print(f"{vmid} started")Troubleshooting#
Task logs are the first stop: pvesh get /nodes/$(hostname)/tasks/<UPID>/log, or the Tasks panel. Daemon logs are in the journal: journalctl -u pvedaemon -u pveproxy -u pve-cluster -u corosync -u pve-ha-lrm -u pve-ha-crm --since -1h (see systemd).
| Symptom | Cause | Fix |
|---|---|---|
VM is locked (backup) or (snapshot) | A task died mid-way, often a backup interrupted by reboot | Confirm no vzdump or qemu-img process holds it (ps aux | grep -E 'vzdump|qemu-img'), then qm unlock 100 |
start failed: QEMU exited with code 1 | Read the task log: missing ISO, storage not active, PCI device busy, kvm: failed to initialize KVM | qm showcmd 100 --pretty and run the printed command by hand to see QEMU’s own message |
TASK ERROR: can't lock file '/var/lock/qemu-server/lock-100.conf' | Another qm operation on the same VM is running or hung | fuser /var/lock/qemu-server/lock-100.conf; wait or kill the stuck process |
cluster not ready - no quorum? (500) | Fewer than half the votes present | pvecm status, corosync-cfgtool -s, fix the network; as a last resort on a lone surviving node pvecm expected 1 |
/etc/pve read-only, pmxcfs errors in the journal | Same as above, or pve-cluster stopped | systemctl restart pve-cluster corosync; check journalctl -u pve-cluster for database corruption |
| Node fenced (rebooted itself) | HA LRM lost quorum for 60 s: corosync network saturated or flapping | Separate corosync link, add link1; corosync-cfgtool -s for link errors |
Storage inactive or unknown in pvesm status | NFS/CIFS server unreachable, iSCSI portal down, ZFS pool not imported | pvesm status --storage nas, journalctl -u pvestatd, mount | grep nas, zpool import |
local-lvm full but guests are small | Thin pool over-provisioned and guests never trimmed | lvs -a pve for Data%; enable discard=on and run fstrim -av in guests, pct fstrim for containers; delete snapshots |
| Root filesystem full on a node | Backups written to /var/lib/vz or a directory storage whose mount failed | du -xsh /var/lib/vz/dump /mnt/*; set is_mountpoint 1; move backups elsewhere |
| Slow disk I/O in a VM | Cache mode, no iothread, VirtIO not used, host swapping, ZFS ARC starving guests | Use virtio-scsi-single with iothread=1, cache=none on block storage; check free -g, arcstat; set zfs_arc_max in /etc/modprobe.d/zfs.conf |
qm migrate fails with CPU flag errors | Source VM uses --cpu host or a model the target lacks | Set a common model (x86-64-v3), stop and start, migrate |
Migration fails with storage ... is not available on node | Storage definition restricted with nodes, or a local storage | --targetstorage, or extend the storage’s nodes list |
| Container will not start after host upgrade | Old distribution needs cgroup v1, or nesting missing for systemd | pct start 200 --debug; lxc-start -n 200 -F -l DEBUG -o /tmp/lxc-200.log; enable nesting=1 |
Bind mount files show as nobody:nogroup | Unprivileged UID mapping | chown 100000:100000 on the host, or an lxc.idmap entry |
| Web UI unreachable on 8006, SSH works | pveproxy down, or certificate expired | systemctl status pveproxy, pvecm updatecerts --force, systemctl restart pveproxy |
apt update returns 401 Unauthorized | Enterprise repository enabled without a subscription | Set Enabled: false in pve-enterprise.sources and ceph.sources |
| Guest has no network after clone | Same MAC as the template, or cloud-init not regenerated | qm set 100 --net0 virtio,bridge=vmbr0 regenerates a MAC; qm cloudinit update 100 |
For a VM that hangs at boot, qm terminal 100 (requires --serial0 socket) shows the kernel console; for a VM with a graphical console, qm vncproxy from the UI is the only option. A guest process stuck in D state on the host usually points at the storage backend; dmesg -T | tail and iostat -x 5 on the host narrow it down (linux-performance).