# Proxmox VE

> Run VMs and containers on Proxmox VE from the shell: qm, pct, pvesm, pvecm, vzdump, cloud-init templates, SDN and the API, plus fixes for locked guests and lost quorum.

Canonical: https://www.wiki.jodisand.me/proxmox/
Reviewed: 2026-09-24
Related: [libvirt and KVM](https://www.wiki.jodisand.me/libvirt/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md), [SSH](https://www.wiki.jodisand.me/ssh/index.md), [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md), [Ansible](https://www.wiki.jodisand.me/ansible/index.md)


## Cheatsheet

| Task | Command |
| --- | --- |
| Guests on this node | `qm list`, `pct list` |
| Every guest in the cluster | `pvesh get /cluster/resources --type vm` |
| Guest configuration | `qm config 100`, `pct config 200` |
| Start, graceful stop, hard stop | `qm start 100`, `qm shutdown 100 --timeout 120`, `qm stop 100` |
| Serial console of a VM | `qm terminal 100` |
| Shell in a container | `pct enter 200` |
| Run a command in a container | `pct exec 200 -- systemctl status sshd` |
| Full clone from a template | `qm clone 9000 100 --name my-app --full` |
| Grow a disk | `qm disk resize 100 scsi0 +20G` |
| Snapshot and roll back | `qm snapshot 100 pre-upgrade`, `qm rollback 100 pre-upgrade` |
| Live migrate | `qm migrate 100 node2 --online` |
| Move a disk to other storage | `qm disk move 100 scsi0 ceph --delete 1` |
| Clear a stale lock | `qm unlock 100`, `pct unlock 200` |
| Storage usage | `pvesm status` |
| What a storage holds | `pvesm list local-lvm --vmid 100` |
| Back up one guest now | `vzdump 100 --storage pbs --mode snapshot` |
| Restore a VM | `qmrestore /var/lib/vz/dump/vzdump-qemu-100-*.vma.zst 101 --storage local-lvm` |
| Cluster health | `pvecm status` |
| HA resource state | `ha-manager status` |
| Node tasks and their result | `pvesh get /nodes/$(hostname)/tasks --limit 20` |
| Package versions | `pveversion -v` |
| Apply network changes without reboot | `ifreload -a` |

Behaviour below is Proxmox VE 9.x (Debian 13, QEMU 10, LXC 6) unless a version is given. Reference: the [Proxmox VE Administration Guide](https://pve.proxmox.com/pve-docs/pve-admin-guide.html).

## The model

Proxmox VE is Debian with a cluster filesystem, an API and a set of Perl daemons on top of QEMU/KVM and LXC. Everything the CLI or web UI does is an API call, and every API call reads or writes plain-text configuration under `/etc/pve`. That directory is `pmxcfs`, a FUSE filesystem replicated to every node through Corosync. Guest configs are `/etc/pve/qemu-server/<vmid>.conf` and `/etc/pve/lxc/<vmid>.conf`, storage is `/etc/pve/storage.cfg`, users and ACLs are `/etc/pve/user.cfg`. The whole tree is limited to a few tens of megabytes and becomes read-only when the node loses quorum, which is why a broken cluster refuses to start guests.

VMIDs are cluster-wide integers (100 upward by default); a VM and a container cannot share one. Each guest belongs to one node at a time, so `qm config 100` only works on the node that owns VM 100. Cluster-wide queries go through `/cluster/resources`.

`qm` manages QEMU VMs, `pct` manages LXC containers, `pvesm` manages storage, `pvecm` the cluster, `ha-manager` high availability, `vzdump` backups, `pveum` users, and `pvesh` is a shell over the whole API. Long-running operations (clone, migrate, backup) return a task UPID; the CLI blocks until it finishes, the API returns immediately.

## VMs with qm

```sh
qm create 100 --name my-app --memory 4096 --cores 2 --cpu host \
  --scsihw virtio-scsi-single --scsi0 local-lvm:32,discard=on,ssd=1,iothread=1 \
  --net0 virtio,bridge=vmbr0,tag=20 --ostype l26 \
  --ide2 local:iso/debian-13-netinst.iso,media=cdrom --boot order=scsi0;ide2 \
  --agent enabled=1,fstrim_cloned_disks=1
qm start 100
qm status 100 --verbose        # includes balloon, disk and net counters
qm pending 100                 # config changes waiting for a restart
qm set 100 --memory 8192 --delete ide2    # change one option, remove another
qm showcmd 100 --pretty        # the exact QEMU command line that will run
qm guest cmd 100 ping          # needs qemu-guest-agent in the guest
qm guest exec 100 -- cat /etc/hostname
qm monitor 100                 # interactive QEMU monitor
qm shutdown 100 --timeout 300 --forceStop 1   # ACPI shutdown, kill after 300 s
qm destroy 100 --purge         # deletes disks; --purge also removes it from backup jobs, replication and HA
```

`--scsihw virtio-scsi-single` with `iothread=1` per disk gives each disk its own I/O thread and is the recommended layout. `--cpu host` exposes the host CPU model, which is fastest but blocks migration to a node with a different CPU; use `x86-64-v2-AES` or `x86-64-v3` in mixed clusters. `discard=on` lets the guest's `fstrim` return space to thin storage. `--agent enabled=1` makes `qm shutdown`, snapshots with RAM and IP reporting reliable; install `qemu-guest-agent` in the guest.

### Clone, template, disk

```sh
qm template 9000                          # converts the VM to a template; irreversible, disks become base images
qm clone 9000 100 --name my-app --full --storage local-lvm   # --full copies disks; without it a linked clone shares the base
qm clone 100 101 --snapname pre-upgrade   # clone from a snapshot
qm disk resize 100 scsi0 +20G             # grows only; the guest must then grow its partition and filesystem
qm disk move 100 scsi0 ceph --delete 1    # copy to another storage, remove the source when done
qm disk import 100 /var/lib/vz/images/disk.qcow2 local-lvm --format raw   # lands as "unused0"; attach with qm set --scsi1 local-lvm:vm-100-disk-1
qm set 100 --scsi0 local-lvm:0,import-from=/var/lib/vz/images/disk.qcow2   # import and attach in one step
qm disk rescan --vmid 100                 # find volumes on storage that the config does not reference
qm disk unlink 100 --idlist unused0       # delete an unused disk
```

Linked clones need a storage that supports base images (qcow2 directories, ZFS, LVM-thin, Ceph) and cannot be migrated away from the base's storage. Full clones are independent.

### Snapshots

```sh
qm snapshot 100 pre-upgrade --description "before 13 upgrade" --vmstate 1   # --vmstate saves RAM too; the guest pauses briefly
qm listsnapshot 100
qm rollback 100 pre-upgrade --start 1     # discards all changes since the snapshot; --start boots it afterwards
qm delsnapshot 100 pre-upgrade
```

Snapshots require a storage that supports them: qcow2 on a directory, ZFS, LVM-thin, Ceph RBD. Raw images on plain LVM or NFS can only be snapshotted through volume chains, which arrived in PVE 9. A long-lived snapshot on qcow2 slows writes and on ZFS holds space forever; delete them once the change is proven.

### Migration

```sh
qm migrate 100 node2 --online                        # live; shared storage
qm migrate 100 node2 --online --with-local-disks     # live with local disks copied over the migration network
qm migrate 100 node2 --targetstorage ceph            # map local storage to a different target storage
qm remote-migrate 100 100 'apitoken=PVEAPIToken=root@pam!migrate=<uuid>,host=pve.example.com' \
  --target-bridge vmbr0 --target-storage ceph --online   # cross-cluster; still marked experimental
```

Online migration needs the same CPU flags on both nodes (hence a portable `--cpu` model), matching bridge names and, for local disks, enough bandwidth on the migration network set in `/etc/pve/datacenter.cfg` (`migration: type=insecure,network=192.0.2.0/24` to move bulk data off SSH). Offline migration is a config move plus a storage copy and works with anything.

## Containers with pct

```sh
pveam update
pveam available --section system | grep debian
pveam download local debian-13-standard_13.0-1_amd64.tar.zst
pct create 200 local:vztmpl/debian-13-standard_13.0-1_amd64.tar.zst \
  --hostname my-app --memory 1024 --swap 512 --cores 2 \
  --rootfs local-lvm:8 --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --unprivileged 1 --features nesting=1 --ostype debian --start 1
pct enter 200                     # root shell, no password
pct exec 200 -- apt-get update
pct push 200 ./app.conf /etc/app.conf --perms 0644
pct pull 200 /var/log/app.log ./app.log
pct set 200 --memory 2048 --cores 4           # applied live
pct resize 200 rootfs +8G                     # grows the filesystem too
pct snapshot 200 before-change; pct rollback 200 before-change
pct migrate 200 node2 --restart               # containers cannot live-migrate; --restart stops, moves, starts
pct fstrim 200                                # return free space to thin storage
pct destroy 200 --purge
```

Unprivileged is the default and the right choice. The container's UID 0 maps to host UID 100000, so a compromise of the container's root is an unprivileged user on the host. Privileged containers (`--unprivileged 0`) run as real root confined only by AppArmor and seccomp; the LXC upstream calls them unsafe, and they are only needed for things like NFS server or kernel-module-adjacent software.

`--features` opens holes on purpose: `nesting=1` exposes `/proc` and `/sys` writable enough for systemd and Docker to run inside; `keyctl=1` is needed by systemd in some distributions; `fuse=1` allows FUSE mounts (avoid, it breaks snapshot-mode backups); `mount=nfs;cifs` allows those mounts and requires a privileged container; `mknod=1` allows device node creation. Only `root@pam` may set most of them.

### Bind mounts and devices

```sh
pct set 200 --mp0 /srv/media,mp=/media                     # bind mount; not backed up, not snapshotted, not migratable
pct set 200 --mp1 local-lvm:20,mp=/var/lib/data,backup=1   # storage-backed mount point; included in backups
pct set 200 --dev0 /dev/ttyUSB0,uid=0,gid=20               # device pass-through with ownership inside the container (PVE 8+)
```

In an unprivileged container a bind mount's host files appear as `nobody:nogroup` unless their host UID lies in the container's mapped range. Either `chown 100000:100000` the host directory (host UID 100000 is container root), or map a specific host UID into the container in `/etc/pve/lxc/200.conf`:

```ini
# map container UID 0-999 to host 100000-100999, container 1000 to host 1000, then the rest
lxc.idmap: u 0 100000 1000
lxc.idmap: g 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: g 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 1001 101001 64535
```

The host user running the mapped ID must be allowed in `/etc/subuid` and `/etc/subgid` (`root:1000:1`). Never bind mount `/`, `/etc`, `/var` or another guest's storage; the container gets whatever permissions the mapping grants.

Docker inside LXC works with `nesting=1` and an unprivileged container on ZFS or a directory storage, but the Proxmox documentation recommends a VM for container workloads because kernel and cgroup changes on the host regularly break it.

## Storage and pvesm

Storage is defined once in `/etc/pve/storage.cfg` and visible on every node; the `nodes` property restricts a definition to a subset. A volume is addressed as `<storage>:<volume>`, for example `local-lvm:vm-100-disk-0` or `local:iso/debian.iso`. Each storage advertises content types: `images` (VM disks), `rootdir` (container disks), `vztmpl`, `iso`, `backup`, `snippets` (cloud-init and hook scripts), `import`.

| Type | Level | Shared | Snapshots | Thin | Notes |
| --- | --- | --- | --- | --- | --- |
| `dir` | file | no | qcow2 only | qcow2 only | Any mounted path; the default `local` is `/var/lib/vz` |
| `nfs`, `cifs` | file | yes | qcow2 only | qcow2 only | Mounted by PVE on every node |
| `lvmthin` | block | no | yes | yes | Default `local-lvm`; fast, local only |
| `lvm` | block | yes with shared LUN | volume chains (PVE 9) | no | Classic shared SAN layout |
| `zfspool` | block | no | yes | yes | Native snapshots, replication with `pvesr` |
| `rbd` | block | yes | yes | yes | Ceph; the usual choice for HA clusters |
| `cephfs` | file | yes | no | n/a | ISOs, templates, backups |
| `pbs` | backup | yes | n/a | dedup | Proxmox Backup Server target |
| `iscsi` | block | yes | no | no | Usually a base for `lvm` on top |

```sh
pvesm status                                   # usage of every storage from this node
pvesm status --content images                  # only storages that can hold VM disks
pvesm list local-lvm                           # every volume; --vmid 100 filters
pvesm path local-lvm:vm-100-disk-0             # the device or file behind a volume
pvesm add nfs nas --server nas.example.com --export /export/pve --content backup,iso --options vers=4.2
pvesm add dir bulk --path /mnt/bulk --content images,backup --is_mountpoint 1   # refuse to write if not mounted
pvesm add pbs pbs --server pbs.example.com --datastore main --username backup@pbs \
  --password "$PBS_PASSWORD" --fingerprint "$PBS_FINGERPRINT"
pvesm set nas --disable 1                      # take a storage offline without removing the definition
pvesm scan nfs nas.example.com                 # list exports before adding
pvesm alloc local-lvm 100 vm-100-disk-9 16G --format raw    # create a volume by hand
pvesm free local-lvm:vm-100-disk-9             # deletes the volume
pvesm prune-backups nas --keep-last 3 --keep-weekly 4 --dry-run 1
pvesm remove nas                               # removes the definition only; data stays
```

`is_mountpoint 1` on a directory storage matters: without it, a failed mount makes PVE happily write disks into the empty mount point on the root filesystem until it fills.

## Networking

The host network is plain Debian `ifupdown2` in `/etc/network/interfaces`. Guests attach to Linux bridges named `vmbr0` to `vmbr4094`. Editing the file through the UI writes `/etc/network/interfaces.new` and applies on the next `ifreload -a` or reboot; editing by hand and running `ifreload -a` applies immediately.

```ini
auto lo
iface lo inet loopback

iface eno1 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4
    bond-miimon 100

auto vmbr0
iface vmbr0 inet static
    address 192.0.2.10/24
    gateway 192.0.2.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.30
iface vmbr0.30 inet static
    address 198.51.100.10/24        # host address on VLAN 30 via the VLAN-aware bridge

source /etc/network/interfaces.d/*
```

A VLAN-aware bridge carries every VLAN in `bridge-vids` as tagged traffic and lets each guest pick one with `tag=` on its NIC (`--net0 virtio,bridge=vmbr0,tag=20`), or several with `trunks=20;30`. The alternative is one bridge per VLAN (`vmbr20` on `bond0.20`), which is easier for firewall rules but scales badly. The host itself reaches a VLAN through a `vmbr0.<id>` sub-interface.

```sh
ifreload -a                        # apply /etc/network/interfaces; ifquery --check -a shows drift first
bridge vlan show                   # VLAN membership per bridge port
bridge fdb show br vmbr0 | grep -v permanent   # learned MACs; a guest missing here is not sending
qm config 100 | grep ^net          # tap interface names are tap<vmid>i<n>
```

The built-in firewall (`pve-firewall`) has cluster, node and guest levels in `/etc/pve/firewall/`; enable it per guest NIC with `firewall=1` on `net0` and the `[OPTIONS] enable: 1` block in `/etc/pve/firewall/100.fw`. See [iproute2](https://www.wiki.jodisand.me/iproute2/) for the underlying bridge and VLAN tools.

### SDN

SDN adds a cluster-wide layer on top of the per-node bridges: a **zone** is an isolation technology, a **VNet** is a bridge created in a zone on every node, a **subnet** attaches IPAM, DHCP and gateway settings to a VNet. Definitions live in `/etc/pve/sdn`, and nothing changes on any node until you apply, at which point `/etc/network/interfaces.d/sdn` is regenerated and reloaded on every node. `/etc/network/interfaces` must end with `source /etc/network/interfaces.d/*`.

| Zone | Use |
| --- | --- |
| `simple` | Isolated bridge per node; optional NAT and DHCP through dnsmasq; no cross-node traffic |
| `vlan` | One VNet per 802.1Q VLAN on an existing bridge; the common case |
| `qinq` | Stacked VLANs; reduce guest MTU by 4 |
| `vxlan` | Overlay over routed underlay; reduce MTU by 50; no encryption, so tunnel between sites |
| `evpn` | VXLAN with BGP control plane and anycast gateway; needs `frr-pythontools` |

```sh
pvesh create /cluster/sdn/zones --type vlan --zone prod --bridge vmbr0
pvesh create /cluster/sdn/vnets --vnet app --zone prod --tag 20 --alias "App tier"
pvesh create /cluster/sdn/subnets --vnet app --subnet 10.20.0.0/24 --gateway 10.20.0.1 --type subnet
pvesh set /cluster/sdn                                    # apply pending changes on every node
pvesh get /cluster/sdn/vnets --output-format json-pretty
qm set 100 --net0 virtio,bridge=app                       # the VNet name is the bridge name
```

## Cloud-init templates

The fastest way to a new VM is a template built from a distribution cloud image with a cloud-init drive. PVE generates a NoCloud ISO from the `ci*` options on each start; the guest reads user, SSH keys, network and hostname from it.

```sh
wget -q https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
qm create 9000 --name debian-13-template --memory 2048 --cores 2 --cpu x86-64-v2-AES \
  --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --ostype l26 --agent enabled=1
qm set 9000 --scsi0 local-lvm:0,import-from=$PWD/debian-13-generic-amd64.qcow2,discard=on,iothread=1
qm set 9000 --ide2 local-lvm:cloudinit          # the cloud-init drive
qm set 9000 --boot order=scsi0
qm set 9000 --serial0 socket --vga serial0      # cloud images log to the serial console
qm set 9000 --ciuser deploy --sshkeys ~/.ssh/id_ed25519.pub --ipconfig0 ip=dhcp --ciupgrade 0
qm disk resize 9000 scsi0 20G
qm template 9000

qm clone 9000 100 --name my-app --full
qm set 100 --ipconfig0 ip=192.0.2.50/24,gw=192.0.2.1 --nameserver 192.0.2.53 --searchdomain example.com
qm set 100 --cicustom "user=local:snippets/my-app-user.yaml"   # full user-data from a snippets storage; replaces ciuser/sshkeys
qm cloudinit dump 100 user       # show the generated user-data
qm cloudinit pending 100         # changes not yet written to the drive
qm cloudinit update 100          # regenerate the drive without a restart
qm start 100
```

```yaml
# /var/lib/vz/snippets/my-app-user.yaml, requires "content snippets" on storage "local"
#cloud-config
hostname: my-app
users:
  - name: deploy
    groups: [sudo]
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... deploy@example.com
package_update: true
packages: [qemu-guest-agent]
runcmd:
  - systemctl enable --now qemu-guest-agent
```

`--cipassword` stores a hash in the config but the option is visible to anyone who can read it; prefer keys. Cloud-init only runs on first boot for most modules, so changes to `--ipconfig0` after that need `cloud-init clean` inside the guest or a `qm cloudinit update` plus a reboot with the network module set to run always. Ubuntu images configure networking through netplan from the same data; nothing extra is needed.

## Backups

`vzdump` is the backup engine behind the datacenter backup jobs (`/etc/pve/jobs.cfg`). It writes `.vma.zst` (VMs) or `.tar.zst` (containers) to a storage with `backup` content, or streams chunks to a Proxmox Backup Server storage where they are deduplicated and can be restored per file.

| Mode | VM | Container |
| --- | --- | --- |
| `snapshot` (default) | QEMU dirty-bitmap backup while running; no downtime | Storage snapshot if supported, else a `suspend` with rsync |
| `suspend` | Pause, back up, resume | Freeze, rsync twice, thaw |
| `stop` | Shut down, back up, start; the only fully consistent option without an agent | Same |

```sh
vzdump 100 --storage pbs --mode snapshot --notes-template '{{guestname}} on {{node}}'
vzdump 100 --storage nas --compress zstd --zstd 4 --prune-backups keep-last=3,keep-weekly=4
vzdump --all --exclude 9000 --storage pbs --mailto ops@example.com --mailnotification failure
vzdump 200 --storage nas --exclude-path /var/cache --exclude-path '/tmp/*'   # container path excludes
vzdump 100 --storage pbs --fleecing enabled=1,storage=local-lvm            # buffer writes locally when the target is slow
vzdump 100 --storage nas --protected 1                                    # cannot be pruned until unprotected
pvesm list nas --content backup
qmrestore nas:backup/vzdump-qemu-100-2026_09_24-02_00_00.vma.zst 101 --storage local-lvm --unique 1   # new MACs
qmrestore pbs:backup/vm/100/2026-09-24T02:00:00Z 101 --storage ceph --live-restore 1   # start while restoring (PBS only)
pct restore 201 nas:backup/vzdump-lxc-200-2026_09_24-02_00_00.tar.zst --storage local-lvm --unprivileged 1
pvesm extractconfig nas:backup/vzdump-qemu-100-2026_09_24-02_00_00.vma.zst   # just the config
```

`--remove 1` (the default) prunes according to the storage's retention after a successful backup. Defaults for every option live in `/etc/vzdump.conf`. A guest with `--agent enabled=1` gets a filesystem freeze during snapshot mode, which is what makes the backup crash-consistent for databases.

### Proxmox Backup Server

PBS stores chunks in a datastore, deduplicates across every guest and can verify, prune and sync to another PBS. From PVE it is just another storage, so the same `vzdump` and restore commands apply; the extra parts are file-level restore (`Backup > File Restore` in the UI or `proxmox-backup-client`) and the client for physical hosts.

```sh
# on the PBS host
proxmox-backup-manager datastore list
proxmox-backup-manager user create backup@pbs
proxmox-backup-manager acl update /datastore/main DatastoreBackup --auth-id backup@pbs
proxmox-backup-manager cert info | grep -i fingerprint     # what pvesm add pbs --fingerprint wants
proxmox-backup-manager verify main --ignore-verified 0     # re-verify everything; slow
proxmox-backup-manager garbage-collection start main       # frees chunks unreferenced for >24 h

# any host with proxmox-backup-client, for example the PVE host itself
export PBS_REPOSITORY='backup@pbs@pbs.example.com:main'
export PBS_PASSWORD="$PBS_PASSWORD"                        # or PBS_FINGERPRINT and a keyfile
proxmox-backup-client backup etc.pxar:/etc --ns hosts --backup-id "$(hostname)"
proxmox-backup-client snapshot list --ns hosts
proxmox-backup-client restore host/pve1/2026-09-24T02:00:00Z etc.pxar /tmp/restore --ns hosts
proxmox-backup-client catalog shell host/pve1/2026-09-24T02:00:00Z etc.pxar --ns hosts   # browse and pick files
proxmox-backup-client map vm/100/2026-09-24T02:00:00Z drive-scsi0.img --ns pve   # exposes the disk as /dev/loopN
```

Retention on PBS is set per datastore or namespace with prune jobs; the PVE storage's `prune-backups` setting applies when `vzdump --remove 1` runs. Sync jobs pull from another PBS to make an offsite copy, and remove-vanished must be set deliberately.

## Clustering and quorum

A cluster is a group of nodes sharing `/etc/pve` through Corosync. Each node has one vote; the cluster is quorate when more than half the votes are present. With quorum lost, `pmxcfs` turns read-only, guests keep running but cannot be started, migrated or edited, and HA nodes fence themselves. Two nodes therefore cannot tolerate losing either one; add a third node or a QDevice (a small `corosync-qnetd` daemon anywhere, including a container on another host) to break ties.

Corosync needs UDP 5405 to 5412 open between nodes, latency under about 5 ms, synchronised clocks and, ideally, its own network so that storage or backup traffic cannot starve it. Up to eight links (`link0` to `link7`) give redundancy; a single link on the same NIC as Ceph is the classic way to lose quorum under load.

```sh
pvecm create my-cluster --link0 192.0.2.10 --link1 198.51.100.10      # on the first node
pvecm add 192.0.2.10 --link0 192.0.2.11 --link1 198.51.100.11         # on each new node; it must have no guests yet
pvecm status            # Quorate: Yes/No, Expected votes, Total votes, membership
pvecm nodes
corosync-cfgtool -s     # link status per ring
pvecm qdevice setup 192.0.2.5          # add a QDevice; corosync-qnetd must be installed there
pvecm delnode node3     # only after node3 is powered off and every guest moved; never boot it back into this cluster
pvecm expected 1        # DANGEROUS: tell corosync to be quorate with only this node
```

> [!WARNING] `pvecm expected 1`
> This lowers the vote requirement so a lone node can write to `/etc/pve`. Two partitions each doing it produce two divergent configurations that Corosync cannot merge. Use it only when the other nodes are known to be off, and restore the real value (`pvecm expected <n>` or a restart of corosync once nodes return).

A removed node must be reinstalled before rejoining. A node that boots with the old cluster config but is no longer a member will fight over the membership; keep it off the network until it is wiped.

## High availability

HA runs the `pve-ha-crm` (cluster resource manager, one active per cluster) and `pve-ha-lrm` (local resource manager, per node) daemons. A resource in state `started` is restarted on failure up to `max_restart` times, then relocated up to `max_relocate` times. When a node loses quorum, its LRM stops updating the watchdog and the node resets itself after about 60 seconds; the CRM then recovers its resources on other nodes about two minutes after the failure. Requirements: at least three votes, shared storage for every HA guest's disks, and a hardware watchdog or the `softdog` module (the default).

```sh
ha-manager add vm:100 --state started --max_restart 2 --max_relocate 1 --comment "app tier"
ha-manager add ct:200 --state started
ha-manager set vm:100 --state stopped         # HA keeps it stopped but still relocates it after a node failure
ha-manager set vm:100 --state disabled        # stop and leave alone
ha-manager set vm:100 --state ignored         # temporarily take it out of HA without losing the entry
ha-manager migrate vm:100 node2               # live if possible
ha-manager relocate vm:100 node2              # stop, move, start
ha-manager status
ha-manager remove vm:100
ha-manager crm-command node-maintenance enable node2    # drain a node before patching; disable afterwards
```

PVE 9 replaced HA groups with rules: node affinity (`ha-manager rules add node-affinity ...`) restricts where a resource may run and resource affinity keeps guests together or apart. `ha-manager rules list` shows what applies. Start a maintenance with `node-maintenance enable` rather than shutting the node down so that HA migrates guests away first; a plain reboot of an HA node triggers fencing-based recovery, which is slower and not graceful.

## Updating

Proxmox publishes `pve-enterprise` (subscription), `pve-no-subscription` and `pve-test` repositories. On PVE 9 they are deb822 files under `/etc/apt/sources.list.d/`. With no subscription, disable the enterprise entries (`Enabled: false` in `pve-enterprise.sources` and `ceph.sources`) or `apt update` fails with 401.

```ini
# /etc/apt/sources.list.d/pve-no-subscription.sources
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
```

```sh
apt update && apt full-upgrade          # never plain "apt upgrade": it holds back kernel and pve-manager dependency changes
pveupgrade                              # wrapper that runs the above and reports whether a reboot is needed
pveversion -v                           # every PVE package version; attach to bug reports
proxmox-boot-tool kernel list           # installed kernels; pin one with "kernel pin <version>" if a new one misbehaves
proxmox-boot-tool refresh               # re-sync ESPs after kernel or config changes on ZFS/UEFI installs
pve8to9 --full                          # major-upgrade checklist; run repeatedly until clean before following the upgrade guide
```

Within a major version, updates are safe to apply node by node: migrate guests off, update, reboot if a new kernel landed, migrate back. Across a major version (8 to 9), follow the wiki upgrade guide exactly; the checklist script catches most of what breaks. Update every node before enabling features that need the new version cluster-wide.

## The API and pvesh

Everything is `https://pve.example.com:8006/api2/json/<path>`. `pvesh` calls the same API locally as root without authentication, which makes it the quickest way to find the path and parameters you need before scripting against the remote endpoint.

```sh
pvesh get /nodes
pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.status=="running") | "\(.node)\t\(.vmid)\t\(.name)"'
pvesh get /nodes/pve1/qemu/100/status/current --output-format json-pretty
pvesh create /nodes/pve1/qemu/100/status/start
pvesh set /nodes/pve1/qemu/100/config --memory 8192
pvesh ls /nodes/pve1/storage                 # children of a path
pvesh usage /nodes/pve1/qemu/100/config -v   # every parameter the endpoint accepts
pvesh get /nodes/pve1/tasks/UPID:pve1:...:qmstart:100:root@pam:/status
```

For remote use, create an API token. `--privsep 0` gives the token the user's permissions; `--privsep 1` (default) requires separate ACLs on the token itself.

```sh
pveum user add automation@pve --comment "CI"
pveum acl modify /vms --users automation@pve --roles PVEVMAdmin
pveum acl modify /storage/local-lvm --users automation@pve --roles PVEDatastoreUser
pveum user token add automation@pve ci --privsep 0     # prints the secret once; store it, never in the repo
pveum user token permissions automation@pve ci
```

```sh
# token stored in $PVE_TOKEN as "automation@pve!ci=<uuid>"
curl -fsS -H "Authorization: PVEAPIToken=$PVE_TOKEN" https://pve.example.com:8006/api2/json/cluster/resources?type=vm | jq '.data[] | {vmid, name, status}'
curl -fsS -X POST -H "Authorization: PVEAPIToken=$PVE_TOKEN" https://pve.example.com:8006/api2/json/nodes/pve1/qemu/100/status/shutdown
```

POST, PUT and DELETE return a UPID; poll `/nodes/<node>/tasks/<upid>/status` until `status` is `stopped` and read `exitstatus`. Ticket authentication (`POST /access/ticket`) also works for scripts but requires a CSRF token header on writes and expires after two hours. The API viewer at `https://pve.example.com:8006/pve-docs/api-viewer/` documents every path. Terraform (`bpg/proxmox`) and Ansible (`community.proxmox`) both use tokens this way; see [Terraform](https://www.wiki.jodisand.me/terraform/) and [Ansible](https://www.wiki.jodisand.me/ansible/).

## Oneliners

```sh
# Every guest in the cluster with node, status and memory
pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | [.node, .vmid, .name, .status, (.maxmem/1073741824|floor)] | @tsv' | sort -k1,1 -k2,2n

# Guests using more than 80 % of their allocated memory (needs the guest agent for VMs)
pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.status=="running" and .mem/.maxmem > 0.8) | "\(.vmid)\t\(.name)\t\(.mem*100/.maxmem|floor)%"'

# Which guests live on a storage
pvesm list ceph | awk 'NR>1 {print $NF}' | sort -n | uniq -c

# Disks on storage that no config references (candidates for cleanup, check first)
qm disk rescan --dryrun 1

# Start every stopped VM tagged "autostart" (tags come from qm set --tags)
for id in $(pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.status=="stopped" and (.tags//"" | test("autostart"))) | .vmid'); do qm start "$id"; done

# Shut down every VM on this node gracefully, 5 minutes each, then kill
for id in $(qm list | awk 'NR>1 && $3=="running" {print $1}'); do qm shutdown "$id" --timeout 300 --forceStop 1; done

# Migrate every guest off this node to node2 (containers restart)
for id in $(qm list | awk 'NR>1 {print $1}'); do qm migrate "$id" node2 --online; done; for id in $(pct list | awk 'NR>1 {print $1}'); do pct migrate "$id" node2 --restart; done

# Guest IP addresses via the agent
qm guest cmd 100 network-get-interfaces | jq -r '.[] | .name as $n | .["ip-addresses"][]? | select(.["ip-address-type"]=="ipv4") | "\($n)\t\(.["ip-address"])"'

# Which node owns VM 100
pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.vmid==100) | .node'

# Locked guests
grep -l '^lock:' /etc/pve/qemu-server/*.conf /etc/pve/lxc/*.conf 2>/dev/null

# Guests with snapshots, and the snapshot names
grep -H '^\[' /etc/pve/qemu-server/*.conf | sed 's/.*\/\([0-9]*\)\.conf:\[\(.*\)\]/\1 \2/'

# Failed tasks in the last day, cluster-wide
pvesh get /cluster/tasks --output-format json | jq -r --arg t "$(date -d '1 day ago' +%s)" '.[] | select(.starttime > ($t|tonumber) and .status != "OK" and .status != null) | "\(.starttime|todate)\t\(.node)\t\(.type)\t\(.id)\t\(.status)"'

# Backup jobs and their schedules
pvesh get /cluster/backup --output-format json | jq -r '.[] | "\(.id)\t\(.schedule)\t\(.storage)\t\(.vmid // "all")"'

# Latest backup per guest on a storage
pvesm list pbs --content backup | awk 'NR>1 {print $1}' | sort | awk -F'[/:]' '{last[$3]=$0} END {for (v in last) print last[v]}'

# Storage that is more than 85 % full
pvesm status | awk 'NR>1 && $7+0 > 85 {print $1, $7"%"}'

# ZFS ARC size and the configured limit
awk '/^size|^c_max/ {printf "%s %.1f GiB\n", $1, $3/1073741824}' /proc/spl/kstat/zfs/arcstats

# Corosync ring health
corosync-cfgtool -s && corosync-quorumtool -s

# The QEMU process for VM 100 and its threads' CPU use
top -H -p "$(cat /var/run/qemu-server/100.pid)" -bn1 | head -20

# Set a tag and a note on a guest
qm set 100 --tags prod,web --description "Owner: platform team"

# Regenerate the cloud-init drive for every clone of template 9000
for id in $(qm list | awk 'NR>1 {print $1}'); do qm config "$id" | grep -q '^ide2:.*cloudinit' && qm cloudinit update "$id"; done

# Temporarily disable HA for a guest, do work, re-enable
ha-manager set vm:100 --state ignored; qm shutdown 100; ...; ha-manager set vm:100 --state started
```

## Scripts

Report every guest that has no backup on the given storage within the last N days.

```sh
#!/usr/bin/env bash
# usage: stale-backups.sh <storage> [days]
set -euo pipefail
storage=${1:?storage id required}
days=${2:-2}
cutoff=$(date -d "$days days ago" +%s)

declare -A latest
while read -r volid _ _ _ ctime _; do
  vmid=$(printf '%s' "$volid" | sed -E 's#.*/(vm|ct)/([0-9]+)/.*#\2#; s#.*vzdump-(qemu|lxc)-([0-9]+)-.*#\2#')
  [[ ${latest[$vmid]:-0} -lt $ctime ]] && latest[$vmid]=$ctime
done < <(pvesh get "/nodes/$(hostname)/storage/$storage/content" --content backup --output-format json \
          | jq -r '.[] | "\(.volid) x x x \(.ctime) x"')

rc=0
while IFS=$'\t' read -r vmid name node; do
  last=${latest[$vmid]:-0}
  if (( last < cutoff )); then
    printf '%s\t%s\t%s\tlast backup: %s\n' "$vmid" "$name" "$node" "$( (( last )) && date -d "@$last" +%F || echo never)"
    rc=1
  fi
done < <(pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.template != 1) | [.vmid, .name, .node] | @tsv')
exit "$rc"
```

Drain a node for maintenance: put it in HA maintenance mode, migrate the rest, and confirm nothing is left running.

```sh
#!/usr/bin/env bash
# usage: drain-node.sh <target-node>
set -euo pipefail
target=${1:?target node required}
self=$(hostname)

ha-manager crm-command node-maintenance enable "$self"
for id in $(qm list | awk 'NR>1 && $3=="running" {print $1}'); do
  if qm config "$id" | grep -q '^lock:'; then printf 'skipping locked VM %s\n' "$id" >&2; continue; fi
  qm migrate "$id" "$target" --online --with-local-disks || printf 'VM %s failed\n' "$id" >&2
done
for id in $(pct list | awk 'NR>1 && $2=="running" {print $1}'); do
  pct migrate "$id" "$target" --restart || printf 'CT %s failed\n' "$id" >&2
done
remaining=$( { qm list | awk 'NR>1 && $3=="running"'; pct list | awk 'NR>1 && $2=="running"'; } | wc -l)
printf '%s guests still running on %s\n' "$remaining" "$self"
(( remaining == 0 ))
```

Create a VM from the cloud-init template with an idempotent check, for use from CI with an API token.

```python
#!/usr/bin/env python3
"""Clone a template if the VMID does not exist yet. Needs PVE_HOST, PVE_TOKEN (user@realm!id=secret)."""
import os, sys, time, requests

host, token = os.environ["PVE_HOST"], os.environ["PVE_TOKEN"]
node, template, vmid, name, ip = "pve1", 9000, int(sys.argv[1]), sys.argv[2], sys.argv[3]
s = requests.Session()
s.headers["Authorization"] = f"PVEAPIToken={token}"
s.verify = os.environ.get("PVE_CA", True)
base = f"https://{host}:8006/api2/json"

def wait(upid):
    while True:
        st = s.get(f"{base}/nodes/{node}/tasks/{upid}/status", timeout=30).json()["data"]
        if st["status"] == "stopped":
            if st["exitstatus"] != "OK":
                sys.exit(f"task failed: {st['exitstatus']}")
            return
        time.sleep(2)

existing = {v["vmid"] for v in s.get(f"{base}/cluster/resources", params={"type": "vm"}, timeout=30).json()["data"]}
if vmid in existing:
    print(f"{vmid} exists, nothing to do"); sys.exit(0)

wait(s.post(f"{base}/nodes/{node}/qemu/{template}/clone", data={"newid": vmid, "name": name, "full": 1}, timeout=30).json()["data"])
s.put(f"{base}/nodes/{node}/qemu/{vmid}/config", data={"ipconfig0": f"ip={ip}/24,gw=192.0.2.1", "tags": "ci"}, timeout=30).raise_for_status()
wait(s.post(f"{base}/nodes/{node}/qemu/{vmid}/status/start", timeout=30).json()["data"])
print(f"{vmid} started")
```

## Troubleshooting

Task logs are the first stop: `pvesh get /nodes/$(hostname)/tasks/<UPID>/log`, or the `Tasks` panel. Daemon logs are in the journal: `journalctl -u pvedaemon -u pveproxy -u pve-cluster -u corosync -u pve-ha-lrm -u pve-ha-crm --since -1h` (see [systemd](https://www.wiki.jodisand.me/systemd/)).

| Symptom | Cause | Fix |
| --- | --- | --- |
| `VM is locked (backup)` or `(snapshot)` | A task died mid-way, often a backup interrupted by reboot | Confirm no vzdump or qemu-img process holds it (`ps aux \| grep -E 'vzdump\|qemu-img'`), then `qm unlock 100` |
| `start failed: QEMU exited with code 1` | Read the task log: missing ISO, storage not active, PCI device busy, `kvm: failed to initialize KVM` | `qm showcmd 100 --pretty` and run the printed command by hand to see QEMU's own message |
| `TASK ERROR: can't lock file '/var/lock/qemu-server/lock-100.conf'` | Another `qm` operation on the same VM is running or hung | `fuser /var/lock/qemu-server/lock-100.conf`; wait or kill the stuck process |
| `cluster not ready - no quorum? (500)` | Fewer than half the votes present | `pvecm status`, `corosync-cfgtool -s`, fix the network; as a last resort on a lone surviving node `pvecm expected 1` |
| `/etc/pve` read-only, `pmxcfs` errors in the journal | Same as above, or `pve-cluster` stopped | `systemctl restart pve-cluster corosync`; check `journalctl -u pve-cluster` for database corruption |
| Node fenced (rebooted itself) | HA LRM lost quorum for 60 s: corosync network saturated or flapping | Separate corosync link, add `link1`; `corosync-cfgtool -s` for link errors |
| Storage `inactive` or `unknown` in `pvesm status` | NFS/CIFS server unreachable, iSCSI portal down, ZFS pool not imported | `pvesm status --storage nas`, `journalctl -u pvestatd`, `mount \| grep nas`, `zpool import` |
| `local-lvm` full but guests are small | Thin pool over-provisioned and guests never trimmed | `lvs -a pve` for `Data%`; enable `discard=on` and run `fstrim -av` in guests, `pct fstrim` for containers; delete snapshots |
| Root filesystem full on a node | Backups written to `/var/lib/vz` or a directory storage whose mount failed | `du -xsh /var/lib/vz/dump /mnt/*`; set `is_mountpoint 1`; move backups elsewhere |
| Slow disk I/O in a VM | Cache mode, no iothread, VirtIO not used, host swapping, ZFS ARC starving guests | Use `virtio-scsi-single` with `iothread=1`, `cache=none` on block storage; check `free -g`, `arcstat`; set `zfs_arc_max` in `/etc/modprobe.d/zfs.conf` |
| `qm migrate` fails with CPU flag errors | Source VM uses `--cpu host` or a model the target lacks | Set a common model (`x86-64-v3`), stop and start, migrate |
| Migration fails with `storage ... is not available on node` | Storage definition restricted with `nodes`, or a local storage | `--targetstorage`, or extend the storage's `nodes` list |
| Container will not start after host upgrade | Old distribution needs cgroup v1, or nesting missing for systemd | `pct start 200 --debug`; `lxc-start -n 200 -F -l DEBUG -o /tmp/lxc-200.log`; enable `nesting=1` |
| Bind mount files show as `nobody:nogroup` | Unprivileged UID mapping | `chown 100000:100000` on the host, or an `lxc.idmap` entry |
| Web UI unreachable on 8006, SSH works | `pveproxy` down, or certificate expired | `systemctl status pveproxy`, `pvecm updatecerts --force`, `systemctl restart pveproxy` |
| `apt update` returns 401 Unauthorized | Enterprise repository enabled without a subscription | Set `Enabled: false` in `pve-enterprise.sources` and `ceph.sources` |
| Guest has no network after clone | Same MAC as the template, or cloud-init not regenerated | `qm set 100 --net0 virtio,bridge=vmbr0` regenerates a MAC; `qm cloudinit update 100` |

For a VM that hangs at boot, `qm terminal 100` (requires `--serial0 socket`) shows the kernel console; for a VM with a graphical console, `qm vncproxy` from the UI is the only option. A guest process stuck in D state on the host usually points at the storage backend; `dmesg -T | tail` and `iostat -x 5` on the host narrow it down ([linux-performance](https://www.wiki.jodisand.me/linux-performance/)).

## Further reading

- [Proxmox VE Administration Guide](https://pve.proxmox.com/pve-docs/pve-admin-guide.html)
- [qm(1)](https://pve.proxmox.com/pve-docs/qm.1.html), [pct(1)](https://pve.proxmox.com/pve-docs/pct.1.html), [pvesm(1)](https://pve.proxmox.com/pve-docs/pvesm.1.html), [vzdump(1)](https://pve.proxmox.com/pve-docs/vzdump.1.html)
- [Cluster Manager](https://pve.proxmox.com/pve-docs/chapter-pvecm.html) and [High Availability](https://pve.proxmox.com/pve-docs/chapter-ha-manager.html)
- [Software-Defined Network](https://pve.proxmox.com/pve-docs/chapter-pvesdn.html)
- [Proxmox VE API](https://pve.proxmox.com/wiki/Proxmox_VE_API) and the [API viewer](https://pve.proxmox.com/pve-docs/api-viewer/)
- [Proxmox Backup Server documentation](https://pbs.proxmox.com/docs/)