Software Engineering WikiSE Wiki

Disks and storage

Partition, format, mount, encrypt and grow Linux block storage with lsblk, parted, LVM, ext4, XFS, Btrfs, mdadm and cryptsetup, and diagnose full or slow disks.

Reviewed MarkdownEdit

On this page

Cheatsheet#

TaskCommand
Block devices as a tree with filesystemslsblk -f
Sizes, models and mountpointslsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL
UUIDs and labelsblkid
Free space per mounted filesystemdf -hT -x tmpfs -x devtmpfs
Inode usagedf -i
What is filling this directorydu -xh --max-depth=1 / | sort -h
Partition tableparted /dev/sdb print, sgdisk -p /dev/sdb
Create a GPT and one full-size partitionparted -s /dev/sdb mklabel gpt mkpart data ext4 1MiB 100%
Make a filesystemmkfs.ext4 -L data /dev/sdb1, mkfs.xfs -L data /dev/sdb1
Mount by labelmount LABEL=data /mnt/data
Mount everything in fstab, verify itmount -a && findmnt --verify
Grow a partition to fill the diskgrowpart /dev/sda 3
Grow the filesystemresize2fs /dev/sda3 (ext4), xfs_growfs / (XFS, by mountpoint)
LVM overviewpvs; vgs; lvs -a -o +devices
Extend a logical volume and its filesystemlvextend -r -L +20G /dev/vg0/data
Disk healthsmartctl -a /dev/sda
Per-device IO utilisationiostat -xz 1
Which process is doing IOiotop -oPa
Trim an SSD or thin volumefstrim -av
Discard the page cache, for benchmarks onlysync; echo 3 > /proc/sys/vm/drop_caches
Deleted but open files holding spacelsof +L1
Filesystem check (unmounted)fsck.ext4 -f /dev/sdb1, xfs_repair /dev/sdb1

Commands assume util-linux 2.40, LVM2 2.03, e2fsprogs 1.47, xfsprogs 6.x, btrfs-progs 6.x, cryptsetup 2.7 and smartmontools 7.4 as shipped on Fedora 42 and RHEL 9. Most of them need root. References: the util-linux manual pages and the Red Hat storage guide.

The stack#

A disk is a block device (/dev/sda, /dev/nvme0n1, /dev/vda). A partition table divides it into partitions (/dev/sda1, /dev/nvme0n1p1). Optionally LVM takes partitions or whole disks as physical volumes, pools them in a volume group and carves logical volumes (/dev/vg0/data, also /dev/mapper/vg0-data). Optionally LUKS wraps any of those in encryption and exposes a /dev/mapper/name device. A filesystem sits on the final device and is mounted at a path. Each layer must be grown in order from the bottom up when a disk gets bigger, and shrunk from the top down.

lsblk -f                                   # tree: disk > partition > crypt > lvm > filesystem, with UUIDs
lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,UUID,MOUNTPOINTS,MODEL,SERIAL,ROTA,DISC-GRAN
lsblk -d -o NAME,SIZE,ROTA,TRAN,MODEL      # disks only; ROTA 1 is spinning, TRAN is sata/nvme/virtio
lsblk -J | jq '.blockdevices[] | select(.type == "disk") | .name'   # JSON for scripts
blkid                                      # UUID, LABEL and TYPE for every filesystem, swap and LUKS header
blkid -s UUID -o value /dev/sda2           # one value, for fstab
findmnt                                    # mounted filesystems as a tree with options
findmnt -T /var/lib/containers             # which mount holds this path
findmnt --verify                           # parse fstab and report problems without mounting
df -hT -x tmpfs -x devtmpfs -x overlay     # usage per filesystem, without the noise
df -i                                      # inode usage; a full inode table also reports "No space left"
cat /proc/partitions; ls -l /dev/disk/by-id/ /dev/disk/by-uuid/ /dev/disk/by-path/
udevadm info -q property /dev/sda | grep -E 'ID_SERIAL|ID_WWN|ID_BUS'

Device names such as /dev/sdb change between boots when disks are added, removed or enumerated in a different order. Refer to filesystems by UUID= or LABEL= in fstab and to disks by /dev/disk/by-id/ in scripts and RAID or LVM configuration.

Partitioning#

GPT is the default on anything current: up to 128 partitions, 64-bit sector addresses (disks over 2 TiB), a backup header at the end of the disk and a checksum. MBR (msdos in parted) remains only for legacy BIOS boot on old images and for compatibility with old firmware.

parted scripts well and understands GPT and MBR. sgdisk is GPT-only, scripts even better and can clone tables. fdisk is interactive by default and fine for a one-off. cfdisk is a curses front end. All of them write a table only, never a filesystem; the kernel re-reads the table when they exit, and partprobe /dev/sdb forces it if a partition is in use.

parted /dev/sdb print                             # table type, size, partitions with flags
parted /dev/sdb unit s print                      # in sectors, to check alignment
parted -s /dev/sdb mklabel gpt                    # DESTROYS the existing table and, in effect, every partition on it
parted -s -a optimal /dev/sdb mkpart data ext4 1MiB 100%       # one partition; "ext4" here only sets the type hint, mkfs still needed
parted -s -a optimal /dev/sdb mkpart efi fat32 1MiB 1025MiB set 1 esp on   # EFI system partition
parted -s /dev/sdb mkpart lvm 1025MiB 100% set 2 lvm on
parted -s /dev/sdb resizepart 2 100%              # grow partition 2 to the end; parted 3.x asks nothing with -s
parted -s /dev/sdb rm 3                           # DESTRUCTIVE: removes partition 3 from the table
parted -s /dev/sdb align-check optimal 1          # "1 aligned"

Start the first partition at 1 MiB. That aligns to every common physical sector and erase-block size, and it is what -a optimal does when you give sizes in MiB or percent. Starting at sector 63 or any odd number of 512-byte sectors on a 4 KiB-sector disk halves write performance.

sgdisk -p /dev/sdb                                # print table
sgdisk -o /dev/sdb                                # DESTRUCTIVE: new empty GPT
sgdisk -n 1:0:+1G -t 1:ef00 -c 1:efi /dev/sdb     # partition 1, 1 GiB from the first free sector, type EFI, name efi
sgdisk -n 2:0:0 -t 2:8e00 -c 2:lvm /dev/sdb       # partition 2, rest of disk, type Linux LVM
sgdisk -L | grep -iE 'linux|efi|lvm|raid|swap'    # type codes: 8300 Linux fs, 8e00 LVM, fd00 RAID, 8200 swap, ef00 ESP
sgdisk -d 2 /dev/sdb                              # DESTRUCTIVE: delete partition 2
sgdisk -e /dev/sdb                                # move the backup GPT header to the actual end of a grown disk
sgdisk -G /dev/sdb                                # randomise disk and partition GUIDs after cloning
sgdisk -R /dev/sdc /dev/sdb                       # replicate sdb's table onto sdc (for RAID mirrors), then sgdisk -G /dev/sdc
sgdisk -b table.bin /dev/sdb; sgdisk -l table.bin /dev/sdb   # back up and restore a table
sgdisk -Z /dev/sdb                                # DESTRUCTIVE: zap GPT and MBR structures

fdisk -l prints every disk’s table. Interactive fdisk /dev/sdb uses g (new GPT), n (new), t (type), d (delete), p (print), w (write) and q (quit without writing). wipefs -a /dev/sdb erases filesystem, RAID, LVM and partition-table signatures so a reused disk does not get auto-assembled or auto-mounted; it is destructive and there is a --no-act flag to preview.

Filesystems#

ext4XFSBtrfs
Default onDebian, UbuntuRHEL, Fedora ServerFedora Workstation, openSUSE
Grow onlineYesYesYes
ShrinkOffline onlyNeverOnline
SnapshotsVia LVMVia LVMNative, subvolume-level
ChecksumsMetadataMetadataData and metadata
InodesFixed at mkfsDynamicDynamic
StrengthsMature, fsck recovers well, small filesLarge files, parallel IO, huge filesystemsSnapshots, send/receive, compression, RAID 1
mkfs.ext4 -L data /dev/sdb1                       # DESTRUCTIVE: writes a filesystem over whatever is there
mkfs.ext4 -L data -m 0.5 -T largefile4 /dev/sdb1  # -m reserved blocks % (default 5, meant for root); -T tunes inode ratio
mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/sdb1   # do the init now rather than in the background after mount
mkfs.xfs -L data /dev/sdb1                        # DESTRUCTIVE; refuses to overwrite an existing filesystem without -f
mkfs.xfs -L data -m reflink=1 -d su=64k,sw=4 /dev/md0   # stripe unit and width for a 4-data-disk RAID
mkfs.btrfs -L data /dev/sdb1                      # DESTRUCTIVE
mkfs.btrfs -L data -m raid1 -d raid1 /dev/sdb /dev/sdc   # two-disk mirror with no mdadm
mkfs.vfat -F32 -n EFI /dev/sdb1                   # EFI system partition

Inspect and tune:

tune2fs -l /dev/sdb1                              # ext4 superblock: features, mount count, last check, reserved blocks
tune2fs -L newlabel /dev/sdb1                     # relabel (ext4; xfs_admin -L for XFS, btrfs filesystem label for Btrfs)
tune2fs -m 1 /dev/sdb1                            # reduce reserved blocks to 1%
tune2fs -O ^has_journal /dev/sdb1                 # remove the journal (unmounted); rarely worth it
dumpe2fs -h /dev/sdb1 | grep -iE 'block size|inode count|free'
xfs_info /mnt/data                                # XFS geometry: block size, agcount, sunit/swidth, reflink
xfs_admin -L data /dev/sdb1                       # label (unmounted)
btrfs filesystem show; btrfs filesystem usage /mnt/data; btrfs device stats /mnt/data

Resize. Grow the underlying device first (partition, LV or virtual disk), then the filesystem. ext4 and XFS grow while mounted; ext4 shrinks only unmounted, XFS never shrinks.

resize2fs /dev/sdb1                               # ext4: grow to fill the device, online
resize2fs /dev/sdb1 50G                           # ext4: to a size; shrinking requires umount and e2fsck -f first
xfs_growfs /mnt/data                              # XFS: takes the mountpoint, grows to fill the device
xfs_growfs -D 26214400 /mnt/data                  # XFS: to a size in filesystem blocks
btrfs filesystem resize max /mnt/data             # Btrfs: grow to fill; also accepts -10G to shrink online
btrfs filesystem resize 2:max /mnt/data           # Btrfs: device ID 2 in a multi-device filesystem

Check and repair, unmounted, or on a snapshot of a mounted volume. fsck never runs on a mounted read-write filesystem.

e2fsck -f /dev/sdb1                               # force a full check; -p auto-fixes safe problems, -y answers yes to everything
e2fsck -n /dev/sdb1                               # read-only check of a mounted filesystem; results may be misleading
xfs_repair -n /dev/sdb1                           # dry run
xfs_repair /dev/sdb1                              # repair; if it complains about the log, mount and umount once first, -L zeroes it (loses recent metadata)
btrfs check --readonly /dev/sdb1                  # check; --repair is a last resort, ask on the mailing list first
btrfs scrub start -B /mnt/data                    # verify every checksum on a mounted filesystem; -B waits

Btrfs subvolumes and snapshots#

A Btrfs subvolume is an independently mountable tree inside the filesystem. Fedora installs / and /home as subvolumes named root and home on one filesystem. A snapshot is a copy-on-write clone of a subvolume and costs nothing until data diverges.

btrfs subvolume list /                            # subvolumes with IDs and paths
btrfs subvolume create /mnt/data/projects
btrfs subvolume snapshot -r /home /home/.snapshots/home-$(date +%F)   # -r: read-only, needed for send
btrfs subvolume delete /home/.snapshots/home-2026-08-01
btrfs send /home/.snapshots/home-2026-09-24 | ssh backup.example.com btrfs receive /backup/home   # full copy
btrfs send -p /home/.snapshots/home-2026-09-23 /home/.snapshots/home-2026-09-24 | ssh backup.example.com btrfs receive /backup/home   # incremental
btrfs property set /mnt/data/vm-images compression none   # or mount -o compress=zstd:3
btrfs filesystem defragment -r -czstd /mnt/data/docs      # compress existing files in place
btrfs balance start -dusage=50 /mnt/data                  # reclaim half-empty data chunks; fixes ENOSPC with df showing free space

LVM#

LVM inserts a mapping layer between block devices and filesystems. Physical volumes (PV) join a volume group (VG), and logical volumes (LV) are allocated from the VG’s extents (4 MiB each by default). LVs grow while in use, can span disks, and can be snapshotted. Metadata lives on every PV, so a VG assembles on any host that sees its disks.

pvs; vgs; lvs                                      # summaries; -v for more, -a includes hidden volumes
pvs -o +pv_used; vgs -o +vg_free_count; lvs -a -o +devices,segtype   # where each LV's extents live
pvdisplay /dev/sdb1; vgdisplay vg0; lvdisplay /dev/vg0/data          # verbose forms
lvs -o lv_name,lv_size,data_percent,metadata_percent,snap_percent    # fill levels of thin pools and snapshots
pvcreate /dev/sdb1                                 # label a partition as a PV; refuses if a filesystem signature exists (wipefs first)
vgcreate vg0 /dev/sdb1 /dev/sdc1                   # VG from two PVs
lvcreate -n data -L 100G vg0                       # LV of 100 GiB
lvcreate -n data -l 100%FREE vg0                   # all remaining extents
lvcreate -n data -l 50%VG vg0                      # half the VG
lvcreate -n fast -L 50G -i 2 -I 64 vg0             # striped across 2 PVs, 64 KiB stripes
lvcreate -n mirror -L 50G -m 1 vg0                 # RAID 1 via the raid1 segment type
mkfs.xfs /dev/vg0/data && mount /dev/vg0/data /mnt/data

Grow: add a PV if the VG is full, then extend the LV with -r so the filesystem grows in the same step.

pvcreate /dev/sdd1 && vgextend vg0 /dev/sdd1       # add a disk to the VG
pvresize /dev/sdb1                                 # after the partition or virtual disk underneath grew
lvextend -r -L +20G /dev/vg0/data                  # add 20 GiB and resize the filesystem (ext4, XFS, Btrfs)
lvextend -r -l +100%FREE /dev/vg0/data             # use everything left
lvextend -r -L 200G /dev/vg0/data                  # to an absolute size
lvreduce -r -L 50G /dev/vg0/data                   # DESTRUCTIVE if the filesystem is larger than 50G; ext4 only, unmounts and shrinks first with -r; XFS cannot shrink
lvremove /dev/vg0/old                              # DESTRUCTIVE: deletes the LV and its data after a prompt
vgreduce vg0 /dev/sdb1                             # remove an empty PV from the VG (pvmove first if it holds extents)
pvmove /dev/sdb1                                   # migrate extents off a PV online, then vgreduce and pvremove
vgrename vg0 vg1; lvrename vg1 data data-old
vgchange -an vg0; vgexport vg0                     # deactivate and mark for moving to another host; vgimport there
vgscan; vgchange -ay                               # find and activate VGs after adding disks

Snapshots are copy-on-write. A classic snapshot needs its own space in the VG to hold the changed blocks of the origin; when it fills, the snapshot becomes invalid (the origin is unaffected). Size it for the writes expected during its life, not the size of the origin.

lvcreate -s -n data-snap -L 10G /dev/vg0/data      # snapshot of data with 10 GiB for changed blocks
lvs -o lv_name,origin,snap_percent                 # watch the fill level
mount -o ro,nouuid /dev/vg0/data-snap /mnt/snap    # XFS needs nouuid because the UUID matches the origin
lvconvert --merge /dev/vg0/data-snap               # roll the origin back to the snapshot; happens at next activation if the origin is mounted; removes the snapshot
lvremove /dev/vg0/data-snap                        # discard the snapshot

Thin provisioning allocates blocks on write from a pool and makes snapshots cheap and unlimited in size, at the cost of a pool that can overfill and take every LV in it offline. Monitor data_percent; lvm.conf thin_pool_autoextend_threshold grows the pool automatically.

lvcreate -L 500G -T vg0/pool                       # thin pool
lvcreate -V 1T -T vg0/pool -n vm-disks             # thin LV bigger than the pool
lvcreate -s -n vm-disks-snap vg0/vm-disks          # thin snapshot; no size needed

/etc/lvm/backup/ and /etc/lvm/archive/ hold metadata backups after every change; vgcfgrestore -l vg0 lists them and vgcfgrestore -f FILE vg0 restores one, which is how an accidental lvremove is undone before any data is overwritten.

fstab and mount options#

/etc/fstab declares what mounts at boot. systemd generates a .mount unit per line, orders local-fs.target after them, and a failing line without nofail drops the boot into emergency mode.

# <device>                                  <mountpoint>   <type>  <options>                                   <dump> <pass>
UUID=3f1a2b4c-0d5e-4f6a-8b7c-9d0e1f2a3b4c   /              xfs     defaults                                    0 0
UUID=A1B2-C3D4                              /boot/efi      vfat    umask=0077,shortname=winnt                  0 2
/dev/mapper/vg0-data                        /srv/data      ext4    defaults,noatime,nofail,x-systemd.device-timeout=10s  0 2
LABEL=backup                                /mnt/backup    xfs     noauto,nofail,x-systemd.automount,x-systemd.idle-timeout=10min  0 0
nas.example.com:/export/media               /mnt/media     nfs4    _netdev,nofail,soft,timeo=150,retrans=3,noatime  0 0
tmpfs                                       /var/tmp/build tmpfs   size=4G,mode=1777,nosuid,nodev              0 0
/swapfile                                   none           swap    defaults                                    0 0

Options that matter:

OptionEffect
defaultsrw,suid,dev,exec,auto,nouser,async
noatimeDo not update access times on read; safe for almost everything and removes a write per read. relatime (kernel default) updates once a day
nofailBoot continues if the device is missing; combine with x-systemd.device-timeout= so it does not wait 90 s
noautoNot mounted by mount -a or at boot; pair with x-systemd.automount to mount on first access
_netdevNetwork filesystem: wait for the network, unmount before it goes down
nosuid,nodev,noexecHarden /tmp, /var/tmp, /home, removable media and data mounts
roRead-only
discardIssue TRIM on every delete (online discard); a fstrim.timer is usually better
x-systemd.requires-mounts-for=Order after another mount, for nested paths
pass 0Never fsck at boot; 1 for root, 2 for the rest on ext4; XFS and Btrfs ignore it
mount -a                                           # mount everything in fstab not yet mounted; the way to test a new line
findmnt --verify                                   # syntax and device existence check without mounting
systemctl daemon-reload                            # regenerate mount units after editing fstab (systemd asks for this)
mount -o remount,ro /srv/data                      # change options on a mounted filesystem
mount --bind /srv/data/www /var/www                # bind mount: same filesystem at a second path
mount -o bind,ro /srv/data/www /var/www            # read-only bind needs a remount step on old kernels; one step on 5.x+
mount -t tmpfs -o size=1G tmpfs /mnt/scratch
mount -o loop image.iso /mnt/iso                   # loop device for an image; losetup -f --show image.img for a raw disk image
losetup -Pf --show disk.img                        # -P scans partitions, giving /dev/loop0p1
umount /mnt/data; umount -l /mnt/data              # -l: lazy, detach now and clean up when no longer busy
systemd-mount /dev/sdb1 /mnt/usb; systemd-umount /mnt/usb   # transient mount unit with automatic dependency handling

A mount unit name is the path with / replaced by -: /srv/data is srv-data.mount. systemctl status srv-data.mount and journalctl -u srv-data.mount show why a mount failed at boot. Writing a .mount unit instead of an fstab line is equivalent; fstab is simpler to review.

Swap#

Swap gives the kernel somewhere to put anonymous pages under memory pressure. Fedora uses zram (compressed RAM) by default; servers commonly add a swap file or partition sized at a few GiB regardless of RAM so that a leak degrades gracefully instead of triggering the OOM killer.

swapon --show; free -h                             # current swap devices and usage
fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile   # ext4 and XFS; Btrfs needs chattr +C and no snapshots on the file
dd if=/dev/zero of=/swapfile bs=1M count=4096 status=progress   # alternative when fallocate is refused
mkswap -L swap /dev/vg0/swap && swapon /dev/vg0/swap            # swap on an LV, growable with lvextend then swapoff/mkswap/swapon
swapoff /swapfile                                  # pages back into RAM; fails if RAM cannot hold them
sysctl vm.swappiness; sysctl -w vm.swappiness=10   # 60 default; lower prefers dropping cache over swapping; persist in /etc/sysctl.d/
zramctl                                            # zram devices and compression ratio

Hibernate needs swap at least the size of RAM and resume= on the kernel command line. Swap on a thin LV or a sparse file is unsafe: the kernel cannot allocate blocks under memory pressure.

SMART#

Drives report their own health through SMART. smartctl reads it; smartd polls and emails or logs when attributes change.

smartctl -i /dev/sda                               # identity: model, serial, firmware, sector sizes, SMART support
smartctl -H /dev/sda                               # overall health: PASSED or FAILED (a pass means little; look at the attributes)
smartctl -a /dev/sda                               # everything: attributes, error log, self-test log
smartctl -a /dev/nvme0                             # NVMe: percentage_used, available_spare, media_errors, unsafe_shutdowns
smartctl -A /dev/sda | grep -E 'Reallocated_Sector|Current_Pending|Offline_Uncorrectable|UDMA_CRC|Power_On_Hours|Temperature'
smartctl -t short /dev/sda; smartctl -t long /dev/sda   # self-tests; check with -l selftest after the time it reports
smartctl -l error /dev/sda                         # ATA error log
smartctl -d sat /dev/sdX; smartctl -d megaraid,0 /dev/sda   # behind a USB bridge or RAID controller
smartctl --scan                                    # devices and the -d type to use

The attributes that predict failure on spinning disks are Reallocated_Sector_Ct (5), Current_Pending_Sector (197) and Offline_Uncorrectable (198): any non-zero raw value means the disk has already lost data or is about to, and a rising count means replace it now. UDMA_CRC_Error_Count (199) is a cable or backplane problem, not the disk. On NVMe, Percentage Used over 100 and Available Spare under threshold mean end of life; Media and Data Integrity Errors should stay at zero. smartd runs from smartmontools with /etc/smartmontools/smartd.conf; DEVICESCAN -a -o on -S on -s (S/../.././02|L/../../6/03) -m root -M exec /usr/libexec/smartmontools/smartdnotify scans every disk, runs a short test nightly and a long test weekly, and notifies on change.

IO performance#

iostat -xz 1                                       # per device each second, skip idle ones: r/s w/s rMB/s wMB/s r_await w_await aqu-sz %util
iostat -xzd 5 3 nvme0n1                            # one device, three samples five seconds apart
iotop -oPa                                         # processes doing IO (-o only active, -P processes not threads, -a accumulated)
pidstat -d 1                                       # per-process read/write rate
vmstat 1                                           # b (blocked on IO), bi/bo, wa (iowait %)
cat /sys/block/sda/queue/scheduler                 # [mq-deadline] none bfq kyber; none for NVMe, mq-deadline for SATA SSD, bfq for desktop HDD
cat /sys/block/sda/queue/rotational /sys/block/sda/queue/discard_granularity
blockdev --getra /dev/sda; blockdev --setra 4096 /dev/sda   # read-ahead in 512-byte sectors
hdparm -tT /dev/sda                                # crude sequential read benchmark (reads only)
fio --name=randread --filename=/mnt/data/fio.test --size=4G --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --direct=1 --runtime=30 --time_based --group_reporting   # IOPS test; creates a 4 GiB file
dd if=/dev/zero of=/mnt/data/dd.test bs=1M count=4096 oflag=direct status=progress   # sequential write; creates a 4 GiB file

%util near 100 on a spinning disk is saturation; on an NVMe with many queues it is not, so look at aqu-sz and await instead. r_await and w_await are the latency the applications see: over 10 ms on an SSD or over 30 ms on a disk means queueing. Bandwidth without high await is a healthy busy disk. See Linux performance for the wider method.

TRIM tells an SSD or a thin LV which blocks are free. Weekly fstrim.timer (enabled by default on Fedora and RHEL) is preferable to the discard mount option, which issues small discards synchronously on every delete.

fstrim -av                                         # trim every mounted filesystem that supports it; prints bytes trimmed
systemctl enable --now fstrim.timer; systemctl list-timers fstrim.timer
lsblk -D                                           # DISC-GRAN and DISC-MAX: 0B means the device does not accept discards

For a VM, the virtual disk must be attached with discard=unmap (virtio-scsi or virtio-blk on QEMU 4.0+) for fstrim in the guest to shrink a thin image or thin LV on the host.

mdadm#

Software RAID from disks or partitions. Use partitions of type fd00 (or linux_raid_member autodetection) so a replacement disk can be partitioned identically.

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1     # DESTRUCTIVE to those partitions; mirror
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]1            # RAID 5 (one disk of parity)
mdadm --create /dev/md0 --level=6 --raid-devices=4 --chunk=512 /dev/sd[bcde]1   # RAID 6 (two disks of parity)
mdadm --create /dev/md0 --level=10 --raid-devices=4 --layout=f2 /dev/sd[bcde]1  # RAID 10, far layout for read speed
cat /proc/mdstat                                   # state and rebuild progress
mdadm --detail /dev/md0                            # members, state, failed and spare counts
mdadm --examine /dev/sdb1                          # superblock on a member
mdadm --detail --scan >> /etc/mdadm.conf           # persist the array definition; then rebuild the initramfs (dracut -f) if it holds root
mdadm --manage /dev/md0 --fail /dev/sdc1 --remove /dev/sdc1   # mark failed and pull it
mdadm --manage /dev/md0 --add /dev/sdd1            # add a replacement; rebuild starts immediately
mdadm --grow /dev/md0 --raid-devices=4 --add /dev/sde1   # reshape to more disks (slow; keep a backup)
mdadm --grow /dev/md0 --size=max                   # after replacing all members with larger disks
echo check > /sys/block/md0/md/sync_action; cat /sys/block/md0/md/mismatch_cnt   # scrub; raid-check.timer does this monthly on RHEL/Fedora
mdadm --stop /dev/md0; mdadm --zero-superblock /dev/sdb1   # dismantle; zero-superblock is DESTRUCTIVE to the member's RAID metadata

echo 200000 > /proc/sys/dev/raid/speed_limit_min speeds a rebuild at the cost of foreground IO. mdadm --monitor --scan --daemonise (or mdmonitor.service) mails MAILADDR from mdadm.conf on failures. RAID is not a backup: it survives a disk, not a deletion.

LUKS with cryptsetup#

LUKS2 stores a header with key slots on the device; each slot holds the master key wrapped by a passphrase or key file. Losing the header loses the data, so back it up. Argon2id is the default key derivation in cryptsetup 2.x and uses about 1 GiB of RAM per unlock by default; lower --pbkdf-memory on small VMs.

cryptsetup luksFormat --type luks2 /dev/sdb1       # DESTRUCTIVE: writes a LUKS header; asks for YES in capitals and a passphrase
cryptsetup luksFormat --type luks2 --pbkdf-memory 262144 --label data-crypt /dev/sdb1   # 256 MiB Argon2 memory
cryptsetup open /dev/sdb1 data-crypt               # unlock as /dev/mapper/data-crypt
mkfs.xfs -L data /dev/mapper/data-crypt && mount /dev/mapper/data-crypt /mnt/data
umount /mnt/data && cryptsetup close data-crypt
cryptsetup luksDump /dev/sdb1                      # header: slots in use, cipher, PBKDF parameters, UUID
cryptsetup luksHeaderBackup /dev/sdb1 --header-backup-file /root/sdb1-luks-header.img   # store off the machine; it grants access with any valid passphrase
cryptsetup luksAddKey /dev/sdb1                    # add a second passphrase (prompts for an existing one first)
cryptsetup luksAddKey /dev/sdb1 /root/data.key     # add a key file: dd if=/dev/urandom of=/root/data.key bs=64 count=1; chmod 600
cryptsetup luksRemoveKey /dev/sdb1                 # remove the passphrase you enter
cryptsetup luksKillSlot /dev/sdb1 1                # remove slot 1
cryptsetup luksChangeKey /dev/sdb1
cryptsetup -v status data-crypt                    # mapping details
cryptsetup reencrypt --disable-locks --resilience journal /dev/sdb1   # rotate the master key online (LUKS2)
cryptsetup open --type plain --key-file /dev/urandom /dev/sdb1 wipe && dd if=/dev/zero of=/dev/mapper/wipe bs=1M status=progress; cryptsetup close wipe   # DESTRUCTIVE: fast random-fill of a disk before formatting

Unlock at boot through /etc/crypttab, then reference the mapper device in fstab:

# <name>      <device>                                    <keyfile>        <options>
data-crypt    UUID=6b1f0c8e-2f8a-4c3d-9e1a-7b6c5d4e3f2a   /root/data.key   luks,discard,nofail
root-crypt    UUID=...                                    none             luks,discard

none prompts on the console. discard passes TRIM through to the SSD (it leaks which blocks are free, which is acceptable for most threat models). A key file for a data volume on an encrypted root is the usual pattern for servers: root asks for a passphrase or a TPM (systemd-cryptenroll --tpm2-device=auto /dev/sda3) and data unlocks itself from the key stored on root. dracut -f rebuilds the initramfs after changing crypttab entries needed for root.

Growing a VM disk end to end#

The host grows the virtual disk; the guest grows the partition, then the PV, then the LV, then the filesystem. No reboot is needed for a virtio disk when the guest can rescan.

On the host (one of):

qemu-img resize /var/lib/libvirt/images/my-vm.qcow2 +50G      # VM stopped; qcow2 or raw
virsh blockresize my-vm /var/lib/libvirt/images/my-vm.qcow2 150G   # VM running
qm resize 100 scsi0 +50G                                       # Proxmox
lvextend -L +50G /dev/vg0/vm-100-disk-0                        # LV-backed disk, then the VM sees it after a rescan or restart

The host-side commands are covered in libvirt and Proxmox.

In the guest, from the top:

lsblk                                              # confirm the disk (vda) is bigger and which partition holds the PV or filesystem
echo 1 > /sys/class/block/vda/device/rescan        # virtio-blk rescan; for virtio-scsi: echo 1 > /sys/class/scsi_device/*/device/rescan
growpart /dev/vda 3                                # cloud-utils-growpart: grow partition 3 to the end of the disk; safe online; fixes the GPT backup header too
# without growpart: parted /dev/vda resizepart 3 100%    (parted 3.2+ works online; older versions want the partition unmounted)
partprobe /dev/vda                                 # if the kernel did not pick up the new size
pvresize /dev/vda3                                 # PV sees the bigger partition
lvextend -r -l +100%FREE /dev/rhel/root            # LV and filesystem in one step (-r calls xfs_growfs or resize2fs)
df -h /                                            # done

If there is no LVM, stop after growpart and run xfs_growfs / or resize2fs /dev/vda3. If a swap partition sits between the root partition and the end of the disk, delete it (swapoff, parted rm), grow root, and recreate swap as a file instead. Adding a second virtual disk and running pvcreate, vgextend, lvextend -r avoids partition surgery entirely and is the simplest path when the layout is awkward.

Oneliners#

# Disks with model, serial, size and transport, no partitions
lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA

# Filesystems over 85% full
df -hP -x tmpfs -x devtmpfs | awk 'NR > 1 && $5 + 0 > 85 {print $5, $6}'

# Largest directories under a mount, staying on one filesystem
du -xh --max-depth=2 /var 2>/dev/null | sort -h | tail -20

# Largest files under a path
find /var -xdev -type f -size +500M -printf '%s\t%p\n' 2>/dev/null | sort -rn | numfmt --field=1 --to=iec | head

# Space held by deleted-but-open files, with the process holding them
lsof -nP +L1 | awk 'NR > 1 {print $2, $1, $7, $10}' | sort -k3,3nr | head

# Truncate a deleted log a process still holds (frees the space without restarting it)
: > /proc/1234/fd/5

# Which mount a path lives on and its options
findmnt -T /var/lib/containers -o TARGET,SOURCE,FSTYPE,OPTIONS

# fstab line for a device, ready to paste
printf 'UUID=%s  /mnt/data  %s  defaults,noatime,nofail  0 2\n' "$(blkid -s UUID -o value /dev/sdb1)" "$(blkid -s TYPE -o value /dev/sdb1)"

# Test fstab without rebooting
findmnt --verify && mount -a && systemctl daemon-reload

# Read-only mounts that should not be (a filesystem remounted ro after errors)
findmnt -rn -o TARGET,OPTIONS | awk '$2 ~ /(^|,)ro(,|$)/'

# Kernel messages about a disk
journalctl -k | grep -iE 'sd[a-z]|nvme|I/O error|ext4-fs error|XFS .*error|remount'

# Sector size and alignment of every partition
for d in /dev/sd? /dev/nvme?n1; do [ -e "$d" ] && parted -s "$d" unit s print 2>/dev/null; done

# Physical and logical sector size
cat /sys/block/sda/queue/physical_block_size /sys/block/sda/queue/logical_block_size

# LVM: free extents per VG and where each LV lives
vgs -o vg_name,vg_size,vg_free; lvs -a -o lv_name,vg_name,lv_size,devices,segtype

# LVM: full snapshot or thin pool warning
lvs -o lv_name,data_percent,snap_percent --noheadings | awk '$2 + 0 > 80 || $3 + 0 > 80'

# Progress of a RAID rebuild, refreshed
watch -n 5 cat /proc/mdstat

# SMART summary of every disk
for d in /dev/sd? /dev/nvme?; do [ -e "$d" ] && { printf '%s: ' "$d"; smartctl -H "$d" | grep -E 'result|overall'; }; done

# Pending and reallocated sectors across disks (non-zero means trouble)
for d in /dev/sd?; do printf '%s ' "$d"; smartctl -A "$d" | awk '/Reallocated_Sector|Current_Pending|Offline_Uncorr/ {printf "%s=%s ", $2, $10} END {print ""}'; done

# Disk temperature
smartctl -A /dev/sda | awk '/Temperature_Celsius|Airflow_Temperature/ {print $10}'; smartctl -a /dev/nvme0 | grep -i '^temperature'

# IO latency by device, one shot after 5 seconds
iostat -xzd 5 2 | awk '/^Device/ {h++} h == 2 && NF > 1 {print $1, "r_await", $6, "w_await", $12, "util", $NF}'

# Processes in D state (blocked on IO)
ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /^D/'

# Drop caches and time a cold read of a file (benchmark only)
sync; echo 3 > /proc/sys/vm/drop_caches; time cat /mnt/data/bigfile > /dev/null

# Bytes TRIM would reclaim (dry run has no flag; -v just prints what it trimmed)
fstrim -v /

# Copy a partition table to a new disk for a mirror and give it new GUIDs
sgdisk -R /dev/sdc /dev/sdb && sgdisk -G /dev/sdc

# Image a failing disk to a file, skipping bad blocks (ddrescue is in the ddrescue package)
ddrescue -d -r3 /dev/sdb /mnt/backup/sdb.img /mnt/backup/sdb.map

# Zero a disk before disposal (DESTRUCTIVE; blkdiscard is instant on SSDs that support it)
blkdiscard /dev/sdb || dd if=/dev/zero of=/dev/sdb bs=4M status=progress oflag=direct

# Secure-erase an NVMe (DESTRUCTIVE)
nvme format /dev/nvme0n1 --ses=1

# Fill-level of Btrfs, which df misreports
btrfs filesystem usage -h /

# Find which package or path is using inodes when df -i is full
find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head

Scripts#

Disk-space report that lists filesystems past a threshold and the biggest directories on each, suitable for a cron job that mails its output.

#!/usr/bin/env bash
# usage: disk-report.sh [threshold-percent]
set -euo pipefail
thr=${1:-80}

df -hP -x tmpfs -x devtmpfs -x overlay -x squashfs | awk -v thr="$thr" 'NR > 1 && $5 + 0 >= thr {print $6, $5}' | while read -r mnt pct; do
  printf '\n== %s at %s ==\n' "$mnt" "$pct"
  du -xh --max-depth=2 "$mnt" 2>/dev/null | sort -h | tail -8
  printf -- '-- deleted but open:\n'
  lsof -nP +L1 -- "$mnt" 2>/dev/null | awk 'NR > 1 {printf "  %s (pid %s) %s %s\n", $1, $2, $7, $10}' | sort -u | head -5
done

SMART health check across every disk with a non-zero exit when any attribute that predicts failure is set, for a monitoring hook or a systemd timer.

#!/usr/bin/env bash
# usage: smart-check.sh   (root; smartmontools installed)
set -euo pipefail
rc=0
while read -r dev _ type _; do                       # smartctl --scan prints: /dev/sda -d scsi # comment
  if [[ $dev == /dev/nvme* ]]; then
    out=$(smartctl -H -A "$dev" 2>/dev/null) || true
    bad=$(awk -F: '/Media and Data Integrity Errors/ {gsub(/[ ,]/, "", $2); print $2 + 0}' <<< "$out")
    used=$(awk -F: '/Percentage Used/ {gsub(/[ %]/, "", $2); print $2 + 0}' <<< "$out")
    health=$(grep -oE 'PASSED|FAILED' <<< "$out" | head -1)
    printf '%-14s %s media_errors=%s used=%s%%\n' "$dev" "${health:-unknown}" "${bad:-?}" "${used:-?}"
    [[ $health == PASSED && ${bad:-1} -eq 0 && ${used:-100} -lt 95 ]] || rc=1
  else
    out=$(smartctl -H -A -d "${type:-auto}" "$dev" 2>/dev/null) || true
    health=$(grep -oE 'PASSED|FAILED' <<< "$out" | head -1)
    counts=$(awk '/Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable/ {printf "%s=%s ", $2, $10; if ($10 + 0 > 0) bad = 1} END {exit bad}' <<< "$out") || rc=1
    printf '%-14s %s %s\n' "$dev" "${health:-unknown}" "$counts"
    [[ $health == PASSED ]] || rc=1
  fi
done < <(smartctl --scan)
exit "$rc"

Grow the root filesystem of a VM after the virtual disk was enlarged, detecting LVM or plain partition layouts. Modifies the partition table and filesystem of the running system; read it before running it.

#!/usr/bin/env bash
# usage: grow-root.sh   (root; needs cloud-utils-growpart)
set -euo pipefail
src=$(findmnt -no SOURCE /)                          # /dev/mapper/rhel-root or /dev/vda3
fstype=$(findmnt -no FSTYPE /)

if [[ $src == /dev/mapper/* ]]; then
  pv=$(pvs --noheadings -o pv_name -S "vg_name=$(lvs --noheadings -o vg_name "$src" | tr -d ' ')" | tr -d ' ' | head -1)
  part=$pv
else
  part=$src
fi
disk=$(lsblk -no PKNAME "$part"); num=$(lsblk -no PARTN "$part")   # PARTN needs util-linux 2.39+; else: num=${part##*[!0-9]}
printf 'root=%s fstype=%s partition=%s disk=/dev/%s number=%s\n' "$src" "$fstype" "$part" "$disk" "$num"

growpart "/dev/$disk" "$num" || { echo 'partition already fills the disk or growpart failed' >&2; }
if [[ $src == /dev/mapper/* ]]; then
  pvresize "$pv"
  lvextend -r -l +100%FREE "$src"
else
  case $fstype in
    xfs) xfs_growfs / ;;
    ext4) resize2fs "$part" ;;
    btrfs) btrfs filesystem resize max / ;;
    *) echo "unsupported filesystem $fstype" >&2; exit 1 ;;
  esac
fi
df -h /

Troubleshooting#

SymptomCauseFix
No space left on device but df shows free spaceInodes exhausted (df -i at 100%), or Btrfs metadata chunks fullDelete or move many small files; btrfs balance start -dusage=50 /
df shows the disk full after deleting big filesA process still holds the deleted file openlsof +L1, restart the process, or truncate via /proc/PID/fd/N
du total is far below df usedFiles under a mountpoint hidden by a later mount, or deleted-open filesdu -x, mount --bind / /mnt/root && du -sh /mnt/root/var, lsof +L1
Filesystem suddenly read-only, writes fail with EROFSKernel remounted it read-only after IO errors (errors=remount-ro)journalctl -k, then smartctl -a; unmount and fsck or xfs_repair when the hardware is sound
Boot drops to emergency shellfstab entry for a missing device without nofailjournalctl -xb, fix or nofail the line, systemctl daemon-reload, systemctl default
Slow IO, %util high, await in the hundreds of msDisk saturated, failing (check SMART), or a RAID rebuildiostat -xz 1, iotop -oPa, cat /proc/mdstat, smartctl -a
Processes stuck in D stateWaiting on IO to a hung device or an NFS serverps -eo pid,stat,wchan:32,comm | awk '$2 ~ /^D/', dmesg, umount -f -l for dead NFS
mount: wrong fs type, bad option, bad superblockWrong -t, missing kernel module, or damaged superblockblkid /dev/sdb1, dmesg | tail, e2fsck -b 32768 /dev/sdb1 for a backup superblock
target is busy on umountOpen files or a shell cd’d into itfuser -vm /mnt/data, lsof +f -- /mnt/data; umount -l as a last resort
Partition grew but lsblk shows the old sizeKernel has not re-read the tablepartprobe /dev/sda, partx -u /dev/sda, or echo 1 > /sys/class/block/sda/device/rescan
pvresize reports no changePartition not grown, or the kernel still sees the old sizelsblk, growpart, partprobe
lvextend says Insufficient free spaceVG has no free extentsvgs; add a PV with vgextend or grow the existing PV
xfs_growfs says data size unchanged, skippingThe device under the filesystem did not growGrow the LV or partition first and check lsblk
Btrfs ENOSPC with free spaceChunk allocation exhausted (unallocated 0)btrfs filesystem usage, btrfs balance start -dusage=20 -musage=20 /
LUKS device asks for passphrase at every bootcrypttab entry uses none or the key file is unreachableCheck /etc/crypttab, key file permissions, and dracut -f if root needs the key
cryptsetup open fails with No key available with this passphraseWrong passphrase, or a keyboard layout difference in the initramfscryptsetup luksDump for slots, try with --key-file, check KEYMAP in /etc/vconsole.conf
RAID degraded after reboot, mdadm: no arrays foundmdadm.conf missing the array or the initramfs is stalemdadm --assemble --scan, mdadm --detail --scan >> /etc/mdadm.conf, dracut -f
smartctl says Unknown USB bridge or no SMARTUSB or RAID controller in the waysmartctl -d sat, -d megaraid,N, -d sntasmedia; smartctl --scan
VM does not see the larger diskGuest needs a rescan, or the disk was resized while the VM was off but not re-readecho 1 > /sys/class/block/vda/device/rescan, or reboot the guest

Further reading#