# Disks and storage > Partition, format, mount, encrypt and grow Linux block storage with lsblk, parted, LVM, ext4, XFS, Btrfs, mdadm and cryptsetup, and diagnose full or slow disks. Canonical: https://www.wiki.jodisand.me/storage/ Reviewed: 2026-09-24 Related: [Files and directories](https://www.wiki.jodisand.me/files/index.md), [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md), [Proxmox VE](https://www.wiki.jodisand.me/proxmox/index.md), [libvirt and KVM](https://www.wiki.jodisand.me/libvirt/index.md), [Shell one-liners](https://www.wiki.jodisand.me/oneliners/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Block devices as a tree with filesystems | `lsblk -f` | | Sizes, models and mountpoints | `lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL` | | UUIDs and labels | `blkid` | | Free space per mounted filesystem | `df -hT -x tmpfs -x devtmpfs` | | Inode usage | `df -i` | | What is filling this directory | `du -xh --max-depth=1 / \| sort -h` | | Partition table | `parted /dev/sdb print`, `sgdisk -p /dev/sdb` | | Create a GPT and one full-size partition | `parted -s /dev/sdb mklabel gpt mkpart data ext4 1MiB 100%` | | Make a filesystem | `mkfs.ext4 -L data /dev/sdb1`, `mkfs.xfs -L data /dev/sdb1` | | Mount by label | `mount LABEL=data /mnt/data` | | Mount everything in fstab, verify it | `mount -a && findmnt --verify` | | Grow a partition to fill the disk | `growpart /dev/sda 3` | | Grow the filesystem | `resize2fs /dev/sda3` (ext4), `xfs_growfs /` (XFS, by mountpoint) | | LVM overview | `pvs; vgs; lvs -a -o +devices` | | Extend a logical volume and its filesystem | `lvextend -r -L +20G /dev/vg0/data` | | Disk health | `smartctl -a /dev/sda` | | Per-device IO utilisation | `iostat -xz 1` | | Which process is doing IO | `iotop -oPa` | | Trim an SSD or thin volume | `fstrim -av` | | Discard the page cache, for benchmarks only | `sync; echo 3 > /proc/sys/vm/drop_caches` | | Deleted but open files holding space | `lsof +L1` | | Filesystem check (unmounted) | `fsck.ext4 -f /dev/sdb1`, `xfs_repair /dev/sdb1` | Commands assume util-linux 2.40, LVM2 2.03, e2fsprogs 1.47, xfsprogs 6.x, btrfs-progs 6.x, cryptsetup 2.7 and smartmontools 7.4 as shipped on Fedora 42 and RHEL 9. Most of them need root. References: the [util-linux](https://www.kernel.org/pub/linux/utils/util-linux/) manual pages and the [Red Hat storage guide](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_storage_devices/index). ## The stack A disk is a block device (`/dev/sda`, `/dev/nvme0n1`, `/dev/vda`). A partition table divides it into partitions (`/dev/sda1`, `/dev/nvme0n1p1`). Optionally LVM takes partitions or whole disks as physical volumes, pools them in a volume group and carves logical volumes (`/dev/vg0/data`, also `/dev/mapper/vg0-data`). Optionally LUKS wraps any of those in encryption and exposes a `/dev/mapper/name` device. A filesystem sits on the final device and is mounted at a path. Each layer must be grown in order from the bottom up when a disk gets bigger, and shrunk from the top down. ```sh lsblk -f # tree: disk > partition > crypt > lvm > filesystem, with UUIDs lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,UUID,MOUNTPOINTS,MODEL,SERIAL,ROTA,DISC-GRAN lsblk -d -o NAME,SIZE,ROTA,TRAN,MODEL # disks only; ROTA 1 is spinning, TRAN is sata/nvme/virtio lsblk -J | jq '.blockdevices[] | select(.type == "disk") | .name' # JSON for scripts blkid # UUID, LABEL and TYPE for every filesystem, swap and LUKS header blkid -s UUID -o value /dev/sda2 # one value, for fstab findmnt # mounted filesystems as a tree with options findmnt -T /var/lib/containers # which mount holds this path findmnt --verify # parse fstab and report problems without mounting df -hT -x tmpfs -x devtmpfs -x overlay # usage per filesystem, without the noise df -i # inode usage; a full inode table also reports "No space left" cat /proc/partitions; ls -l /dev/disk/by-id/ /dev/disk/by-uuid/ /dev/disk/by-path/ udevadm info -q property /dev/sda | grep -E 'ID_SERIAL|ID_WWN|ID_BUS' ``` Device names such as `/dev/sdb` change between boots when disks are added, removed or enumerated in a different order. Refer to filesystems by `UUID=` or `LABEL=` in fstab and to disks by `/dev/disk/by-id/` in scripts and RAID or LVM configuration. ## Partitioning GPT is the default on anything current: up to 128 partitions, 64-bit sector addresses (disks over 2 TiB), a backup header at the end of the disk and a checksum. MBR (`msdos` in parted) remains only for legacy BIOS boot on old images and for compatibility with old firmware. `parted` scripts well and understands GPT and MBR. `sgdisk` is GPT-only, scripts even better and can clone tables. `fdisk` is interactive by default and fine for a one-off. `cfdisk` is a curses front end. All of them write a table only, never a filesystem; the kernel re-reads the table when they exit, and `partprobe /dev/sdb` forces it if a partition is in use. ```sh parted /dev/sdb print # table type, size, partitions with flags parted /dev/sdb unit s print # in sectors, to check alignment parted -s /dev/sdb mklabel gpt # DESTROYS the existing table and, in effect, every partition on it parted -s -a optimal /dev/sdb mkpart data ext4 1MiB 100% # one partition; "ext4" here only sets the type hint, mkfs still needed parted -s -a optimal /dev/sdb mkpart efi fat32 1MiB 1025MiB set 1 esp on # EFI system partition parted -s /dev/sdb mkpart lvm 1025MiB 100% set 2 lvm on parted -s /dev/sdb resizepart 2 100% # grow partition 2 to the end; parted 3.x asks nothing with -s parted -s /dev/sdb rm 3 # DESTRUCTIVE: removes partition 3 from the table parted -s /dev/sdb align-check optimal 1 # "1 aligned" ``` Start the first partition at 1 MiB. That aligns to every common physical sector and erase-block size, and it is what `-a optimal` does when you give sizes in MiB or percent. Starting at sector 63 or any odd number of 512-byte sectors on a 4 KiB-sector disk halves write performance. ```sh sgdisk -p /dev/sdb # print table sgdisk -o /dev/sdb # DESTRUCTIVE: new empty GPT sgdisk -n 1:0:+1G -t 1:ef00 -c 1:efi /dev/sdb # partition 1, 1 GiB from the first free sector, type EFI, name efi sgdisk -n 2:0:0 -t 2:8e00 -c 2:lvm /dev/sdb # partition 2, rest of disk, type Linux LVM sgdisk -L | grep -iE 'linux|efi|lvm|raid|swap' # type codes: 8300 Linux fs, 8e00 LVM, fd00 RAID, 8200 swap, ef00 ESP sgdisk -d 2 /dev/sdb # DESTRUCTIVE: delete partition 2 sgdisk -e /dev/sdb # move the backup GPT header to the actual end of a grown disk sgdisk -G /dev/sdb # randomise disk and partition GUIDs after cloning sgdisk -R /dev/sdc /dev/sdb # replicate sdb's table onto sdc (for RAID mirrors), then sgdisk -G /dev/sdc sgdisk -b table.bin /dev/sdb; sgdisk -l table.bin /dev/sdb # back up and restore a table sgdisk -Z /dev/sdb # DESTRUCTIVE: zap GPT and MBR structures ``` `fdisk -l` prints every disk's table. Interactive `fdisk /dev/sdb` uses `g` (new GPT), `n` (new), `t` (type), `d` (delete), `p` (print), `w` (write) and `q` (quit without writing). `wipefs -a /dev/sdb` erases filesystem, RAID, LVM and partition-table signatures so a reused disk does not get auto-assembled or auto-mounted; it is destructive and there is a `--no-act` flag to preview. ## Filesystems | | ext4 | XFS | Btrfs | | --- | --- | --- | --- | | Default on | Debian, Ubuntu | RHEL, Fedora Server | Fedora Workstation, openSUSE | | Grow online | Yes | Yes | Yes | | Shrink | Offline only | Never | Online | | Snapshots | Via LVM | Via LVM | Native, subvolume-level | | Checksums | Metadata | Metadata | Data and metadata | | Inodes | Fixed at mkfs | Dynamic | Dynamic | | Strengths | Mature, `fsck` recovers well, small files | Large files, parallel IO, huge filesystems | Snapshots, send/receive, compression, RAID 1 | ```sh mkfs.ext4 -L data /dev/sdb1 # DESTRUCTIVE: writes a filesystem over whatever is there mkfs.ext4 -L data -m 0.5 -T largefile4 /dev/sdb1 # -m reserved blocks % (default 5, meant for root); -T tunes inode ratio mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/sdb1 # do the init now rather than in the background after mount mkfs.xfs -L data /dev/sdb1 # DESTRUCTIVE; refuses to overwrite an existing filesystem without -f mkfs.xfs -L data -m reflink=1 -d su=64k,sw=4 /dev/md0 # stripe unit and width for a 4-data-disk RAID mkfs.btrfs -L data /dev/sdb1 # DESTRUCTIVE mkfs.btrfs -L data -m raid1 -d raid1 /dev/sdb /dev/sdc # two-disk mirror with no mdadm mkfs.vfat -F32 -n EFI /dev/sdb1 # EFI system partition ``` Inspect and tune: ```sh tune2fs -l /dev/sdb1 # ext4 superblock: features, mount count, last check, reserved blocks tune2fs -L newlabel /dev/sdb1 # relabel (ext4; xfs_admin -L for XFS, btrfs filesystem label for Btrfs) tune2fs -m 1 /dev/sdb1 # reduce reserved blocks to 1% tune2fs -O ^has_journal /dev/sdb1 # remove the journal (unmounted); rarely worth it dumpe2fs -h /dev/sdb1 | grep -iE 'block size|inode count|free' xfs_info /mnt/data # XFS geometry: block size, agcount, sunit/swidth, reflink xfs_admin -L data /dev/sdb1 # label (unmounted) btrfs filesystem show; btrfs filesystem usage /mnt/data; btrfs device stats /mnt/data ``` Resize. Grow the underlying device first (partition, LV or virtual disk), then the filesystem. ext4 and XFS grow while mounted; ext4 shrinks only unmounted, XFS never shrinks. ```sh resize2fs /dev/sdb1 # ext4: grow to fill the device, online resize2fs /dev/sdb1 50G # ext4: to a size; shrinking requires umount and e2fsck -f first xfs_growfs /mnt/data # XFS: takes the mountpoint, grows to fill the device xfs_growfs -D 26214400 /mnt/data # XFS: to a size in filesystem blocks btrfs filesystem resize max /mnt/data # Btrfs: grow to fill; also accepts -10G to shrink online btrfs filesystem resize 2:max /mnt/data # Btrfs: device ID 2 in a multi-device filesystem ``` Check and repair, unmounted, or on a snapshot of a mounted volume. `fsck` never runs on a mounted read-write filesystem. ```sh e2fsck -f /dev/sdb1 # force a full check; -p auto-fixes safe problems, -y answers yes to everything e2fsck -n /dev/sdb1 # read-only check of a mounted filesystem; results may be misleading xfs_repair -n /dev/sdb1 # dry run xfs_repair /dev/sdb1 # repair; if it complains about the log, mount and umount once first, -L zeroes it (loses recent metadata) btrfs check --readonly /dev/sdb1 # check; --repair is a last resort, ask on the mailing list first btrfs scrub start -B /mnt/data # verify every checksum on a mounted filesystem; -B waits ``` ### Btrfs subvolumes and snapshots A Btrfs subvolume is an independently mountable tree inside the filesystem. Fedora installs `/` and `/home` as subvolumes named `root` and `home` on one filesystem. A snapshot is a copy-on-write clone of a subvolume and costs nothing until data diverges. ```sh btrfs subvolume list / # subvolumes with IDs and paths btrfs subvolume create /mnt/data/projects btrfs subvolume snapshot -r /home /home/.snapshots/home-$(date +%F) # -r: read-only, needed for send btrfs subvolume delete /home/.snapshots/home-2026-08-01 btrfs send /home/.snapshots/home-2026-09-24 | ssh backup.example.com btrfs receive /backup/home # full copy btrfs send -p /home/.snapshots/home-2026-09-23 /home/.snapshots/home-2026-09-24 | ssh backup.example.com btrfs receive /backup/home # incremental btrfs property set /mnt/data/vm-images compression none # or mount -o compress=zstd:3 btrfs filesystem defragment -r -czstd /mnt/data/docs # compress existing files in place btrfs balance start -dusage=50 /mnt/data # reclaim half-empty data chunks; fixes ENOSPC with df showing free space ``` ## LVM LVM inserts a mapping layer between block devices and filesystems. Physical volumes (PV) join a volume group (VG), and logical volumes (LV) are allocated from the VG's extents (4 MiB each by default). LVs grow while in use, can span disks, and can be snapshotted. Metadata lives on every PV, so a VG assembles on any host that sees its disks. ```sh pvs; vgs; lvs # summaries; -v for more, -a includes hidden volumes pvs -o +pv_used; vgs -o +vg_free_count; lvs -a -o +devices,segtype # where each LV's extents live pvdisplay /dev/sdb1; vgdisplay vg0; lvdisplay /dev/vg0/data # verbose forms lvs -o lv_name,lv_size,data_percent,metadata_percent,snap_percent # fill levels of thin pools and snapshots pvcreate /dev/sdb1 # label a partition as a PV; refuses if a filesystem signature exists (wipefs first) vgcreate vg0 /dev/sdb1 /dev/sdc1 # VG from two PVs lvcreate -n data -L 100G vg0 # LV of 100 GiB lvcreate -n data -l 100%FREE vg0 # all remaining extents lvcreate -n data -l 50%VG vg0 # half the VG lvcreate -n fast -L 50G -i 2 -I 64 vg0 # striped across 2 PVs, 64 KiB stripes lvcreate -n mirror -L 50G -m 1 vg0 # RAID 1 via the raid1 segment type mkfs.xfs /dev/vg0/data && mount /dev/vg0/data /mnt/data ``` Grow: add a PV if the VG is full, then extend the LV with `-r` so the filesystem grows in the same step. ```sh pvcreate /dev/sdd1 && vgextend vg0 /dev/sdd1 # add a disk to the VG pvresize /dev/sdb1 # after the partition or virtual disk underneath grew lvextend -r -L +20G /dev/vg0/data # add 20 GiB and resize the filesystem (ext4, XFS, Btrfs) lvextend -r -l +100%FREE /dev/vg0/data # use everything left lvextend -r -L 200G /dev/vg0/data # to an absolute size lvreduce -r -L 50G /dev/vg0/data # DESTRUCTIVE if the filesystem is larger than 50G; ext4 only, unmounts and shrinks first with -r; XFS cannot shrink lvremove /dev/vg0/old # DESTRUCTIVE: deletes the LV and its data after a prompt vgreduce vg0 /dev/sdb1 # remove an empty PV from the VG (pvmove first if it holds extents) pvmove /dev/sdb1 # migrate extents off a PV online, then vgreduce and pvremove vgrename vg0 vg1; lvrename vg1 data data-old vgchange -an vg0; vgexport vg0 # deactivate and mark for moving to another host; vgimport there vgscan; vgchange -ay # find and activate VGs after adding disks ``` Snapshots are copy-on-write. A classic snapshot needs its own space in the VG to hold the changed blocks of the origin; when it fills, the snapshot becomes invalid (the origin is unaffected). Size it for the writes expected during its life, not the size of the origin. ```sh lvcreate -s -n data-snap -L 10G /dev/vg0/data # snapshot of data with 10 GiB for changed blocks lvs -o lv_name,origin,snap_percent # watch the fill level mount -o ro,nouuid /dev/vg0/data-snap /mnt/snap # XFS needs nouuid because the UUID matches the origin lvconvert --merge /dev/vg0/data-snap # roll the origin back to the snapshot; happens at next activation if the origin is mounted; removes the snapshot lvremove /dev/vg0/data-snap # discard the snapshot ``` Thin provisioning allocates blocks on write from a pool and makes snapshots cheap and unlimited in size, at the cost of a pool that can overfill and take every LV in it offline. Monitor `data_percent`; `lvm.conf` `thin_pool_autoextend_threshold` grows the pool automatically. ```sh lvcreate -L 500G -T vg0/pool # thin pool lvcreate -V 1T -T vg0/pool -n vm-disks # thin LV bigger than the pool lvcreate -s -n vm-disks-snap vg0/vm-disks # thin snapshot; no size needed ``` `/etc/lvm/backup/` and `/etc/lvm/archive/` hold metadata backups after every change; `vgcfgrestore -l vg0` lists them and `vgcfgrestore -f FILE vg0` restores one, which is how an accidental `lvremove` is undone before any data is overwritten. ## fstab and mount options `/etc/fstab` declares what mounts at boot. systemd generates a `.mount` unit per line, orders `local-fs.target` after them, and a failing line without `nofail` drops the boot into emergency mode. ```ini # UUID=3f1a2b4c-0d5e-4f6a-8b7c-9d0e1f2a3b4c / xfs defaults 0 0 UUID=A1B2-C3D4 /boot/efi vfat umask=0077,shortname=winnt 0 2 /dev/mapper/vg0-data /srv/data ext4 defaults,noatime,nofail,x-systemd.device-timeout=10s 0 2 LABEL=backup /mnt/backup xfs noauto,nofail,x-systemd.automount,x-systemd.idle-timeout=10min 0 0 nas.example.com:/export/media /mnt/media nfs4 _netdev,nofail,soft,timeo=150,retrans=3,noatime 0 0 tmpfs /var/tmp/build tmpfs size=4G,mode=1777,nosuid,nodev 0 0 /swapfile none swap defaults 0 0 ``` Options that matter: | Option | Effect | | --- | --- | | `defaults` | `rw,suid,dev,exec,auto,nouser,async` | | `noatime` | Do not update access times on read; safe for almost everything and removes a write per read. `relatime` (kernel default) updates once a day | | `nofail` | Boot continues if the device is missing; combine with `x-systemd.device-timeout=` so it does not wait 90 s | | `noauto` | Not mounted by `mount -a` or at boot; pair with `x-systemd.automount` to mount on first access | | `_netdev` | Network filesystem: wait for the network, unmount before it goes down | | `nosuid,nodev,noexec` | Harden `/tmp`, `/var/tmp`, `/home`, removable media and data mounts | | `ro` | Read-only | | `discard` | Issue TRIM on every delete (online discard); a `fstrim.timer` is usually better | | `x-systemd.requires-mounts-for=` | Order after another mount, for nested paths | | `pass` `0` | Never fsck at boot; `1` for root, `2` for the rest on ext4; XFS and Btrfs ignore it | ```sh mount -a # mount everything in fstab not yet mounted; the way to test a new line findmnt --verify # syntax and device existence check without mounting systemctl daemon-reload # regenerate mount units after editing fstab (systemd asks for this) mount -o remount,ro /srv/data # change options on a mounted filesystem mount --bind /srv/data/www /var/www # bind mount: same filesystem at a second path mount -o bind,ro /srv/data/www /var/www # read-only bind needs a remount step on old kernels; one step on 5.x+ mount -t tmpfs -o size=1G tmpfs /mnt/scratch mount -o loop image.iso /mnt/iso # loop device for an image; losetup -f --show image.img for a raw disk image losetup -Pf --show disk.img # -P scans partitions, giving /dev/loop0p1 umount /mnt/data; umount -l /mnt/data # -l: lazy, detach now and clean up when no longer busy systemd-mount /dev/sdb1 /mnt/usb; systemd-umount /mnt/usb # transient mount unit with automatic dependency handling ``` A mount unit name is the path with `/` replaced by `-`: `/srv/data` is `srv-data.mount`. `systemctl status srv-data.mount` and `journalctl -u srv-data.mount` show why a mount failed at boot. Writing a `.mount` unit instead of an fstab line is equivalent; fstab is simpler to review. ## Swap Swap gives the kernel somewhere to put anonymous pages under memory pressure. Fedora uses zram (compressed RAM) by default; servers commonly add a swap file or partition sized at a few GiB regardless of RAM so that a leak degrades gracefully instead of triggering the OOM killer. ```sh swapon --show; free -h # current swap devices and usage fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile # ext4 and XFS; Btrfs needs chattr +C and no snapshots on the file dd if=/dev/zero of=/swapfile bs=1M count=4096 status=progress # alternative when fallocate is refused mkswap -L swap /dev/vg0/swap && swapon /dev/vg0/swap # swap on an LV, growable with lvextend then swapoff/mkswap/swapon swapoff /swapfile # pages back into RAM; fails if RAM cannot hold them sysctl vm.swappiness; sysctl -w vm.swappiness=10 # 60 default; lower prefers dropping cache over swapping; persist in /etc/sysctl.d/ zramctl # zram devices and compression ratio ``` Hibernate needs swap at least the size of RAM and `resume=` on the kernel command line. Swap on a thin LV or a sparse file is unsafe: the kernel cannot allocate blocks under memory pressure. ## SMART Drives report their own health through SMART. `smartctl` reads it; `smartd` polls and emails or logs when attributes change. ```sh smartctl -i /dev/sda # identity: model, serial, firmware, sector sizes, SMART support smartctl -H /dev/sda # overall health: PASSED or FAILED (a pass means little; look at the attributes) smartctl -a /dev/sda # everything: attributes, error log, self-test log smartctl -a /dev/nvme0 # NVMe: percentage_used, available_spare, media_errors, unsafe_shutdowns smartctl -A /dev/sda | grep -E 'Reallocated_Sector|Current_Pending|Offline_Uncorrectable|UDMA_CRC|Power_On_Hours|Temperature' smartctl -t short /dev/sda; smartctl -t long /dev/sda # self-tests; check with -l selftest after the time it reports smartctl -l error /dev/sda # ATA error log smartctl -d sat /dev/sdX; smartctl -d megaraid,0 /dev/sda # behind a USB bridge or RAID controller smartctl --scan # devices and the -d type to use ``` The attributes that predict failure on spinning disks are `Reallocated_Sector_Ct` (5), `Current_Pending_Sector` (197) and `Offline_Uncorrectable` (198): any non-zero raw value means the disk has already lost data or is about to, and a rising count means replace it now. `UDMA_CRC_Error_Count` (199) is a cable or backplane problem, not the disk. On NVMe, `Percentage Used` over 100 and `Available Spare` under threshold mean end of life; `Media and Data Integrity Errors` should stay at zero. `smartd` runs from `smartmontools` with `/etc/smartmontools/smartd.conf`; `DEVICESCAN -a -o on -S on -s (S/../.././02|L/../../6/03) -m root -M exec /usr/libexec/smartmontools/smartdnotify` scans every disk, runs a short test nightly and a long test weekly, and notifies on change. ## IO performance ```sh iostat -xz 1 # per device each second, skip idle ones: r/s w/s rMB/s wMB/s r_await w_await aqu-sz %util iostat -xzd 5 3 nvme0n1 # one device, three samples five seconds apart iotop -oPa # processes doing IO (-o only active, -P processes not threads, -a accumulated) pidstat -d 1 # per-process read/write rate vmstat 1 # b (blocked on IO), bi/bo, wa (iowait %) cat /sys/block/sda/queue/scheduler # [mq-deadline] none bfq kyber; none for NVMe, mq-deadline for SATA SSD, bfq for desktop HDD cat /sys/block/sda/queue/rotational /sys/block/sda/queue/discard_granularity blockdev --getra /dev/sda; blockdev --setra 4096 /dev/sda # read-ahead in 512-byte sectors hdparm -tT /dev/sda # crude sequential read benchmark (reads only) fio --name=randread --filename=/mnt/data/fio.test --size=4G --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --direct=1 --runtime=30 --time_based --group_reporting # IOPS test; creates a 4 GiB file dd if=/dev/zero of=/mnt/data/dd.test bs=1M count=4096 oflag=direct status=progress # sequential write; creates a 4 GiB file ``` `%util` near 100 on a spinning disk is saturation; on an NVMe with many queues it is not, so look at `aqu-sz` and `await` instead. `r_await` and `w_await` are the latency the applications see: over 10 ms on an SSD or over 30 ms on a disk means queueing. Bandwidth without high `await` is a healthy busy disk. See [Linux performance](https://www.wiki.jodisand.me/linux-performance/#disk) for the wider method. TRIM tells an SSD or a thin LV which blocks are free. Weekly `fstrim.timer` (enabled by default on Fedora and RHEL) is preferable to the `discard` mount option, which issues small discards synchronously on every delete. ```sh fstrim -av # trim every mounted filesystem that supports it; prints bytes trimmed systemctl enable --now fstrim.timer; systemctl list-timers fstrim.timer lsblk -D # DISC-GRAN and DISC-MAX: 0B means the device does not accept discards ``` For a VM, the virtual disk must be attached with `discard=unmap` (virtio-scsi or virtio-blk on QEMU 4.0+) for `fstrim` in the guest to shrink a thin image or thin LV on the host. ## mdadm Software RAID from disks or partitions. Use partitions of type `fd00` (or `linux_raid_member` autodetection) so a replacement disk can be partitioned identically. ```sh mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 # DESTRUCTIVE to those partitions; mirror mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]1 # RAID 5 (one disk of parity) mdadm --create /dev/md0 --level=6 --raid-devices=4 --chunk=512 /dev/sd[bcde]1 # RAID 6 (two disks of parity) mdadm --create /dev/md0 --level=10 --raid-devices=4 --layout=f2 /dev/sd[bcde]1 # RAID 10, far layout for read speed cat /proc/mdstat # state and rebuild progress mdadm --detail /dev/md0 # members, state, failed and spare counts mdadm --examine /dev/sdb1 # superblock on a member mdadm --detail --scan >> /etc/mdadm.conf # persist the array definition; then rebuild the initramfs (dracut -f) if it holds root mdadm --manage /dev/md0 --fail /dev/sdc1 --remove /dev/sdc1 # mark failed and pull it mdadm --manage /dev/md0 --add /dev/sdd1 # add a replacement; rebuild starts immediately mdadm --grow /dev/md0 --raid-devices=4 --add /dev/sde1 # reshape to more disks (slow; keep a backup) mdadm --grow /dev/md0 --size=max # after replacing all members with larger disks echo check > /sys/block/md0/md/sync_action; cat /sys/block/md0/md/mismatch_cnt # scrub; raid-check.timer does this monthly on RHEL/Fedora mdadm --stop /dev/md0; mdadm --zero-superblock /dev/sdb1 # dismantle; zero-superblock is DESTRUCTIVE to the member's RAID metadata ``` `echo 200000 > /proc/sys/dev/raid/speed_limit_min` speeds a rebuild at the cost of foreground IO. `mdadm --monitor --scan --daemonise` (or `mdmonitor.service`) mails `MAILADDR` from `mdadm.conf` on failures. RAID is not a backup: it survives a disk, not a deletion. ## LUKS with cryptsetup LUKS2 stores a header with key slots on the device; each slot holds the master key wrapped by a passphrase or key file. Losing the header loses the data, so back it up. Argon2id is the default key derivation in cryptsetup 2.x and uses about 1 GiB of RAM per unlock by default; lower `--pbkdf-memory` on small VMs. ```sh cryptsetup luksFormat --type luks2 /dev/sdb1 # DESTRUCTIVE: writes a LUKS header; asks for YES in capitals and a passphrase cryptsetup luksFormat --type luks2 --pbkdf-memory 262144 --label data-crypt /dev/sdb1 # 256 MiB Argon2 memory cryptsetup open /dev/sdb1 data-crypt # unlock as /dev/mapper/data-crypt mkfs.xfs -L data /dev/mapper/data-crypt && mount /dev/mapper/data-crypt /mnt/data umount /mnt/data && cryptsetup close data-crypt cryptsetup luksDump /dev/sdb1 # header: slots in use, cipher, PBKDF parameters, UUID cryptsetup luksHeaderBackup /dev/sdb1 --header-backup-file /root/sdb1-luks-header.img # store off the machine; it grants access with any valid passphrase cryptsetup luksAddKey /dev/sdb1 # add a second passphrase (prompts for an existing one first) cryptsetup luksAddKey /dev/sdb1 /root/data.key # add a key file: dd if=/dev/urandom of=/root/data.key bs=64 count=1; chmod 600 cryptsetup luksRemoveKey /dev/sdb1 # remove the passphrase you enter cryptsetup luksKillSlot /dev/sdb1 1 # remove slot 1 cryptsetup luksChangeKey /dev/sdb1 cryptsetup -v status data-crypt # mapping details cryptsetup reencrypt --disable-locks --resilience journal /dev/sdb1 # rotate the master key online (LUKS2) cryptsetup open --type plain --key-file /dev/urandom /dev/sdb1 wipe && dd if=/dev/zero of=/dev/mapper/wipe bs=1M status=progress; cryptsetup close wipe # DESTRUCTIVE: fast random-fill of a disk before formatting ``` Unlock at boot through `/etc/crypttab`, then reference the mapper device in fstab: ```ini # data-crypt UUID=6b1f0c8e-2f8a-4c3d-9e1a-7b6c5d4e3f2a /root/data.key luks,discard,nofail root-crypt UUID=... none luks,discard ``` `none` prompts on the console. `discard` passes TRIM through to the SSD (it leaks which blocks are free, which is acceptable for most threat models). A key file for a data volume on an encrypted root is the usual pattern for servers: root asks for a passphrase or a TPM (`systemd-cryptenroll --tpm2-device=auto /dev/sda3`) and data unlocks itself from the key stored on root. `dracut -f` rebuilds the initramfs after changing crypttab entries needed for root. ## Growing a VM disk end to end The host grows the virtual disk; the guest grows the partition, then the PV, then the LV, then the filesystem. No reboot is needed for a virtio disk when the guest can rescan. On the host (one of): ```sh qemu-img resize /var/lib/libvirt/images/my-vm.qcow2 +50G # VM stopped; qcow2 or raw virsh blockresize my-vm /var/lib/libvirt/images/my-vm.qcow2 150G # VM running qm resize 100 scsi0 +50G # Proxmox lvextend -L +50G /dev/vg0/vm-100-disk-0 # LV-backed disk, then the VM sees it after a rescan or restart ``` The host-side commands are covered in [libvirt](https://www.wiki.jodisand.me/libvirt/) and [Proxmox](https://www.wiki.jodisand.me/proxmox/). In the guest, from the top: ```sh lsblk # confirm the disk (vda) is bigger and which partition holds the PV or filesystem echo 1 > /sys/class/block/vda/device/rescan # virtio-blk rescan; for virtio-scsi: echo 1 > /sys/class/scsi_device/*/device/rescan growpart /dev/vda 3 # cloud-utils-growpart: grow partition 3 to the end of the disk; safe online; fixes the GPT backup header too # without growpart: parted /dev/vda resizepart 3 100% (parted 3.2+ works online; older versions want the partition unmounted) partprobe /dev/vda # if the kernel did not pick up the new size pvresize /dev/vda3 # PV sees the bigger partition lvextend -r -l +100%FREE /dev/rhel/root # LV and filesystem in one step (-r calls xfs_growfs or resize2fs) df -h / # done ``` If there is no LVM, stop after `growpart` and run `xfs_growfs /` or `resize2fs /dev/vda3`. If a swap partition sits between the root partition and the end of the disk, delete it (`swapoff`, `parted rm`), grow root, and recreate swap as a file instead. Adding a second virtual disk and running `pvcreate`, `vgextend`, `lvextend -r` avoids partition surgery entirely and is the simplest path when the layout is awkward. ## Oneliners ```sh # Disks with model, serial, size and transport, no partitions lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA # Filesystems over 85% full df -hP -x tmpfs -x devtmpfs | awk 'NR > 1 && $5 + 0 > 85 {print $5, $6}' # Largest directories under a mount, staying on one filesystem du -xh --max-depth=2 /var 2>/dev/null | sort -h | tail -20 # Largest files under a path find /var -xdev -type f -size +500M -printf '%s\t%p\n' 2>/dev/null | sort -rn | numfmt --field=1 --to=iec | head # Space held by deleted-but-open files, with the process holding them lsof -nP +L1 | awk 'NR > 1 {print $2, $1, $7, $10}' | sort -k3,3nr | head # Truncate a deleted log a process still holds (frees the space without restarting it) : > /proc/1234/fd/5 # Which mount a path lives on and its options findmnt -T /var/lib/containers -o TARGET,SOURCE,FSTYPE,OPTIONS # fstab line for a device, ready to paste printf 'UUID=%s /mnt/data %s defaults,noatime,nofail 0 2\n' "$(blkid -s UUID -o value /dev/sdb1)" "$(blkid -s TYPE -o value /dev/sdb1)" # Test fstab without rebooting findmnt --verify && mount -a && systemctl daemon-reload # Read-only mounts that should not be (a filesystem remounted ro after errors) findmnt -rn -o TARGET,OPTIONS | awk '$2 ~ /(^|,)ro(,|$)/' # Kernel messages about a disk journalctl -k | grep -iE 'sd[a-z]|nvme|I/O error|ext4-fs error|XFS .*error|remount' # Sector size and alignment of every partition for d in /dev/sd? /dev/nvme?n1; do [ -e "$d" ] && parted -s "$d" unit s print 2>/dev/null; done # Physical and logical sector size cat /sys/block/sda/queue/physical_block_size /sys/block/sda/queue/logical_block_size # LVM: free extents per VG and where each LV lives vgs -o vg_name,vg_size,vg_free; lvs -a -o lv_name,vg_name,lv_size,devices,segtype # LVM: full snapshot or thin pool warning lvs -o lv_name,data_percent,snap_percent --noheadings | awk '$2 + 0 > 80 || $3 + 0 > 80' # Progress of a RAID rebuild, refreshed watch -n 5 cat /proc/mdstat # SMART summary of every disk for d in /dev/sd? /dev/nvme?; do [ -e "$d" ] && { printf '%s: ' "$d"; smartctl -H "$d" | grep -E 'result|overall'; }; done # Pending and reallocated sectors across disks (non-zero means trouble) for d in /dev/sd?; do printf '%s ' "$d"; smartctl -A "$d" | awk '/Reallocated_Sector|Current_Pending|Offline_Uncorr/ {printf "%s=%s ", $2, $10} END {print ""}'; done # Disk temperature smartctl -A /dev/sda | awk '/Temperature_Celsius|Airflow_Temperature/ {print $10}'; smartctl -a /dev/nvme0 | grep -i '^temperature' # IO latency by device, one shot after 5 seconds iostat -xzd 5 2 | awk '/^Device/ {h++} h == 2 && NF > 1 {print $1, "r_await", $6, "w_await", $12, "util", $NF}' # Processes in D state (blocked on IO) ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /^D/' # Drop caches and time a cold read of a file (benchmark only) sync; echo 3 > /proc/sys/vm/drop_caches; time cat /mnt/data/bigfile > /dev/null # Bytes TRIM would reclaim (dry run has no flag; -v just prints what it trimmed) fstrim -v / # Copy a partition table to a new disk for a mirror and give it new GUIDs sgdisk -R /dev/sdc /dev/sdb && sgdisk -G /dev/sdc # Image a failing disk to a file, skipping bad blocks (ddrescue is in the ddrescue package) ddrescue -d -r3 /dev/sdb /mnt/backup/sdb.img /mnt/backup/sdb.map # Zero a disk before disposal (DESTRUCTIVE; blkdiscard is instant on SSDs that support it) blkdiscard /dev/sdb || dd if=/dev/zero of=/dev/sdb bs=4M status=progress oflag=direct # Secure-erase an NVMe (DESTRUCTIVE) nvme format /dev/nvme0n1 --ses=1 # Fill-level of Btrfs, which df misreports btrfs filesystem usage -h / # Find which package or path is using inodes when df -i is full find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head ``` ## Scripts Disk-space report that lists filesystems past a threshold and the biggest directories on each, suitable for a cron job that mails its output. ```sh #!/usr/bin/env bash # usage: disk-report.sh [threshold-percent] set -euo pipefail thr=${1:-80} df -hP -x tmpfs -x devtmpfs -x overlay -x squashfs | awk -v thr="$thr" 'NR > 1 && $5 + 0 >= thr {print $6, $5}' | while read -r mnt pct; do printf '\n== %s at %s ==\n' "$mnt" "$pct" du -xh --max-depth=2 "$mnt" 2>/dev/null | sort -h | tail -8 printf -- '-- deleted but open:\n' lsof -nP +L1 -- "$mnt" 2>/dev/null | awk 'NR > 1 {printf " %s (pid %s) %s %s\n", $1, $2, $7, $10}' | sort -u | head -5 done ``` SMART health check across every disk with a non-zero exit when any attribute that predicts failure is set, for a monitoring hook or a systemd timer. ```sh #!/usr/bin/env bash # usage: smart-check.sh (root; smartmontools installed) set -euo pipefail rc=0 while read -r dev _ type _; do # smartctl --scan prints: /dev/sda -d scsi # comment if [[ $dev == /dev/nvme* ]]; then out=$(smartctl -H -A "$dev" 2>/dev/null) || true bad=$(awk -F: '/Media and Data Integrity Errors/ {gsub(/[ ,]/, "", $2); print $2 + 0}' <<< "$out") used=$(awk -F: '/Percentage Used/ {gsub(/[ %]/, "", $2); print $2 + 0}' <<< "$out") health=$(grep -oE 'PASSED|FAILED' <<< "$out" | head -1) printf '%-14s %s media_errors=%s used=%s%%\n' "$dev" "${health:-unknown}" "${bad:-?}" "${used:-?}" [[ $health == PASSED && ${bad:-1} -eq 0 && ${used:-100} -lt 95 ]] || rc=1 else out=$(smartctl -H -A -d "${type:-auto}" "$dev" 2>/dev/null) || true health=$(grep -oE 'PASSED|FAILED' <<< "$out" | head -1) counts=$(awk '/Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable/ {printf "%s=%s ", $2, $10; if ($10 + 0 > 0) bad = 1} END {exit bad}' <<< "$out") || rc=1 printf '%-14s %s %s\n' "$dev" "${health:-unknown}" "$counts" [[ $health == PASSED ]] || rc=1 fi done < <(smartctl --scan) exit "$rc" ``` Grow the root filesystem of a VM after the virtual disk was enlarged, detecting LVM or plain partition layouts. Modifies the partition table and filesystem of the running system; read it before running it. ```sh #!/usr/bin/env bash # usage: grow-root.sh (root; needs cloud-utils-growpart) set -euo pipefail src=$(findmnt -no SOURCE /) # /dev/mapper/rhel-root or /dev/vda3 fstype=$(findmnt -no FSTYPE /) if [[ $src == /dev/mapper/* ]]; then pv=$(pvs --noheadings -o pv_name -S "vg_name=$(lvs --noheadings -o vg_name "$src" | tr -d ' ')" | tr -d ' ' | head -1) part=$pv else part=$src fi disk=$(lsblk -no PKNAME "$part"); num=$(lsblk -no PARTN "$part") # PARTN needs util-linux 2.39+; else: num=${part##*[!0-9]} printf 'root=%s fstype=%s partition=%s disk=/dev/%s number=%s\n' "$src" "$fstype" "$part" "$disk" "$num" growpart "/dev/$disk" "$num" || { echo 'partition already fills the disk or growpart failed' >&2; } if [[ $src == /dev/mapper/* ]]; then pvresize "$pv" lvextend -r -l +100%FREE "$src" else case $fstype in xfs) xfs_growfs / ;; ext4) resize2fs "$part" ;; btrfs) btrfs filesystem resize max / ;; *) echo "unsupported filesystem $fstype" >&2; exit 1 ;; esac fi df -h / ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `No space left on device` but `df` shows free space | Inodes exhausted (`df -i` at 100%), or Btrfs metadata chunks full | Delete or move many small files; `btrfs balance start -dusage=50 /` | | `df` shows the disk full after deleting big files | A process still holds the deleted file open | `lsof +L1`, restart the process, or truncate via `/proc/PID/fd/N` | | `du` total is far below `df` used | Files under a mountpoint hidden by a later mount, or deleted-open files | `du -x`, `mount --bind / /mnt/root && du -sh /mnt/root/var`, `lsof +L1` | | Filesystem suddenly read-only, writes fail with `EROFS` | Kernel remounted it read-only after IO errors (`errors=remount-ro`) | `journalctl -k`, then `smartctl -a`; unmount and `fsck` or `xfs_repair` when the hardware is sound | | Boot drops to emergency shell | fstab entry for a missing device without `nofail` | `journalctl -xb`, fix or `nofail` the line, `systemctl daemon-reload`, `systemctl default` | | Slow IO, `%util` high, `await` in the hundreds of ms | Disk saturated, failing (check SMART), or a RAID rebuild | `iostat -xz 1`, `iotop -oPa`, `cat /proc/mdstat`, `smartctl -a` | | Processes stuck in `D` state | Waiting on IO to a hung device or an NFS server | `ps -eo pid,stat,wchan:32,comm \| awk '$2 ~ /^D/'`, `dmesg`, `umount -f -l` for dead NFS | | `mount: wrong fs type, bad option, bad superblock` | Wrong `-t`, missing kernel module, or damaged superblock | `blkid /dev/sdb1`, `dmesg \| tail`, `e2fsck -b 32768 /dev/sdb1` for a backup superblock | | `target is busy` on umount | Open files or a shell `cd`'d into it | `fuser -vm /mnt/data`, `lsof +f -- /mnt/data`; `umount -l` as a last resort | | Partition grew but `lsblk` shows the old size | Kernel has not re-read the table | `partprobe /dev/sda`, `partx -u /dev/sda`, or `echo 1 > /sys/class/block/sda/device/rescan` | | `pvresize` reports no change | Partition not grown, or the kernel still sees the old size | `lsblk`, `growpart`, `partprobe` | | `lvextend` says `Insufficient free space` | VG has no free extents | `vgs`; add a PV with `vgextend` or grow the existing PV | | `xfs_growfs` says `data size unchanged, skipping` | The device under the filesystem did not grow | Grow the LV or partition first and check `lsblk` | | Btrfs `ENOSPC` with free space | Chunk allocation exhausted (unallocated 0) | `btrfs filesystem usage`, `btrfs balance start -dusage=20 -musage=20 /` | | LUKS device asks for passphrase at every boot | crypttab entry uses `none` or the key file is unreachable | Check `/etc/crypttab`, key file permissions, and `dracut -f` if root needs the key | | `cryptsetup open` fails with `No key available with this passphrase` | Wrong passphrase, or a keyboard layout difference in the initramfs | `cryptsetup luksDump` for slots, try with `--key-file`, check `KEYMAP` in `/etc/vconsole.conf` | | RAID degraded after reboot, `mdadm: no arrays found` | `mdadm.conf` missing the array or the initramfs is stale | `mdadm --assemble --scan`, `mdadm --detail --scan >> /etc/mdadm.conf`, `dracut -f` | | `smartctl` says `Unknown USB bridge` or no SMART | USB or RAID controller in the way | `smartctl -d sat`, `-d megaraid,N`, `-d sntasmedia`; `smartctl --scan` | | VM does not see the larger disk | Guest needs a rescan, or the disk was resized while the VM was off but not re-read | `echo 1 > /sys/class/block/vda/device/rescan`, or reboot the guest | ## Further reading - [Red Hat: Managing storage devices](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_storage_devices/index): partitions, LVM, RAID, LUKS and NVMe on RHEL 9. - [Red Hat: Configuring and managing logical volumes](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_logical_volumes/index): LVM including thin provisioning, snapshots and RAID LVs. - [XFS documentation](https://xfs.wiki.kernel.org/) and [xfs(5)](https://man7.org/linux/man-pages/man5/xfs.5.html): mount options, geometry and repair. - [ext4 kernel documentation](https://www.kernel.org/doc/html/latest/admin-guide/ext4.html): features and mount options. - [Btrfs documentation](https://btrfs.readthedocs.io/en/latest/): subvolumes, balance, scrub, RAID profiles and the FAQ on free space. - [cryptsetup and LUKS2](https://gitlab.com/cryptsetup/cryptsetup/-/wikis/home): the LUKS2 on-disk format, FAQ and recovery guidance. - [Linux RAID wiki](https://raid.wiki.kernel.org/index.php/Linux_Raid): mdadm, recovery and the RAID setup guides. - [smartmontools](https://www.smartmontools.org/wiki/TocDoc): `smartctl`, `smartd.conf` and the attribute meanings per vendor.