Disks and storage
Partition, format, mount, encrypt and grow Linux block storage with lsblk, parted, LVM, ext4, XFS, Btrfs, mdadm and cryptsetup, and diagnose full or slow disks.
On this page
Cheatsheet#
| Task | Command |
|---|---|
| Block devices as a tree with filesystems | lsblk -f |
| Sizes, models and mountpoints | lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL |
| UUIDs and labels | blkid |
| Free space per mounted filesystem | df -hT -x tmpfs -x devtmpfs |
| Inode usage | df -i |
| What is filling this directory | du -xh --max-depth=1 / | sort -h |
| Partition table | parted /dev/sdb print, sgdisk -p /dev/sdb |
| Create a GPT and one full-size partition | parted -s /dev/sdb mklabel gpt mkpart data ext4 1MiB 100% |
| Make a filesystem | mkfs.ext4 -L data /dev/sdb1, mkfs.xfs -L data /dev/sdb1 |
| Mount by label | mount LABEL=data /mnt/data |
| Mount everything in fstab, verify it | mount -a && findmnt --verify |
| Grow a partition to fill the disk | growpart /dev/sda 3 |
| Grow the filesystem | resize2fs /dev/sda3 (ext4), xfs_growfs / (XFS, by mountpoint) |
| LVM overview | pvs; vgs; lvs -a -o +devices |
| Extend a logical volume and its filesystem | lvextend -r -L +20G /dev/vg0/data |
| Disk health | smartctl -a /dev/sda |
| Per-device IO utilisation | iostat -xz 1 |
| Which process is doing IO | iotop -oPa |
| Trim an SSD or thin volume | fstrim -av |
| Discard the page cache, for benchmarks only | sync; echo 3 > /proc/sys/vm/drop_caches |
| Deleted but open files holding space | lsof +L1 |
| Filesystem check (unmounted) | fsck.ext4 -f /dev/sdb1, xfs_repair /dev/sdb1 |
Commands assume util-linux 2.40, LVM2 2.03, e2fsprogs 1.47, xfsprogs 6.x, btrfs-progs 6.x, cryptsetup 2.7 and smartmontools 7.4 as shipped on Fedora 42 and RHEL 9. Most of them need root. References: the util-linux manual pages and the Red Hat storage guide.
The stack#
A disk is a block device (/dev/sda, /dev/nvme0n1, /dev/vda). A partition table divides it into partitions (/dev/sda1, /dev/nvme0n1p1). Optionally LVM takes partitions or whole disks as physical volumes, pools them in a volume group and carves logical volumes (/dev/vg0/data, also /dev/mapper/vg0-data). Optionally LUKS wraps any of those in encryption and exposes a /dev/mapper/name device. A filesystem sits on the final device and is mounted at a path. Each layer must be grown in order from the bottom up when a disk gets bigger, and shrunk from the top down.
lsblk -f # tree: disk > partition > crypt > lvm > filesystem, with UUIDs
lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,UUID,MOUNTPOINTS,MODEL,SERIAL,ROTA,DISC-GRAN
lsblk -d -o NAME,SIZE,ROTA,TRAN,MODEL # disks only; ROTA 1 is spinning, TRAN is sata/nvme/virtio
lsblk -J | jq '.blockdevices[] | select(.type == "disk") | .name' # JSON for scripts
blkid # UUID, LABEL and TYPE for every filesystem, swap and LUKS header
blkid -s UUID -o value /dev/sda2 # one value, for fstab
findmnt # mounted filesystems as a tree with options
findmnt -T /var/lib/containers # which mount holds this path
findmnt --verify # parse fstab and report problems without mounting
df -hT -x tmpfs -x devtmpfs -x overlay # usage per filesystem, without the noise
df -i # inode usage; a full inode table also reports "No space left"
cat /proc/partitions; ls -l /dev/disk/by-id/ /dev/disk/by-uuid/ /dev/disk/by-path/
udevadm info -q property /dev/sda | grep -E 'ID_SERIAL|ID_WWN|ID_BUS'Device names such as /dev/sdb change between boots when disks are added, removed or enumerated in a different order. Refer to filesystems by UUID= or LABEL= in fstab and to disks by /dev/disk/by-id/ in scripts and RAID or LVM configuration.
Partitioning#
GPT is the default on anything current: up to 128 partitions, 64-bit sector addresses (disks over 2 TiB), a backup header at the end of the disk and a checksum. MBR (msdos in parted) remains only for legacy BIOS boot on old images and for compatibility with old firmware.
parted scripts well and understands GPT and MBR. sgdisk is GPT-only, scripts even better and can clone tables. fdisk is interactive by default and fine for a one-off. cfdisk is a curses front end. All of them write a table only, never a filesystem; the kernel re-reads the table when they exit, and partprobe /dev/sdb forces it if a partition is in use.
parted /dev/sdb print # table type, size, partitions with flags
parted /dev/sdb unit s print # in sectors, to check alignment
parted -s /dev/sdb mklabel gpt # DESTROYS the existing table and, in effect, every partition on it
parted -s -a optimal /dev/sdb mkpart data ext4 1MiB 100% # one partition; "ext4" here only sets the type hint, mkfs still needed
parted -s -a optimal /dev/sdb mkpart efi fat32 1MiB 1025MiB set 1 esp on # EFI system partition
parted -s /dev/sdb mkpart lvm 1025MiB 100% set 2 lvm on
parted -s /dev/sdb resizepart 2 100% # grow partition 2 to the end; parted 3.x asks nothing with -s
parted -s /dev/sdb rm 3 # DESTRUCTIVE: removes partition 3 from the table
parted -s /dev/sdb align-check optimal 1 # "1 aligned"Start the first partition at 1 MiB. That aligns to every common physical sector and erase-block size, and it is what -a optimal does when you give sizes in MiB or percent. Starting at sector 63 or any odd number of 512-byte sectors on a 4 KiB-sector disk halves write performance.
sgdisk -p /dev/sdb # print table
sgdisk -o /dev/sdb # DESTRUCTIVE: new empty GPT
sgdisk -n 1:0:+1G -t 1:ef00 -c 1:efi /dev/sdb # partition 1, 1 GiB from the first free sector, type EFI, name efi
sgdisk -n 2:0:0 -t 2:8e00 -c 2:lvm /dev/sdb # partition 2, rest of disk, type Linux LVM
sgdisk -L | grep -iE 'linux|efi|lvm|raid|swap' # type codes: 8300 Linux fs, 8e00 LVM, fd00 RAID, 8200 swap, ef00 ESP
sgdisk -d 2 /dev/sdb # DESTRUCTIVE: delete partition 2
sgdisk -e /dev/sdb # move the backup GPT header to the actual end of a grown disk
sgdisk -G /dev/sdb # randomise disk and partition GUIDs after cloning
sgdisk -R /dev/sdc /dev/sdb # replicate sdb's table onto sdc (for RAID mirrors), then sgdisk -G /dev/sdc
sgdisk -b table.bin /dev/sdb; sgdisk -l table.bin /dev/sdb # back up and restore a table
sgdisk -Z /dev/sdb # DESTRUCTIVE: zap GPT and MBR structuresfdisk -l prints every disk’s table. Interactive fdisk /dev/sdb uses g (new GPT), n (new), t (type), d (delete), p (print), w (write) and q (quit without writing). wipefs -a /dev/sdb erases filesystem, RAID, LVM and partition-table signatures so a reused disk does not get auto-assembled or auto-mounted; it is destructive and there is a --no-act flag to preview.
Filesystems#
| ext4 | XFS | Btrfs | |
|---|---|---|---|
| Default on | Debian, Ubuntu | RHEL, Fedora Server | Fedora Workstation, openSUSE |
| Grow online | Yes | Yes | Yes |
| Shrink | Offline only | Never | Online |
| Snapshots | Via LVM | Via LVM | Native, subvolume-level |
| Checksums | Metadata | Metadata | Data and metadata |
| Inodes | Fixed at mkfs | Dynamic | Dynamic |
| Strengths | Mature, fsck recovers well, small files | Large files, parallel IO, huge filesystems | Snapshots, send/receive, compression, RAID 1 |
mkfs.ext4 -L data /dev/sdb1 # DESTRUCTIVE: writes a filesystem over whatever is there
mkfs.ext4 -L data -m 0.5 -T largefile4 /dev/sdb1 # -m reserved blocks % (default 5, meant for root); -T tunes inode ratio
mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/sdb1 # do the init now rather than in the background after mount
mkfs.xfs -L data /dev/sdb1 # DESTRUCTIVE; refuses to overwrite an existing filesystem without -f
mkfs.xfs -L data -m reflink=1 -d su=64k,sw=4 /dev/md0 # stripe unit and width for a 4-data-disk RAID
mkfs.btrfs -L data /dev/sdb1 # DESTRUCTIVE
mkfs.btrfs -L data -m raid1 -d raid1 /dev/sdb /dev/sdc # two-disk mirror with no mdadm
mkfs.vfat -F32 -n EFI /dev/sdb1 # EFI system partitionInspect and tune:
tune2fs -l /dev/sdb1 # ext4 superblock: features, mount count, last check, reserved blocks
tune2fs -L newlabel /dev/sdb1 # relabel (ext4; xfs_admin -L for XFS, btrfs filesystem label for Btrfs)
tune2fs -m 1 /dev/sdb1 # reduce reserved blocks to 1%
tune2fs -O ^has_journal /dev/sdb1 # remove the journal (unmounted); rarely worth it
dumpe2fs -h /dev/sdb1 | grep -iE 'block size|inode count|free'
xfs_info /mnt/data # XFS geometry: block size, agcount, sunit/swidth, reflink
xfs_admin -L data /dev/sdb1 # label (unmounted)
btrfs filesystem show; btrfs filesystem usage /mnt/data; btrfs device stats /mnt/dataResize. Grow the underlying device first (partition, LV or virtual disk), then the filesystem. ext4 and XFS grow while mounted; ext4 shrinks only unmounted, XFS never shrinks.
resize2fs /dev/sdb1 # ext4: grow to fill the device, online
resize2fs /dev/sdb1 50G # ext4: to a size; shrinking requires umount and e2fsck -f first
xfs_growfs /mnt/data # XFS: takes the mountpoint, grows to fill the device
xfs_growfs -D 26214400 /mnt/data # XFS: to a size in filesystem blocks
btrfs filesystem resize max /mnt/data # Btrfs: grow to fill; also accepts -10G to shrink online
btrfs filesystem resize 2:max /mnt/data # Btrfs: device ID 2 in a multi-device filesystemCheck and repair, unmounted, or on a snapshot of a mounted volume. fsck never runs on a mounted read-write filesystem.
e2fsck -f /dev/sdb1 # force a full check; -p auto-fixes safe problems, -y answers yes to everything
e2fsck -n /dev/sdb1 # read-only check of a mounted filesystem; results may be misleading
xfs_repair -n /dev/sdb1 # dry run
xfs_repair /dev/sdb1 # repair; if it complains about the log, mount and umount once first, -L zeroes it (loses recent metadata)
btrfs check --readonly /dev/sdb1 # check; --repair is a last resort, ask on the mailing list first
btrfs scrub start -B /mnt/data # verify every checksum on a mounted filesystem; -B waitsBtrfs subvolumes and snapshots#
A Btrfs subvolume is an independently mountable tree inside the filesystem. Fedora installs / and /home as subvolumes named root and home on one filesystem. A snapshot is a copy-on-write clone of a subvolume and costs nothing until data diverges.
btrfs subvolume list / # subvolumes with IDs and paths
btrfs subvolume create /mnt/data/projects
btrfs subvolume snapshot -r /home /home/.snapshots/home-$(date +%F) # -r: read-only, needed for send
btrfs subvolume delete /home/.snapshots/home-2026-08-01
btrfs send /home/.snapshots/home-2026-09-24 | ssh backup.example.com btrfs receive /backup/home # full copy
btrfs send -p /home/.snapshots/home-2026-09-23 /home/.snapshots/home-2026-09-24 | ssh backup.example.com btrfs receive /backup/home # incremental
btrfs property set /mnt/data/vm-images compression none # or mount -o compress=zstd:3
btrfs filesystem defragment -r -czstd /mnt/data/docs # compress existing files in place
btrfs balance start -dusage=50 /mnt/data # reclaim half-empty data chunks; fixes ENOSPC with df showing free spaceLVM#
LVM inserts a mapping layer between block devices and filesystems. Physical volumes (PV) join a volume group (VG), and logical volumes (LV) are allocated from the VG’s extents (4 MiB each by default). LVs grow while in use, can span disks, and can be snapshotted. Metadata lives on every PV, so a VG assembles on any host that sees its disks.
pvs; vgs; lvs # summaries; -v for more, -a includes hidden volumes
pvs -o +pv_used; vgs -o +vg_free_count; lvs -a -o +devices,segtype # where each LV's extents live
pvdisplay /dev/sdb1; vgdisplay vg0; lvdisplay /dev/vg0/data # verbose forms
lvs -o lv_name,lv_size,data_percent,metadata_percent,snap_percent # fill levels of thin pools and snapshots
pvcreate /dev/sdb1 # label a partition as a PV; refuses if a filesystem signature exists (wipefs first)
vgcreate vg0 /dev/sdb1 /dev/sdc1 # VG from two PVs
lvcreate -n data -L 100G vg0 # LV of 100 GiB
lvcreate -n data -l 100%FREE vg0 # all remaining extents
lvcreate -n data -l 50%VG vg0 # half the VG
lvcreate -n fast -L 50G -i 2 -I 64 vg0 # striped across 2 PVs, 64 KiB stripes
lvcreate -n mirror -L 50G -m 1 vg0 # RAID 1 via the raid1 segment type
mkfs.xfs /dev/vg0/data && mount /dev/vg0/data /mnt/dataGrow: add a PV if the VG is full, then extend the LV with -r so the filesystem grows in the same step.
pvcreate /dev/sdd1 && vgextend vg0 /dev/sdd1 # add a disk to the VG
pvresize /dev/sdb1 # after the partition or virtual disk underneath grew
lvextend -r -L +20G /dev/vg0/data # add 20 GiB and resize the filesystem (ext4, XFS, Btrfs)
lvextend -r -l +100%FREE /dev/vg0/data # use everything left
lvextend -r -L 200G /dev/vg0/data # to an absolute size
lvreduce -r -L 50G /dev/vg0/data # DESTRUCTIVE if the filesystem is larger than 50G; ext4 only, unmounts and shrinks first with -r; XFS cannot shrink
lvremove /dev/vg0/old # DESTRUCTIVE: deletes the LV and its data after a prompt
vgreduce vg0 /dev/sdb1 # remove an empty PV from the VG (pvmove first if it holds extents)
pvmove /dev/sdb1 # migrate extents off a PV online, then vgreduce and pvremove
vgrename vg0 vg1; lvrename vg1 data data-old
vgchange -an vg0; vgexport vg0 # deactivate and mark for moving to another host; vgimport there
vgscan; vgchange -ay # find and activate VGs after adding disksSnapshots are copy-on-write. A classic snapshot needs its own space in the VG to hold the changed blocks of the origin; when it fills, the snapshot becomes invalid (the origin is unaffected). Size it for the writes expected during its life, not the size of the origin.
lvcreate -s -n data-snap -L 10G /dev/vg0/data # snapshot of data with 10 GiB for changed blocks
lvs -o lv_name,origin,snap_percent # watch the fill level
mount -o ro,nouuid /dev/vg0/data-snap /mnt/snap # XFS needs nouuid because the UUID matches the origin
lvconvert --merge /dev/vg0/data-snap # roll the origin back to the snapshot; happens at next activation if the origin is mounted; removes the snapshot
lvremove /dev/vg0/data-snap # discard the snapshotThin provisioning allocates blocks on write from a pool and makes snapshots cheap and unlimited in size, at the cost of a pool that can overfill and take every LV in it offline. Monitor data_percent; lvm.conf thin_pool_autoextend_threshold grows the pool automatically.
lvcreate -L 500G -T vg0/pool # thin pool
lvcreate -V 1T -T vg0/pool -n vm-disks # thin LV bigger than the pool
lvcreate -s -n vm-disks-snap vg0/vm-disks # thin snapshot; no size needed/etc/lvm/backup/ and /etc/lvm/archive/ hold metadata backups after every change; vgcfgrestore -l vg0 lists them and vgcfgrestore -f FILE vg0 restores one, which is how an accidental lvremove is undone before any data is overwritten.
fstab and mount options#
/etc/fstab declares what mounts at boot. systemd generates a .mount unit per line, orders local-fs.target after them, and a failing line without nofail drops the boot into emergency mode.
# <device> <mountpoint> <type> <options> <dump> <pass>
UUID=3f1a2b4c-0d5e-4f6a-8b7c-9d0e1f2a3b4c / xfs defaults 0 0
UUID=A1B2-C3D4 /boot/efi vfat umask=0077,shortname=winnt 0 2
/dev/mapper/vg0-data /srv/data ext4 defaults,noatime,nofail,x-systemd.device-timeout=10s 0 2
LABEL=backup /mnt/backup xfs noauto,nofail,x-systemd.automount,x-systemd.idle-timeout=10min 0 0
nas.example.com:/export/media /mnt/media nfs4 _netdev,nofail,soft,timeo=150,retrans=3,noatime 0 0
tmpfs /var/tmp/build tmpfs size=4G,mode=1777,nosuid,nodev 0 0
/swapfile none swap defaults 0 0Options that matter:
| Option | Effect |
|---|---|
defaults | rw,suid,dev,exec,auto,nouser,async |
noatime | Do not update access times on read; safe for almost everything and removes a write per read. relatime (kernel default) updates once a day |
nofail | Boot continues if the device is missing; combine with x-systemd.device-timeout= so it does not wait 90 s |
noauto | Not mounted by mount -a or at boot; pair with x-systemd.automount to mount on first access |
_netdev | Network filesystem: wait for the network, unmount before it goes down |
nosuid,nodev,noexec | Harden /tmp, /var/tmp, /home, removable media and data mounts |
ro | Read-only |
discard | Issue TRIM on every delete (online discard); a fstrim.timer is usually better |
x-systemd.requires-mounts-for= | Order after another mount, for nested paths |
pass 0 | Never fsck at boot; 1 for root, 2 for the rest on ext4; XFS and Btrfs ignore it |
mount -a # mount everything in fstab not yet mounted; the way to test a new line
findmnt --verify # syntax and device existence check without mounting
systemctl daemon-reload # regenerate mount units after editing fstab (systemd asks for this)
mount -o remount,ro /srv/data # change options on a mounted filesystem
mount --bind /srv/data/www /var/www # bind mount: same filesystem at a second path
mount -o bind,ro /srv/data/www /var/www # read-only bind needs a remount step on old kernels; one step on 5.x+
mount -t tmpfs -o size=1G tmpfs /mnt/scratch
mount -o loop image.iso /mnt/iso # loop device for an image; losetup -f --show image.img for a raw disk image
losetup -Pf --show disk.img # -P scans partitions, giving /dev/loop0p1
umount /mnt/data; umount -l /mnt/data # -l: lazy, detach now and clean up when no longer busy
systemd-mount /dev/sdb1 /mnt/usb; systemd-umount /mnt/usb # transient mount unit with automatic dependency handlingA mount unit name is the path with / replaced by -: /srv/data is srv-data.mount. systemctl status srv-data.mount and journalctl -u srv-data.mount show why a mount failed at boot. Writing a .mount unit instead of an fstab line is equivalent; fstab is simpler to review.
Swap#
Swap gives the kernel somewhere to put anonymous pages under memory pressure. Fedora uses zram (compressed RAM) by default; servers commonly add a swap file or partition sized at a few GiB regardless of RAM so that a leak degrades gracefully instead of triggering the OOM killer.
swapon --show; free -h # current swap devices and usage
fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile # ext4 and XFS; Btrfs needs chattr +C and no snapshots on the file
dd if=/dev/zero of=/swapfile bs=1M count=4096 status=progress # alternative when fallocate is refused
mkswap -L swap /dev/vg0/swap && swapon /dev/vg0/swap # swap on an LV, growable with lvextend then swapoff/mkswap/swapon
swapoff /swapfile # pages back into RAM; fails if RAM cannot hold them
sysctl vm.swappiness; sysctl -w vm.swappiness=10 # 60 default; lower prefers dropping cache over swapping; persist in /etc/sysctl.d/
zramctl # zram devices and compression ratioHibernate needs swap at least the size of RAM and resume= on the kernel command line. Swap on a thin LV or a sparse file is unsafe: the kernel cannot allocate blocks under memory pressure.
SMART#
Drives report their own health through SMART. smartctl reads it; smartd polls and emails or logs when attributes change.
smartctl -i /dev/sda # identity: model, serial, firmware, sector sizes, SMART support
smartctl -H /dev/sda # overall health: PASSED or FAILED (a pass means little; look at the attributes)
smartctl -a /dev/sda # everything: attributes, error log, self-test log
smartctl -a /dev/nvme0 # NVMe: percentage_used, available_spare, media_errors, unsafe_shutdowns
smartctl -A /dev/sda | grep -E 'Reallocated_Sector|Current_Pending|Offline_Uncorrectable|UDMA_CRC|Power_On_Hours|Temperature'
smartctl -t short /dev/sda; smartctl -t long /dev/sda # self-tests; check with -l selftest after the time it reports
smartctl -l error /dev/sda # ATA error log
smartctl -d sat /dev/sdX; smartctl -d megaraid,0 /dev/sda # behind a USB bridge or RAID controller
smartctl --scan # devices and the -d type to useThe attributes that predict failure on spinning disks are Reallocated_Sector_Ct (5), Current_Pending_Sector (197) and Offline_Uncorrectable (198): any non-zero raw value means the disk has already lost data or is about to, and a rising count means replace it now. UDMA_CRC_Error_Count (199) is a cable or backplane problem, not the disk. On NVMe, Percentage Used over 100 and Available Spare under threshold mean end of life; Media and Data Integrity Errors should stay at zero. smartd runs from smartmontools with /etc/smartmontools/smartd.conf; DEVICESCAN -a -o on -S on -s (S/../.././02|L/../../6/03) -m root -M exec /usr/libexec/smartmontools/smartdnotify scans every disk, runs a short test nightly and a long test weekly, and notifies on change.
IO performance#
iostat -xz 1 # per device each second, skip idle ones: r/s w/s rMB/s wMB/s r_await w_await aqu-sz %util
iostat -xzd 5 3 nvme0n1 # one device, three samples five seconds apart
iotop -oPa # processes doing IO (-o only active, -P processes not threads, -a accumulated)
pidstat -d 1 # per-process read/write rate
vmstat 1 # b (blocked on IO), bi/bo, wa (iowait %)
cat /sys/block/sda/queue/scheduler # [mq-deadline] none bfq kyber; none for NVMe, mq-deadline for SATA SSD, bfq for desktop HDD
cat /sys/block/sda/queue/rotational /sys/block/sda/queue/discard_granularity
blockdev --getra /dev/sda; blockdev --setra 4096 /dev/sda # read-ahead in 512-byte sectors
hdparm -tT /dev/sda # crude sequential read benchmark (reads only)
fio --name=randread --filename=/mnt/data/fio.test --size=4G --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --direct=1 --runtime=30 --time_based --group_reporting # IOPS test; creates a 4 GiB file
dd if=/dev/zero of=/mnt/data/dd.test bs=1M count=4096 oflag=direct status=progress # sequential write; creates a 4 GiB file%util near 100 on a spinning disk is saturation; on an NVMe with many queues it is not, so look at aqu-sz and await instead. r_await and w_await are the latency the applications see: over 10 ms on an SSD or over 30 ms on a disk means queueing. Bandwidth without high await is a healthy busy disk. See Linux performance for the wider method.
TRIM tells an SSD or a thin LV which blocks are free. Weekly fstrim.timer (enabled by default on Fedora and RHEL) is preferable to the discard mount option, which issues small discards synchronously on every delete.
fstrim -av # trim every mounted filesystem that supports it; prints bytes trimmed
systemctl enable --now fstrim.timer; systemctl list-timers fstrim.timer
lsblk -D # DISC-GRAN and DISC-MAX: 0B means the device does not accept discardsFor a VM, the virtual disk must be attached with discard=unmap (virtio-scsi or virtio-blk on QEMU 4.0+) for fstrim in the guest to shrink a thin image or thin LV on the host.
mdadm#
Software RAID from disks or partitions. Use partitions of type fd00 (or linux_raid_member autodetection) so a replacement disk can be partitioned identically.
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 # DESTRUCTIVE to those partitions; mirror
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]1 # RAID 5 (one disk of parity)
mdadm --create /dev/md0 --level=6 --raid-devices=4 --chunk=512 /dev/sd[bcde]1 # RAID 6 (two disks of parity)
mdadm --create /dev/md0 --level=10 --raid-devices=4 --layout=f2 /dev/sd[bcde]1 # RAID 10, far layout for read speed
cat /proc/mdstat # state and rebuild progress
mdadm --detail /dev/md0 # members, state, failed and spare counts
mdadm --examine /dev/sdb1 # superblock on a member
mdadm --detail --scan >> /etc/mdadm.conf # persist the array definition; then rebuild the initramfs (dracut -f) if it holds root
mdadm --manage /dev/md0 --fail /dev/sdc1 --remove /dev/sdc1 # mark failed and pull it
mdadm --manage /dev/md0 --add /dev/sdd1 # add a replacement; rebuild starts immediately
mdadm --grow /dev/md0 --raid-devices=4 --add /dev/sde1 # reshape to more disks (slow; keep a backup)
mdadm --grow /dev/md0 --size=max # after replacing all members with larger disks
echo check > /sys/block/md0/md/sync_action; cat /sys/block/md0/md/mismatch_cnt # scrub; raid-check.timer does this monthly on RHEL/Fedora
mdadm --stop /dev/md0; mdadm --zero-superblock /dev/sdb1 # dismantle; zero-superblock is DESTRUCTIVE to the member's RAID metadataecho 200000 > /proc/sys/dev/raid/speed_limit_min speeds a rebuild at the cost of foreground IO. mdadm --monitor --scan --daemonise (or mdmonitor.service) mails MAILADDR from mdadm.conf on failures. RAID is not a backup: it survives a disk, not a deletion.
LUKS with cryptsetup#
LUKS2 stores a header with key slots on the device; each slot holds the master key wrapped by a passphrase or key file. Losing the header loses the data, so back it up. Argon2id is the default key derivation in cryptsetup 2.x and uses about 1 GiB of RAM per unlock by default; lower --pbkdf-memory on small VMs.
cryptsetup luksFormat --type luks2 /dev/sdb1 # DESTRUCTIVE: writes a LUKS header; asks for YES in capitals and a passphrase
cryptsetup luksFormat --type luks2 --pbkdf-memory 262144 --label data-crypt /dev/sdb1 # 256 MiB Argon2 memory
cryptsetup open /dev/sdb1 data-crypt # unlock as /dev/mapper/data-crypt
mkfs.xfs -L data /dev/mapper/data-crypt && mount /dev/mapper/data-crypt /mnt/data
umount /mnt/data && cryptsetup close data-crypt
cryptsetup luksDump /dev/sdb1 # header: slots in use, cipher, PBKDF parameters, UUID
cryptsetup luksHeaderBackup /dev/sdb1 --header-backup-file /root/sdb1-luks-header.img # store off the machine; it grants access with any valid passphrase
cryptsetup luksAddKey /dev/sdb1 # add a second passphrase (prompts for an existing one first)
cryptsetup luksAddKey /dev/sdb1 /root/data.key # add a key file: dd if=/dev/urandom of=/root/data.key bs=64 count=1; chmod 600
cryptsetup luksRemoveKey /dev/sdb1 # remove the passphrase you enter
cryptsetup luksKillSlot /dev/sdb1 1 # remove slot 1
cryptsetup luksChangeKey /dev/sdb1
cryptsetup -v status data-crypt # mapping details
cryptsetup reencrypt --disable-locks --resilience journal /dev/sdb1 # rotate the master key online (LUKS2)
cryptsetup open --type plain --key-file /dev/urandom /dev/sdb1 wipe && dd if=/dev/zero of=/dev/mapper/wipe bs=1M status=progress; cryptsetup close wipe # DESTRUCTIVE: fast random-fill of a disk before formattingUnlock at boot through /etc/crypttab, then reference the mapper device in fstab:
# <name> <device> <keyfile> <options>
data-crypt UUID=6b1f0c8e-2f8a-4c3d-9e1a-7b6c5d4e3f2a /root/data.key luks,discard,nofail
root-crypt UUID=... none luks,discardnone prompts on the console. discard passes TRIM through to the SSD (it leaks which blocks are free, which is acceptable for most threat models). A key file for a data volume on an encrypted root is the usual pattern for servers: root asks for a passphrase or a TPM (systemd-cryptenroll --tpm2-device=auto /dev/sda3) and data unlocks itself from the key stored on root. dracut -f rebuilds the initramfs after changing crypttab entries needed for root.
Growing a VM disk end to end#
The host grows the virtual disk; the guest grows the partition, then the PV, then the LV, then the filesystem. No reboot is needed for a virtio disk when the guest can rescan.
On the host (one of):
qemu-img resize /var/lib/libvirt/images/my-vm.qcow2 +50G # VM stopped; qcow2 or raw
virsh blockresize my-vm /var/lib/libvirt/images/my-vm.qcow2 150G # VM running
qm resize 100 scsi0 +50G # Proxmox
lvextend -L +50G /dev/vg0/vm-100-disk-0 # LV-backed disk, then the VM sees it after a rescan or restartThe host-side commands are covered in libvirt and Proxmox.
In the guest, from the top:
lsblk # confirm the disk (vda) is bigger and which partition holds the PV or filesystem
echo 1 > /sys/class/block/vda/device/rescan # virtio-blk rescan; for virtio-scsi: echo 1 > /sys/class/scsi_device/*/device/rescan
growpart /dev/vda 3 # cloud-utils-growpart: grow partition 3 to the end of the disk; safe online; fixes the GPT backup header too
# without growpart: parted /dev/vda resizepart 3 100% (parted 3.2+ works online; older versions want the partition unmounted)
partprobe /dev/vda # if the kernel did not pick up the new size
pvresize /dev/vda3 # PV sees the bigger partition
lvextend -r -l +100%FREE /dev/rhel/root # LV and filesystem in one step (-r calls xfs_growfs or resize2fs)
df -h / # doneIf there is no LVM, stop after growpart and run xfs_growfs / or resize2fs /dev/vda3. If a swap partition sits between the root partition and the end of the disk, delete it (swapoff, parted rm), grow root, and recreate swap as a file instead. Adding a second virtual disk and running pvcreate, vgextend, lvextend -r avoids partition surgery entirely and is the simplest path when the layout is awkward.
Oneliners#
# Disks with model, serial, size and transport, no partitions
lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA
# Filesystems over 85% full
df -hP -x tmpfs -x devtmpfs | awk 'NR > 1 && $5 + 0 > 85 {print $5, $6}'
# Largest directories under a mount, staying on one filesystem
du -xh --max-depth=2 /var 2>/dev/null | sort -h | tail -20
# Largest files under a path
find /var -xdev -type f -size +500M -printf '%s\t%p\n' 2>/dev/null | sort -rn | numfmt --field=1 --to=iec | head
# Space held by deleted-but-open files, with the process holding them
lsof -nP +L1 | awk 'NR > 1 {print $2, $1, $7, $10}' | sort -k3,3nr | head
# Truncate a deleted log a process still holds (frees the space without restarting it)
: > /proc/1234/fd/5
# Which mount a path lives on and its options
findmnt -T /var/lib/containers -o TARGET,SOURCE,FSTYPE,OPTIONS
# fstab line for a device, ready to paste
printf 'UUID=%s /mnt/data %s defaults,noatime,nofail 0 2\n' "$(blkid -s UUID -o value /dev/sdb1)" "$(blkid -s TYPE -o value /dev/sdb1)"
# Test fstab without rebooting
findmnt --verify && mount -a && systemctl daemon-reload
# Read-only mounts that should not be (a filesystem remounted ro after errors)
findmnt -rn -o TARGET,OPTIONS | awk '$2 ~ /(^|,)ro(,|$)/'
# Kernel messages about a disk
journalctl -k | grep -iE 'sd[a-z]|nvme|I/O error|ext4-fs error|XFS .*error|remount'
# Sector size and alignment of every partition
for d in /dev/sd? /dev/nvme?n1; do [ -e "$d" ] && parted -s "$d" unit s print 2>/dev/null; done
# Physical and logical sector size
cat /sys/block/sda/queue/physical_block_size /sys/block/sda/queue/logical_block_size
# LVM: free extents per VG and where each LV lives
vgs -o vg_name,vg_size,vg_free; lvs -a -o lv_name,vg_name,lv_size,devices,segtype
# LVM: full snapshot or thin pool warning
lvs -o lv_name,data_percent,snap_percent --noheadings | awk '$2 + 0 > 80 || $3 + 0 > 80'
# Progress of a RAID rebuild, refreshed
watch -n 5 cat /proc/mdstat
# SMART summary of every disk
for d in /dev/sd? /dev/nvme?; do [ -e "$d" ] && { printf '%s: ' "$d"; smartctl -H "$d" | grep -E 'result|overall'; }; done
# Pending and reallocated sectors across disks (non-zero means trouble)
for d in /dev/sd?; do printf '%s ' "$d"; smartctl -A "$d" | awk '/Reallocated_Sector|Current_Pending|Offline_Uncorr/ {printf "%s=%s ", $2, $10} END {print ""}'; done
# Disk temperature
smartctl -A /dev/sda | awk '/Temperature_Celsius|Airflow_Temperature/ {print $10}'; smartctl -a /dev/nvme0 | grep -i '^temperature'
# IO latency by device, one shot after 5 seconds
iostat -xzd 5 2 | awk '/^Device/ {h++} h == 2 && NF > 1 {print $1, "r_await", $6, "w_await", $12, "util", $NF}'
# Processes in D state (blocked on IO)
ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /^D/'
# Drop caches and time a cold read of a file (benchmark only)
sync; echo 3 > /proc/sys/vm/drop_caches; time cat /mnt/data/bigfile > /dev/null
# Bytes TRIM would reclaim (dry run has no flag; -v just prints what it trimmed)
fstrim -v /
# Copy a partition table to a new disk for a mirror and give it new GUIDs
sgdisk -R /dev/sdc /dev/sdb && sgdisk -G /dev/sdc
# Image a failing disk to a file, skipping bad blocks (ddrescue is in the ddrescue package)
ddrescue -d -r3 /dev/sdb /mnt/backup/sdb.img /mnt/backup/sdb.map
# Zero a disk before disposal (DESTRUCTIVE; blkdiscard is instant on SSDs that support it)
blkdiscard /dev/sdb || dd if=/dev/zero of=/dev/sdb bs=4M status=progress oflag=direct
# Secure-erase an NVMe (DESTRUCTIVE)
nvme format /dev/nvme0n1 --ses=1
# Fill-level of Btrfs, which df misreports
btrfs filesystem usage -h /
# Find which package or path is using inodes when df -i is full
find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | headScripts#
Disk-space report that lists filesystems past a threshold and the biggest directories on each, suitable for a cron job that mails its output.
#!/usr/bin/env bash
# usage: disk-report.sh [threshold-percent]
set -euo pipefail
thr=${1:-80}
df -hP -x tmpfs -x devtmpfs -x overlay -x squashfs | awk -v thr="$thr" 'NR > 1 && $5 + 0 >= thr {print $6, $5}' | while read -r mnt pct; do
printf '\n== %s at %s ==\n' "$mnt" "$pct"
du -xh --max-depth=2 "$mnt" 2>/dev/null | sort -h | tail -8
printf -- '-- deleted but open:\n'
lsof -nP +L1 -- "$mnt" 2>/dev/null | awk 'NR > 1 {printf " %s (pid %s) %s %s\n", $1, $2, $7, $10}' | sort -u | head -5
doneSMART health check across every disk with a non-zero exit when any attribute that predicts failure is set, for a monitoring hook or a systemd timer.
#!/usr/bin/env bash
# usage: smart-check.sh (root; smartmontools installed)
set -euo pipefail
rc=0
while read -r dev _ type _; do # smartctl --scan prints: /dev/sda -d scsi # comment
if [[ $dev == /dev/nvme* ]]; then
out=$(smartctl -H -A "$dev" 2>/dev/null) || true
bad=$(awk -F: '/Media and Data Integrity Errors/ {gsub(/[ ,]/, "", $2); print $2 + 0}' <<< "$out")
used=$(awk -F: '/Percentage Used/ {gsub(/[ %]/, "", $2); print $2 + 0}' <<< "$out")
health=$(grep -oE 'PASSED|FAILED' <<< "$out" | head -1)
printf '%-14s %s media_errors=%s used=%s%%\n' "$dev" "${health:-unknown}" "${bad:-?}" "${used:-?}"
[[ $health == PASSED && ${bad:-1} -eq 0 && ${used:-100} -lt 95 ]] || rc=1
else
out=$(smartctl -H -A -d "${type:-auto}" "$dev" 2>/dev/null) || true
health=$(grep -oE 'PASSED|FAILED' <<< "$out" | head -1)
counts=$(awk '/Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable/ {printf "%s=%s ", $2, $10; if ($10 + 0 > 0) bad = 1} END {exit bad}' <<< "$out") || rc=1
printf '%-14s %s %s\n' "$dev" "${health:-unknown}" "$counts"
[[ $health == PASSED ]] || rc=1
fi
done < <(smartctl --scan)
exit "$rc"Grow the root filesystem of a VM after the virtual disk was enlarged, detecting LVM or plain partition layouts. Modifies the partition table and filesystem of the running system; read it before running it.
#!/usr/bin/env bash
# usage: grow-root.sh (root; needs cloud-utils-growpart)
set -euo pipefail
src=$(findmnt -no SOURCE /) # /dev/mapper/rhel-root or /dev/vda3
fstype=$(findmnt -no FSTYPE /)
if [[ $src == /dev/mapper/* ]]; then
pv=$(pvs --noheadings -o pv_name -S "vg_name=$(lvs --noheadings -o vg_name "$src" | tr -d ' ')" | tr -d ' ' | head -1)
part=$pv
else
part=$src
fi
disk=$(lsblk -no PKNAME "$part"); num=$(lsblk -no PARTN "$part") # PARTN needs util-linux 2.39+; else: num=${part##*[!0-9]}
printf 'root=%s fstype=%s partition=%s disk=/dev/%s number=%s\n' "$src" "$fstype" "$part" "$disk" "$num"
growpart "/dev/$disk" "$num" || { echo 'partition already fills the disk or growpart failed' >&2; }
if [[ $src == /dev/mapper/* ]]; then
pvresize "$pv"
lvextend -r -l +100%FREE "$src"
else
case $fstype in
xfs) xfs_growfs / ;;
ext4) resize2fs "$part" ;;
btrfs) btrfs filesystem resize max / ;;
*) echo "unsupported filesystem $fstype" >&2; exit 1 ;;
esac
fi
df -h /Troubleshooting#
| Symptom | Cause | Fix |
|---|---|---|
No space left on device but df shows free space | Inodes exhausted (df -i at 100%), or Btrfs metadata chunks full | Delete or move many small files; btrfs balance start -dusage=50 / |
df shows the disk full after deleting big files | A process still holds the deleted file open | lsof +L1, restart the process, or truncate via /proc/PID/fd/N |
du total is far below df used | Files under a mountpoint hidden by a later mount, or deleted-open files | du -x, mount --bind / /mnt/root && du -sh /mnt/root/var, lsof +L1 |
Filesystem suddenly read-only, writes fail with EROFS | Kernel remounted it read-only after IO errors (errors=remount-ro) | journalctl -k, then smartctl -a; unmount and fsck or xfs_repair when the hardware is sound |
| Boot drops to emergency shell | fstab entry for a missing device without nofail | journalctl -xb, fix or nofail the line, systemctl daemon-reload, systemctl default |
Slow IO, %util high, await in the hundreds of ms | Disk saturated, failing (check SMART), or a RAID rebuild | iostat -xz 1, iotop -oPa, cat /proc/mdstat, smartctl -a |
Processes stuck in D state | Waiting on IO to a hung device or an NFS server | ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /^D/', dmesg, umount -f -l for dead NFS |
mount: wrong fs type, bad option, bad superblock | Wrong -t, missing kernel module, or damaged superblock | blkid /dev/sdb1, dmesg | tail, e2fsck -b 32768 /dev/sdb1 for a backup superblock |
target is busy on umount | Open files or a shell cd’d into it | fuser -vm /mnt/data, lsof +f -- /mnt/data; umount -l as a last resort |
Partition grew but lsblk shows the old size | Kernel has not re-read the table | partprobe /dev/sda, partx -u /dev/sda, or echo 1 > /sys/class/block/sda/device/rescan |
pvresize reports no change | Partition not grown, or the kernel still sees the old size | lsblk, growpart, partprobe |
lvextend says Insufficient free space | VG has no free extents | vgs; add a PV with vgextend or grow the existing PV |
xfs_growfs says data size unchanged, skipping | The device under the filesystem did not grow | Grow the LV or partition first and check lsblk |
Btrfs ENOSPC with free space | Chunk allocation exhausted (unallocated 0) | btrfs filesystem usage, btrfs balance start -dusage=20 -musage=20 / |
| LUKS device asks for passphrase at every boot | crypttab entry uses none or the key file is unreachable | Check /etc/crypttab, key file permissions, and dracut -f if root needs the key |
cryptsetup open fails with No key available with this passphrase | Wrong passphrase, or a keyboard layout difference in the initramfs | cryptsetup luksDump for slots, try with --key-file, check KEYMAP in /etc/vconsole.conf |
RAID degraded after reboot, mdadm: no arrays found | mdadm.conf missing the array or the initramfs is stale | mdadm --assemble --scan, mdadm --detail --scan >> /etc/mdadm.conf, dracut -f |
smartctl says Unknown USB bridge or no SMART | USB or RAID controller in the way | smartctl -d sat, -d megaraid,N, -d sntasmedia; smartctl --scan |
| VM does not see the larger disk | Guest needs a rescan, or the disk was resized while the VM was off but not re-read | echo 1 > /sys/class/block/vda/device/rescan, or reboot the guest |
Further reading#
- Red Hat: Managing storage devices: partitions, LVM, RAID, LUKS and NVMe on RHEL 9.
- Red Hat: Configuring and managing logical volumes: LVM including thin provisioning, snapshots and RAID LVs.
- XFS documentation and xfs(5): mount options, geometry and repair.
- ext4 kernel documentation: features and mount options.
- Btrfs documentation: subvolumes, balance, scrub, RAID profiles and the FAQ on free space.
- cryptsetup and LUKS2: the LUKS2 on-disk format, FAQ and recovery guidance.
- Linux RAID wiki: mdadm, recovery and the RAID setup guides.
- smartmontools:
smartctl,smartd.confand the attribute meanings per vendor.