# Files and directories

> Find, copy, sync, archive and watch files on Linux with find, fd, rsync, tar, zstd, du, lsof and inotifywait, and handle permissions and hostile filenames safely.

Canonical: https://www.wiki.jodisand.me/files/
Reviewed: 2026-09-24
Related: [Bash](https://www.wiki.jodisand.me/bash/index.md), [Disks and storage](https://www.wiki.jodisand.me/storage/index.md), [SSH](https://www.wiki.jodisand.me/ssh/index.md), [SCP](https://www.wiki.jodisand.me/scp/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md), [Regular expressions](https://www.wiki.jodisand.me/regex/index.md)


## Cheatsheet

| Task | Command |
| --- | --- |
| Files by name, case-insensitive | `find . -iname '*.log'` or `fd -i '\.log$'` |
| Modified in the last day | `find . -mtime -1` or `fd --changed-within 1d` |
| Larger than 100 MiB | `find . -size +100M` or `fd -S +100M` |
| Delete matches safely | `find . -name '*.tmp' -type f -delete` |
| Run a command per file | `find . -name '*.png' -exec optipng {} \;` or `fd -e png -x optipng` |
| Run once with all files | `find . -name '*.py' -exec ruff check {} +` or `fd -e py -X ruff check` |
| NUL-safe pipeline | `find . -print0 \| xargs -0 cmd` or `fd -0 \| xargs -0 cmd` |
| Mirror a directory | `rsync -a --delete src/ dst/` |
| Preview a sync | `rsync -ain --delete src/ dst/` |
| Copy over SSH with progress | `rsync -a --info=progress2 src/ host.example.com:/dst/` |
| Archive with zstd | `tar --zstd -cf backup.tar.zst dir/` |
| Extract into a directory | `tar -xf backup.tar.zst -C /restore/` |
| What is using space here | `du -xh --max-depth=1 . \| sort -h` or `ncdu -x .` |
| Free space and inodes | `df -hT`, `df -i` |
| Who has this file open | `lsof /var/log/app.log` or `fuser -v /var/log/app.log` |
| Deleted files still held open | `lsof -nP +L1` |
| Watch for changes | `inotifywait -m -r -e close_write,moved_to dir/` |
| Permissions as octal | `stat -c '%a %U:%G %n' file` |
| Make a directory tree | `mkdir -p a/b/c` |
| Resolve a symlink chain | `readlink -f path` |
| Hard link count and inode | `ls -li file` |
| Delete a file named `-rf` | `rm -- -rf` or `rm ./-rf` |
| Safe temp file | `t=$(mktemp)` |

Commands assume GNU findutils 4.10, coreutils 9.x, rsync 3.2+, fd 10 and tar 1.35 on a Linux host. macOS ships BSD variants of `find`, `du`, `stat` and `tar` whose flags differ. References: [GNU findutils](https://www.gnu.org/software/findutils/manual/html_mono/find.html), [rsync](https://download.samba.org/pub/rsync/rsync.1), [GNU tar](https://www.gnu.org/software/tar/manual/tar.html).

## The model: inodes, names and links

A file is an inode: metadata plus data blocks, identified by a number unique within its filesystem. A directory is a table mapping names to inode numbers. `rm` removes a name and decrements the inode's link count; the data is freed only when the count reaches zero and no process holds the file open. That single fact explains hard links, why `df` disagrees with `du` after deleting a log that a daemon still writes, and why `mv` within a filesystem is instant while `mv` across filesystems is a copy followed by a delete.

```sh
ls -li                     # inode number, link count, then the usual columns
stat file                  # inode, links, permissions, owner, three timestamps, filesystem block usage
stat -c '%i %h %s %y %n' file    # inode, link count, size, mtime, name
df --output=source,fstype,size,used,avail,pcent,target /var
```

Three timestamps: `mtime` (content changed), `ctime` (inode changed: content, permissions, owner or link count) and `atime` (read, updated lazily under `relatime`). `touch -d` and `cp -p` set `mtime`; nothing but the clock sets `ctime`, which makes it useful for spotting tampering.

## find

`find` walks a tree and evaluates an expression against each entry. The expression is a sequence of tests and actions joined by implicit `-a` (and); `-o` is or, `!` negates and `\( \)` groups. Evaluation is left to right with short-circuiting, so order matters: `-prune` must come before what it protects, and `-delete` or `-exec` must come last.

```sh
find /var/log -type f -name '*.log' -mtime +30              # regular files, name glob, mtime older than 30 days
find . -type d -name node_modules -prune -o -type f -print  # skip node_modules trees, print everything else
find / -xdev -size +1G -printf '%s\t%p\n' 2>/dev/null | sort -n   # stay on one filesystem; size and path
find . -newer reference.txt                                  # mtime more recent than the reference file
find . -newermt '2026-09-01' ! -newermt '2026-09-08'         # a date window (GNU)
find . -mmin -60                                             # modified in the last 60 minutes
find . -type f -empty -delete                                # empty files; -delete implies -depth
find . -maxdepth 1 -type l ! -exec test -e {} \; -print      # dangling symlinks
find . -perm -4000 -type f                                   # setuid files (at least these bits)
find . -perm /o+w ! -type l                                  # world-writable (any of these bits), ignoring symlinks
find . -user nobody -o -nouser                               # owned by nobody, or by a deleted uid
find . -samefile file.txt                                    # every hard link to that inode
find . -regextype posix-extended -regex '.*/[0-9]{8}\.log'   # regex on the full path
```

`-mtime +30` means more than 30 whole days ago; `-mtime 30` means exactly 30 to 31 days ago; `-mtime -30` within the last 30 days. Use `-mmin` for minute resolution and `-daystart` to count from midnight. `-size +100M` rounds up to the unit; `-size +100M` excludes a 100.5 MiB file only when the unit is `k` and it is written as `-size +102400k`.

Actions:

```sh
find . -name '*.orig' -exec rm -v {} \;        # one rm per file; \; ends the command
find . -name '*.py' -exec ruff check {} +      # as many files per invocation as fit in ARG_MAX
find . -name '*.log' -execdir gzip {} \;       # run in the file's directory with a ./name argument
find . -name core -ok rm {} \;                 # prompt per file
find . -type f -print0 | xargs -0 -P4 sha256sum   # NUL-delimited names survive spaces and newlines; four parallel jobs
find . -type f -printf '%TY-%Tm-%Td %TH:%TM %10s %p\n' | sort   # mtime, size, path
find . -name '*.tmp' -delete                   # no confirmation; -delete forces -depth and refuses to run with ordering tests after it
```

> [!WARNING] `-delete` and `-exec rm` run whatever the tests select
> Run the same expression with `-print` first. `find . -name '*.tmp' -o -name '*.bak' -delete` deletes only `*.bak`, because `-delete` binds to the second test; group with `\( ... \)`.

Filenames may contain newlines, so the only safe way to hand `find` results to another program is `-print0` with `xargs -0`, `-exec`, or `while IFS= read -r -d '' f`. Never `for f in $(find ...)`.

## fd

`fd` searches by regex (or glob with `-g`) on the basename by default, ignores hidden files and anything matched by `.gitignore`, colours output, and runs in parallel. It is the everyday tool; `find` is for expressions `fd` cannot express and for hosts where it is not installed.

```sh
fd pattern                           # regex on the file name, current directory, recursive
fd -e log -e txt                     # by extension
fd -t f -t l                         # type: f file, d directory, l symlink, x executable, e empty
fd -H -I pattern                     # include hidden (-H) and ignored (-I) entries
fd -g '*.tar.*'                      # glob instead of regex
fd -p 'src/.*/test_'                 # match against the full path
fd -d 2 pattern                      # max depth
fd --changed-within 2h               # or 1d, 2weeks; --changed-before for the other side
fd -S +100M -S -1G                   # size between 100 MiB and 1 GiB
fd -E node_modules -E '*.min.js'     # exclude patterns
fd -e jpg -x convert {} {.}.webp     # one command per file; {} path, {.} without extension, {/} basename, {//} parent
fd -e py -X ruff check               # one command with all results as arguments
fd -0 -e log | xargs -0 gzip         # NUL-separated for pipelines
fd -o root -t f /etc                 # owned by root; -o :group and -o user:group also work
fd pattern --exec-batch rm -v        # long form of -X
```

`-x` runs jobs in parallel (`-j` limits it), so commands that write to the same output need `-X` or `-j1`. Use `--no-ignore-vcs` when a `.gitignore` hides what you are looking for and `-u` (unrestricted, twice for everything) when in doubt.

## rsync

rsync sends only the parts of files that differ, decided per file by size and mtime (`-c` compares checksums instead, reading every byte on both sides). It works locally, over SSH, or against an rsync daemon. Everything in the flags below is about which metadata to keep and what to do with files that exist only on the destination.

```sh
rsync -a src/ dst/                           # archive: -rlptgoD (recursive, links, perms, times, group, owner, devices)
rsync -av --info=progress2 src/ dst/         # -v lists files; progress2 shows a single whole-transfer progress line
rsync -aHAX --numeric-ids src/ dst/          # also hard links, ACLs, xattrs; keep uid/gid numbers, for system backups
rsync -ain --delete src/ dst/                # -n dry run, -i itemise what would change; always do this before --delete
rsync -a --delete --delete-excluded --exclude='*.tmp' --exclude='/cache/' src/ dst/
rsync -a --exclude-from=exclude.txt --files-from=list.txt / dst/   # explicit file list; paths relative to the source
rsync -az -e 'ssh -p 2222' src/ user@host.example.com:/srv/dst/     # over SSH with compression (-z)
rsync -a --rsync-path='sudo rsync' src/ host.example.com:/etc/dst/  # elevate on the remote side
rsync -aP --bwlimit=5m src/ host.example.com:/dst/                 # -P = --partial --progress; 5 MiB/s cap
rsync -a --partial-dir=.rsync-partial src/ host.example.com:/dst/  # keep partial files out of the way until complete
rsync -a --remove-source-files src/ dst/     # move semantics for files; empty source directories remain
rsync -a --link-dest=../2026-09-23 src/ dst/2026-09-24/   # unchanged files become hard links to yesterday's copy
rsync -a --mkpath src/ dst/new/deep/path/    # create missing destination directories (3.2.3+)
rsync -a --chown=app:app src/ dst/           # remap ownership on the receiver
```

The trailing slash on the source decides whether the directory itself or its contents are copied. `rsync -a src dst/` creates `dst/src/...`; `rsync -a src/ dst/` creates `dst/...`. The trailing slash on the destination changes nothing. Think of `src/` as "the contents of src".

`--delete` removes files from the destination that are absent from the source, after considering excludes. With `--exclude` alone, excluded files already on the destination stay; `--delete-excluded` removes them too. Deletion happens during the transfer by default; `--delete-after` waits until every file has arrived, which is safer when the destination is served live. A dry run with `-n` combined with `-i` shows each planned action as a string such as `>f.st......` (file transferred because size and time differ) or `*deleting`.

> [!WARNING] `--delete` with a wrong path empties the destination
> `rsync -a --delete empty-or-wrong-dir/ /srv/data/` deletes everything under `/srv/data/`. Run with `-n` first, and prefer variables that fail loudly: `"${SRC:?}/"`.

Interrupted transfers: `--partial` keeps the received part of a file so the next run resumes instead of restarting; `--append-verify` extends a file that was only partly written, and is wrong for files that change in the middle. `--inplace` writes directly to the destination file instead of a temporary file plus rename, which suits large files on a full disk or block devices but leaves a corrupt destination if interrupted. Version 3.2.4 changed remote argument handling (`--old-args` restores the old behaviour when a wrapper depends on shell expansion of remote paths); use `-s` to send arguments through the protocol untouched.

## Archives and compression

`tar` bundles a tree into one stream with permissions, ownership and timestamps; compression is a separate filter. Modern GNU tar detects the compressor on extraction from the file content, so `-xf` works for `.tar.gz`, `.tar.xz` and `.tar.zst` alike.

```sh
tar -cf site.tar site/                        # no compression
tar -czf site.tar.gz site/                    # gzip: fast, everywhere
tar -cJf site.tar.xz site/                    # xz: smallest, slowest
tar --zstd -cf site.tar.zst site/             # zstd: near-gzip ratio at several times the speed
tar -I 'zstd -T0 -19' -cf site.tar.zst site/  # explicit compressor with options: all cores, level 19
tar -tf site.tar.zst | head                   # list
tar -xf site.tar.zst -C /restore/             # extract into a directory that must exist
tar -xf site.tar.zst --strip-components=1 -C /srv/site/   # drop the leading directory
tar -xf site.tar.zst 'site/config/*'          # extract a subset
tar -cf - dir/ | ssh host.example.com 'tar -xf - -C /dst/'   # stream over SSH; see ssh for options
tar --one-file-system --exclude='./proc' --exclude='./sys' -cf - / | zstd -T0 > root.tar.zst   # system snapshot
tar -czf - dir/ | split -b 2G - part.tar.gz.   # split into 2 GiB pieces; cat part.tar.gz.* | tar -xzf -
```

Archive with relative paths (`tar -C /var/www -cf site.tar site`) so extraction lands where the operator chooses. GNU tar strips a leading `/` by default and warns; other tars do not. Verify with `-t` before extracting anything from an untrusted source, and extract as an unprivileged user; GNU tar refuses `..` components unless `-P` is given.

```sh
zstd -T0 -3 big.log                  # writes big.log.zst, keeps the original; -19 for archival, --long for large similar files
zstd -d big.log.zst                  # decompress
zstdcat big.log.zst | grep ERROR     # stream without extracting
zstd --rm -T0 *.log                  # compress and delete originals
gzip -k file; xz -T0 file            # keep original (-k); xz threads (-T0)
zip -r site.zip site/ -x '*.git*'    # zip for Windows recipients; -e prompts for a password (weak legacy encryption)
unzip -l site.zip; unzip -o site.zip -d /restore/   # list; overwrite into a directory
```

zstd level 3 is the default and already beats gzip at any level for speed. `-T0` uses every core for compression; decompression is single-threaded but fast. `--long=31` allows a 2 GiB match window, which helps enormously on VM images and database dumps but requires the same flag to decompress.

## Disk usage

`du` counts allocated blocks under a path; `df` reports what the filesystem says is free. They disagree when deleted files are still open (see `lsof` below), when reserved blocks (5 % on ext4 by default) are counted, when bind mounts or other filesystems sit under the path, or when files are sparse or reflinked.

```sh
du -xsh /var/*  2>/dev/null | sort -h        # per top-level directory, one filesystem (-x), human sizes, sorted
du -xh --max-depth=2 /var | sort -h | tail   # deepest offenders two levels down
du -xa /var/log | sort -n | tail -20         # every file (-a), largest last
du -sh --apparent-size file                  # logical size rather than blocks: shows sparse files as their full length
du -xsh --time /home/*                       # with last modification time per entry
df -hT                                       # every mounted filesystem with type
df -h /var/log                               # the filesystem holding a path
df -i /                                      # inodes; a full inode table looks like a full disk with free space
ncdu -x /                                    # interactive; d deletes (irreversibly), i shows info, n/s/C sort
ncdu -x -o /root/scan.json / && ncdu -f /root/scan.json   # scan once, browse later or elsewhere
```

## Open files and busy mounts

```sh
lsof /var/log/app.log                 # processes with that file open
lsof +D /srv/data                     # everything open under a tree (slow on large trees)
lsof -nP -p 1234                      # every fd of one process; -n no DNS, -P no port names
lsof -nP +L1                          # open files whose link count is 0: deleted but still consuming space
lsof -nP -iTCP -sTCP:LISTEN           # listening sockets, since it is often the same investigation
fuser -vm /mnt/backup                 # processes using anything on that mount (why umount says busy)
fuser -k /mnt/backup                  # send SIGKILL to them; -TERM for a gentler signal, -i to confirm
fuser -v /dev/ttyUSB0                 # who holds a device
```

A deleted-but-open file shows in `lsof` as `(deleted)` and in `/proc/<pid>/fd/` as a symlink. Truncating it through that path (`: > /proc/1234/fd/5`) returns the space without restarting the process; restarting the process or asking it to reopen logs (`kill -HUP`, `logrotate` with `copytruncate`) is the tidy fix.

## Watching for changes

`inotifywait` (package `inotify-tools`) blocks until a filesystem event, or with `-m` streams events forever. It uses the kernel inotify API, which watches directories and the files in them but does not recurse on its own; `-r` adds a watch per subdirectory, bounded by `fs.inotify.max_user_watches`.

```sh
inotifywait -e close_write /etc/app/config.yaml       # wait for one save, then exit
inotifywait -m -r -e close_write,moved_to,create,delete --format '%T %w%f %e' --timefmt '%F %T' /srv/uploads/
inotifywait -m -e modify /var/log/app.log | while read -r _ _ f; do echo "changed: $f"; done
inotifywait -m -r --exclude '\.(swp|tmp)$' -e close_write src/ | while read -r dir ev file; do make; done   # rebuild on save
```

`close_write` is the event to act on for "file finished being written"; `modify` fires for every write call. Editors that save by writing a temporary file and renaming it produce `moved_to`, not `close_write`, on the watched name. For watches that must survive reboots, use a `systemd.path` unit ([systemd](https://www.wiki.jodisand.me/systemd/)) instead of a long-running `inotifywait`; for huge trees or whole-filesystem watching use `fanotify` or `watchman`.

## Permissions, ownership and umask

Each file carries an owner, a group and nine mode bits in three triplets: user, group, other, each `rwx`. On a directory, `r` lists names, `w` creates or removes entries (regardless of the entries' own permissions) and `x` allows traversal into it; a directory with `w` but not `x` is useless. Three extra bits: setuid (4) runs an executable as its owner, setgid (2) on a directory makes new files inherit the directory's group, and the sticky bit (1) on a directory restricts deletion to the entry's owner, as on `/tmp`.

```sh
chmod 640 file                     # rw- r-- ---
chmod u+x,go-w script.sh           # symbolic: add x for user, remove w for group and other
chmod -R u=rwX,go=rX dir/          # X: execute only for directories and files already executable; safe for recursion
chmod 2775 shared/                 # setgid directory: files created inside get the directory's group
chmod 1777 scratch/                # sticky: users delete only their own files
chown app:app file; chown -R app:app dir/; chgrp -R web dir/
chown --reference=other file       # copy owner and group from another file
umask                              # 0022: new files 644, new directories 755
umask 077                          # new files 600, directories 700, for the rest of this shell
install -m 0640 -o app -g app config.yaml /etc/app/config.yaml   # copy with mode and ownership in one step
stat -c '%A %a %U %G %n' /etc/shadow      # ----------  0 root root: symbolic, octal, owner, group
```

The umask is subtracted from the mode a program requests (usually 666 for files and 777 for directories), so `umask 022` gives 644 and 755. It is inherited per process; set it in the service unit (`UMask=` in [systemd](https://www.wiki.jodisand.me/systemd/#writing-a-unit)), in `/etc/login.defs` or in a profile script, not by hoping.

ACLs extend the nine bits to named users and groups, and default ACLs on a directory propagate to new entries. A `+` after the mode in `ls -l` means an ACL is present; `getfacl` shows it.

```sh
setfacl -m u:deploy:rwx /srv/app          # grant one user
setfacl -m d:g:web:rx /srv/app            # default ACL: new entries inside get it
setfacl -R -m g:web:rX /srv/app           # recursive
setfacl -b /srv/app                       # remove all ACL entries
getfacl /srv/app
```

`chattr +i file` makes a file immutable even to root until `chattr -i`; `+a` allows append only, useful for audit logs. `lsattr` lists these. On SELinux hosts the security label is a fourth axis: `ls -Z`, `restorecon -Rv dir/` after moving files into a labelled location, and `chcon` or `semanage fcontext` for persistent exceptions.

## Hard links and symlinks

A hard link is a second name for the same inode: same permissions, same data, same filesystem, indistinguishable from the original. Deleting one name leaves the other intact. Directories cannot be hard-linked. A symlink is a small file containing a path; it can cross filesystems and point at things that do not exist, and permissions on it are ignored in favour of the target's.

```sh
ln file.txt hard.txt               # hard link; ls -li shows the same inode and link count 2
ln -s /srv/app/releases/42 /srv/app/current      # symlink to an absolute path
ln -sfn releases/43 /srv/app/current             # replace atomically-ish: -f overwrite, -n treat existing symlink as a file not a directory
ln -s ../shared/config.yaml config.yaml          # relative target: survives moving the whole tree
readlink current                   # the stored target text
readlink -f current                # fully resolved absolute path, every component
realpath --relative-to=. /srv/app/releases/42
find . -type l -xtype l            # broken symlinks (GNU: -xtype tests the target)
find /srv -links +1 -type f        # files with more than one name
cp -a dir/ copy/                   # preserves symlinks as symlinks; cp -L follows them
rsync -a --copy-links src/ dst/    # follow symlinks and copy the targets instead
```

`ln -sfn` without `-n` on an existing symlink to a directory creates the new link inside the target directory rather than replacing the link. Atomic switch of a `current` symlink: create `current.tmp` then `mv -T current.tmp current`; `mv` is a single `rename(2)`.

Relative symlinks inside a tree that gets copied or bind-mounted keep working; absolute ones point back at the original location. `cp -a` and `rsync -a` preserve the link text, so a relative link is copied relative.

## Odd filenames

Any byte except `/` and NUL is legal in a name, including newlines, leading dashes, spaces, tabs, glob characters and terminal escape sequences. Scripts that handle names they did not create must assume all of them.

```sh
ls -b                              # C escapes for non-printables; ls -q replaces them with ?
ls -li                             # find the inode of the unmanageable one
find . -inum 1234567 -delete       # delete by inode; also: find . -inum 1234567 -exec mv {} sane-name \;
rm -- -rf                          # -- ends option parsing; or rm ./-rf
rm -i -- *                         # confirm each; leading-dash names cannot become options after --
printf '%q\n' "$name"              # show what the shell would need to type it
for f in *; do mv -- "$f" "$(printf '%s' "$f" | tr -c 'A-Za-z0-9._-\n' '_')"; done   # sanitise; check for collisions first
find . -name $'*\n*'               # names containing a newline
find . -depth -name '* *' -execdir rename 's/ /_/g' {} +   # perl rename, package prename or perl-File-Rename; -n previews
detox -r -n dir/                   # detox package: normalise names recursively; -n dry run
iconv -f latin1 -t utf-8 <<< "$name"   # decode a name written under another locale; convmv -r -f latin1 -t utf-8 --notest dir/ renames in place
```

Quote every expansion (`"$f"`), separate with NUL between programs (`-print0`, `-0`, `-z` for `sort`, `xargs`, `grep`, `sed`), and pass `--` before filenames in any command that accepts options. Iterating a glob (`for f in *`) is safe; iterating `$(ls)` is not. See the quoting section of [Bash](https://www.wiki.jodisand.me/bash/#quoting).

## Oneliners

```sh
# Biggest 20 files on this filesystem
find / -xdev -type f -printf '%s\t%p\n' 2>/dev/null | sort -n | tail -20 | numfmt --field=1 --to=iec

# Directories over 1 GiB, one filesystem
du -xh --threshold=1G / 2>/dev/null | sort -h

# Files modified in the last 10 minutes under /etc, newest first
find /etc -type f -mmin -10 -printf '%TF %TT %p\n' | sort -r

# Count files per extension in a tree
fd -t f | sed -E 's/.*\.//' | sort | uniq -c | sort -rn | head

# Delete logs older than 14 days, showing each
find /var/log/app -name '*.log' -type f -mtime +14 -print -delete

# Compress rotated logs that are not yet compressed
find /var/log -name '*.[0-9]' -type f -mtime +1 -exec zstd --rm -T0 -q {} +

# Remove empty directories bottom-up
find /srv/uploads -mindepth 1 -type d -empty -delete

# Duplicate files by content (size first, then hash)
find . -type f -printf '%s %p\n' | sort -n | uniq -Dw10 | awk '{print $2}' | xargs -d '\n' sha256sum | sort | uniq -Dw64

# Same, with fdupes or jdupes when installed
jdupes -r .

# Total size of files matching a glob, NUL-safe
find . -name '*.mp4' -type f -print0 | du -ch --files0-from=- | tail -1

# Sync a tree to a host, deleting extras, showing per-file actions
rsync -aHi --delete --info=progress2 /srv/site/ deploy@host.example.com:/srv/site/

# Pull a remote directory, resumable, capped at 10 MiB/s
rsync -aP --bwlimit=10m host.example.com:/var/backups/ /mnt/backups/

# Copy a directory across hosts preserving everything, root on the far side
rsync -aHAX --numeric-ids --rsync-path='sudo rsync' /etc/ admin@host.example.com:/srv/etc-copy/

# Verify two trees match by checksum, listing differences only
rsync -rcn --delete -i src/ dst/ | grep -v '^\.'

# Stream a directory through zstd over SSH without a temporary archive
tar -C /srv -cf - data | zstd -T0 | ssh host.example.com 'zstd -d | tar -xf - -C /restore'

# Extract a single file from an archive to stdout
tar -xOf backup.tar.zst etc/hosts

# Deleted files holding space, sorted by size
lsof -nP +L1 2>/dev/null | awk 'NR>1 {print $7, $1, $2, $10}' | sort -n | tail

# Which process is writing to this directory right now
inotifywait -m -e modify,create /var/lib/app 2>/dev/null | head -20; lsof +D /var/lib/app

# Wait until a file appears, up to five minutes
timeout 300 inotifywait -q -e create --include 'done\.flag' /srv/jobs/ 

# World-writable files and directories outside /tmp and /proc
find / -xdev \( -path /tmp -o -path /var/tmp \) -prune -o -perm -o+w ! -type l -print 2>/dev/null

# Setuid and setgid binaries, with owner
find / -xdev -type f \( -perm -4000 -o -perm -2000 \) -printf '%M %u %p\n' 2>/dev/null

# Files changed (ctime) in the last day, for spotting unexpected edits
find /usr /etc -xdev -ctime -1 -printf '%CF %CT %p\n' | sort

# Make a directory tree with the same layout but no files
find src -type d -exec mkdir -p dst/{} \;

# Copy with reflink on btrfs or xfs (instant, shares blocks until modified)
cp -a --reflink=auto big.img big-copy.img

# Fill a file with zeros quickly, or allocate sparse
fallocate -l 10G swapfile-candidate; truncate -s 10G sparse.img

# Byte-exact compare two files and show the first difference
cmp file-a file-b

# Rename by regex, previewing first (perl rename)
rename -n 's/\.jpeg$/.jpg/' *.jpeg
```

## Scripts

Back up a directory into dated zstd archives and keep the newest N, deleting older ones.

```sh
#!/usr/bin/env bash
# usage: backup-rotate.sh <source-dir> <backup-dir> [keep=7]
set -euo pipefail
src=${1:?source dir} dst=${2:?backup dir} keep=${3:-7}
[[ -d $src ]] || { printf 'no such directory: %s\n' "$src" >&2; exit 2; }
mkdir -p -- "$dst"
name=$(basename -- "$src")
archive="$dst/$name-$(date +%Y%m%dT%H%M%S).tar.zst"
tmp="$archive.partial"
trap 'rm -f -- "$tmp"' EXIT
tar -C "$(dirname -- "$src")" --one-file-system -cf - -- "$name" | zstd -T0 -q -o "$tmp"
mv -- "$tmp" "$archive"
trap - EXIT
printf 'wrote %s (%s)\n' "$archive" "$(du -h -- "$archive" | cut -f1)"
# Rotation: newest first by name (timestamps sort lexically), delete beyond $keep
mapfile -t old < <(ls -1 -- "$dst"/"$name"-*.tar.zst 2>/dev/null | sort -r | tail -n +"$((keep + 1))")
for f in "${old[@]}"; do rm -v -- "$f"; done
```

Report the largest files and directories on a filesystem, plus deleted-but-open files, in one pass. Run as root for a full view.

```sh
#!/usr/bin/env bash
# usage: disk-report.sh [mountpoint=/] [top=15]
set -euo pipefail
mnt=${1:-/} top=${2:-15}
printf '== %s ==\n' "$mnt"; df -hT -- "$mnt" | tail -1
printf '\n== inodes ==\n'; df -i -- "$mnt" | tail -1
printf '\n== largest directories (one filesystem) ==\n'
du -x --max-depth=3 -- "$mnt" 2>/dev/null | sort -n | tail -n "$top" | numfmt --field=1 --from-unit=1024 --to=iec
printf '\n== largest files ==\n'
find "$mnt" -xdev -type f -printf '%s\t%TF\t%p\n' 2>/dev/null | sort -n | tail -n "$top" | numfmt --field=1 --to=iec
printf '\n== deleted but open ==\n'
lsof -nP +L1 2>/dev/null | awk 'NR>1 && $7 > 1048576 {printf "%s\t%s\t%s\t%s\n", $7, $1, $2, $10}' | sort -n | numfmt --field=1 --to=iec || true
```

Mirror a directory to a remote host with a lock, a dry-run summary on demand and a bandwidth cap, suitable for a systemd timer.

```sh
#!/usr/bin/env bash
# usage: mirror.sh [-n] <src/> <user@host:/dst/>
set -euo pipefail
dry=(); [[ ${1:-} == -n ]] && { dry=(-n -i); shift; }
src=${1:?source} dst=${2:?destination}
exec 9>"/run/lock/mirror-$(basename -- "${src%/}").lock"
flock -n 9 || { echo 'another mirror is running' >&2; exit 0; }
rsync -aH --delete --delete-after --partial-dir=.rsync-partial --bwlimit=20m --timeout=120 \
      --exclude='.rsync-partial/' --exclude='*.tmp' "${dry[@]}" -- "$src" "$dst"
rc=$?
case $rc in
  0) ;;
  24) echo 'warning: some source files vanished during transfer' >&2 ;;   # rsync exit 24 is benign on live trees
  *) echo "rsync failed with $rc" >&2; exit "$rc" ;;
esac
```

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `df` shows the disk full but `du` finds far less | Deleted files still open, or a mount hiding data underneath | `lsof -nP +L1`; `mount --bind / /mnt/root && du -xsh /mnt/root/var` to see under mounts |
| `No space left on device` with free space in `df -h` | Inodes exhausted | `df -i`; find the directory with millions of small files: `find / -xdev -printf '%h\n' \| sort \| uniq -c \| sort -n \| tail` |
| `umount: target is busy` | A process has a file open or its cwd on the mount | `fuser -vm /mnt/x`; `lsof +D /mnt/x`; stop it, or `umount -l` for a lazy detach |
| `Argument list too long` | Glob expanded past `ARG_MAX` | `find ... -exec cmd {} +`, `fd -X`, or `xargs -0` |
| `find: paths must precede expression` | Unquoted glob expanded by the shell before `find` saw it | Quote the pattern: `-name '*.log'` |
| `rsync: connection unexpectedly closed` | Remote rsync missing, SSH login printing output, or `sudo` needing a TTY | `ssh host rsync --version`; silence shell startup output; use `--rsync-path='sudo rsync'` with passwordless sudo |
| rsync copies everything every time | Times not preserved (`-t` missing) or a filesystem with coarse timestamps (FAT, some SMB) | Use `-a`; `--modify-window=1` for 2 s FAT resolution; `--size-only` as a last resort |
| rsync created `dst/src/` instead of syncing contents | Missing trailing slash on the source | `rsync -a src/ dst/` |
| `rsync: failed to set times` or `chown` errors | Destination not owned by you, or no `CAP_CHOWN` | Drop `-o -g` (`rsync -rlptD`), or run as root; `--no-perms --no-owner --no-group` on foreign filesystems |
| `tar: Removing leading '/' from member names` | Absolute paths in the archive | Expected; use `-C` and relative paths when creating |
| `tar: Cannot open: Permission denied` on extract | Extracting as a user into a root-owned directory, or setuid files | Extract as root with `--same-owner`, or into a directory you own |
| `inotifywait: Failed to watch; upper limit on inotify watches reached` | `fs.inotify.max_user_watches` too low for `-r` | `sysctl fs.inotify.max_user_watches=524288`, persist in `/etc/sysctl.d/` |
| `Permission denied` despite `rwx` on the file | Missing `x` on a parent directory, an ACL, SELinux, or `chattr +i` | `namei -l /path/to/file`; `getfacl`; `ls -Z` and `ausearch -m avc -ts recent`; `lsattr` |
| A file cannot be deleted even by root | Immutable or append-only attribute | `lsattr file`; `chattr -i file` |
| `ls` output looks corrupted, names contain control characters | Names with escapes or a wrong locale | `ls -b`; `LC_ALL=C ls`; rename by inode |
| Symlink to a directory ends up inside the target | `ln -sf` without `-n` | `ln -sfn target link`, or `mv -T` a fresh link over the old one |
| `mv` across filesystems is slow and non-atomic | It is a copy plus delete | `rsync -a --remove-source-files` for resumability; keep staging and final directories on one filesystem for atomic renames |

## Further reading

- [GNU findutils manual](https://www.gnu.org/software/findutils/manual/html_mono/find.html)
- [fd documentation](https://github.com/sharkdp/fd#readme)
- [rsync manual](https://download.samba.org/pub/rsync/rsync.1)
- [GNU tar manual](https://www.gnu.org/software/tar/manual/tar.html)
- [Zstandard manual](https://facebook.github.io/zstd/zstd_manual.html) and [zstd(1)](https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md)
- [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html), [path_resolution(7)](https://man7.org/linux/man-pages/man7/path_resolution.7.html)