Software Engineering WikiSE Wiki

SSH

Configure OpenSSH clients and servers, manage keys and agents, forward ports through bastions and read the debug output that names the real failure.

Reviewed MarkdownEdit

On this page

Cheatsheet#

TaskCommand
Connect as another userssh alice@host.example.com
Why did it failssh -vvv host.example.com
Which key was offered and acceptedssh -v host 2>&1 | grep -E 'Offering|Server accepts|Authenticated'
Run one commandssh host 'uptime'
Copy a public key to a hostssh-copy-id -i ~/.ssh/id_ed25519.pub host
New keyssh-keygen -t ed25519 -C 'alice@laptop'
Load a key into the agentssh-add ~/.ssh/id_ed25519
Keys the agent holdsssh-add -l
Through a bastionssh -J bastion.example.com host
Local forwardssh -L 5432:db.internal.example:5432 bastion
Remote forwardssh -R 8080:localhost:3000 host
SOCKS proxyssh -D 1080 bastion
Host key fingerprintssh-keyscan host | ssh-keygen -lf -
Drop a stale host keyssh-keygen -R host
Effective client config for a hostssh -G host
Effective server configsudo sshd -T
Reuse a connectionControlMaster auto plus ControlPath, ControlPersist in config

How an SSH connection is set up#

Knowing the phases tells you which one failed when ssh -v stops.

  1. TCP connect to port 22. A timeout here is routing or a firewall, not SSH. See iproute2.
  2. Version exchange and key exchange. Client and server agree on algorithms and derive session keys. OpenSSH 10.0 made the hybrid post-quantum mlkem768x25519-sha256 the default key exchange, and 10.1 prints a warning when a connection falls back to a non-post-quantum one.
  3. Host key verification. The server proves it holds the private host key; the client compares the public key with ~/.ssh/known_hosts. A mismatch stops the connection here.
  4. User authentication. The client tries methods in order (public key, then keyboard-interactive or password if allowed). With public keys the client offers each key and the server says which it will accept, so the agent contents and IdentitiesOnly matter.
  5. Channels. The shell, commands, forwarded ports and the agent all run as multiplexed channels over the one encrypted connection.

Client configuration#

~/.ssh/config is read top to bottom and the first value obtained for each option wins, so put specific hosts above wildcards and Host * last. /etc/ssh/ssh_config and its ssh_config.d/*.conf drop-ins are read after the user file.

Host bastion
  HostName bastion.example.com
  User alice
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes

Host db-*
  ProxyJump bastion
  User postgres
  ForwardAgent no

Host *
  AddKeysToAgent yes
  ServerAliveInterval 30
  ServerAliveCountMax 3
  ControlMaster auto
  ControlPath ~/.ssh/cm-%C
  ControlPersist 10m
  HashKnownHosts yes
  StrictHostKeyChecking accept-new
OptionWhy
IdentitiesOnly yesOffer only the listed key. Without it the agent offers every key it holds and the server can hit MaxAuthTries before reaching the right one
ControlMaster/ControlPersistLater sessions reuse one authenticated connection, so repeat logins skip key exchange and authentication
ControlPath ~/.ssh/cm-%C%C is a hash of local host, remote host, port and user, so each target gets its own socket and the path stays short
ServerAliveIntervalSends an encrypted keepalive so NAT and firewall idle timers do not silently drop the session
ProxyJumpConnect through a bastion. The bastion only forwards TCP, so neither your key nor your agent is exposed on it
ForwardAgentAnyone with root on the remote host can use your agent while you are connected. Prefer ProxyJump
StrictHostKeyChecking accept-newRecord a first-time host key automatically but still refuse a changed one
ssh -G host                       # every option that will apply, after Host/Match evaluation
ssh -o ProxyJump=none host        # override config for one command
ssh -F /dev/null host             # ignore all config files, useful to rule config out

Note

Fedora and RHEL ship /etc/ssh/ssh_config.d/50-redhat.conf, which sets GSSAPIAuthentication yes and includes the system crypto policy. ssh -G shows the result of those files too.

Match, Include and ProxyCommand#

Host matches only the name typed on the command line. Match evaluates conditions, so one block can apply by user, by canonicalised hostname, by network, or by the result of a command. Include splits a large config into files; the included lines are processed at the point of the Include, so first-match-wins still applies across files.

Include ~/.ssh/config.d/*.conf          # per-project files; must sit above any Host * block that would win first

Match host *.internal.example !exec "nc -zw1 192.0.2.1 22"   # not on the office network: go via the bastion
  ProxyJump bastion

Match localnetwork 192.0.2.0/24          # OpenSSH 9.4+: match when a local interface has an address in this range
  ProxyJump none

Match user root
  IdentityFile ~/.ssh/id_root
  IdentitiesOnly yes

Match tagged prod                        # selected with ssh -P prod host (OpenSSH 9.4+)
  RequestTTY yes
  RemoteCommand tmux new -A -s main

Match canonical host db-*
  User postgres

Match final all                          # runs once more after canonicalisation, useful for defaults
  ServerAliveInterval 30

exec runs a shell command and matches on exit status; keep it fast because it runs on every connection. canonical and final re-evaluate the config after CanonicalizeHostname has expanded short names with CanonicalDomains, which is how Host db-01 can pick up a domain and still hit the right block.

ProxyJump is a wrapper around ProxyCommand ssh -W %h:%p bastion. Write ProxyCommand yourself when the hop is not SSH, for example a cloud console or a SOCKS proxy:

Host i-*                                                       # AWS instance IDs through SSM (plugin installed separately)
  ProxyCommand aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p
Host *.corp.example
  ProxyCommand nc -X 5 -x 127.0.0.1:1080 %h %p                 # through a SOCKS5 proxy

%h and %p are the target host and port after Host and HostName substitution; %r is the remote user.

Keys and the agent#

Ed25519 is the ssh-keygen default since OpenSSH 9.5: short keys, fast, and no parameters to get wrong. Use RSA (3072 bits or more) only for systems that do not support Ed25519. DSA support was removed in OpenSSH 10.0. For hardware-backed keys use -t ed25519-sk with a FIDO2 token.

ssh-keygen -t ed25519 -C 'alice@laptop'                 # prompts for a passphrase
ssh-keygen -t ed25519 -f ~/.ssh/id_deploy -N ''         # no passphrase: automation only
ssh-copy-id -i ~/.ssh/id_ed25519.pub host               # appends to remote authorized_keys
ssh-keygen -lf ~/.ssh/id_ed25519.pub                    # SHA256 fingerprint
ssh-keygen -y -f ~/.ssh/id_ed25519 > id_ed25519.pub     # regenerate a lost public key
ssh-keygen -p -f ~/.ssh/id_ed25519                      # change the passphrase

The agent holds decrypted keys in memory so you type the passphrase once. Most desktops start one; otherwise start it yourself.

eval "$(ssh-agent -s)"                 # start an agent and export SSH_AUTH_SOCK
ssh-add ~/.ssh/id_ed25519
ssh-add -l                             # fingerprints of loaded keys
ssh-add -t 3600 ~/.ssh/id_ed25519      # key expires from the agent after an hour
ssh-add -D                             # remove every key from the agent

sshd refuses keys when permissions are loose (StrictModes yes, the default). Use 700 on ~/.ssh, 600 on private keys and authorized_keys, and make sure the home directory is not group or world writable. A good key with bad permissions produces a plain Permission denied (publickey); the server log gives the real reason.

Restrict what a key may do with options at the start of its authorized_keys line:

restrict,pty,command="/usr/local/bin/backup" ssh-ed25519 AAAA...placeholder backup@ci
from="198.51.100.0/24",restrict ssh-ed25519 AAAA...placeholder deploy@ci

restrict turns off forwarding, the agent, X11 and the PTY; add back only what is needed. command= forces that command whatever the client asked for. from= limits source addresses.

ssh-keygen -t ed25519 -a 100 -f ~/.ssh/id_ed25519      # -a: KDF rounds protecting the passphrase (default 16)
ssh-keygen -c -C 'alice@laptop-2026' -f ~/.ssh/id_ed25519   # change the comment
ssh-add -L                                              # public keys in the agent, in authorized_keys format
ssh-add -d ~/.ssh/id_ed25519                            # remove one key
ssh-add -x                                              # lock the agent with a password; -X unlocks

Certificates#

A certificate is a public key signed by a CA key with an identity, a list of principals and a validity window. The server trusts the CA instead of each key, so authorized_keys files disappear and revocation is a validity date rather than a fleet-wide edit. Host certificates work the same way in reverse and end Host key verification failed after rebuilds.

ssh-keygen -t ed25519 -f ~/ca/user_ca -C 'user CA'      # keep this offline or in a signing service

# Sign a user key: identity (logged by sshd), principals (login names it may use), 12-hour validity
ssh-keygen -s ~/ca/user_ca -I alice@laptop -n alice,deploy -V +12h ~/.ssh/id_ed25519.pub
# writes ~/.ssh/id_ed25519-cert.pub; ssh sends it automatically next to the private key

# Restrict what the certificate allows, regardless of sshd settings
ssh-keygen -s ~/ca/user_ca -I ci-runner -n deploy -V +1h -O clear -O force-command=/usr/local/bin/deploy -O source-address=198.51.100.0/24 id_ci.pub

# Host certificate: principals are the names clients will type
ssh-keygen -s ~/ca/host_ca -I host-01 -h -n host-01.example.com,192.0.2.10 -V -1d:+52w /etc/ssh/ssh_host_ed25519_key.pub

ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub             # print identity, principals, validity, options

Server side, trust the user CA and present the host certificate:

TrustedUserCAKeys /etc/ssh/user_ca.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%u    # optional: accept principals listed here, not only the login name
RevokedKeys /etc/ssh/revoked_keys                  # KRL from ssh-keygen -k, checked on every login

Client side, trust the host CA in known_hosts so every host it signs is accepted without a prompt. The pattern limits which names the CA may vouch for:

@cert-authority *.example.com,192.0.2.* ssh-ed25519 AAAA...placeholder host CA
@revoked host-old.example.com ssh-ed25519 AAAA...placeholder

Revoke before expiry with a key revocation list. sshd re-reads the file on each authentication, so the change is immediate:

ssh-keygen -k -f /etc/ssh/revoked_keys -s ~/ca/user_ca.pub -z 1 compromised-cert.pub   # -s CA: revoke by serial/ID; -z KRL version
ssh-keygen -k -u -f /etc/ssh/revoked_keys another.pub                                  # -u: add to the existing KRL
ssh-keygen -Q -f /etc/ssh/revoked_keys ~/.ssh/id_ed25519-cert.pub                       # is this key or cert revoked

Principals are matched against the login name unless AuthorizedPrincipalsFile is set. A certificate whose principals do not include the user you asked for fails with the same Permission denied (publickey) as a missing key; sshd -T and the journal line Certificate invalid: name is not a listed principal tell them apart.

Port forwarding#

ssh -L 5432:db.internal.example:5432 bastion      # local 5432 -> db.internal.example:5432, via bastion
ssh -R 8080:localhost:3000 host                   # host's port 8080 -> my local port 3000
ssh -D 1080 bastion                               # SOCKS5 proxy on local 1080
ssh -fN -o ExitOnForwardFailure=yes -L 5432:db.internal.example:5432 bastion   # background, no shell
ssh -J bastion db-01                              # jump host, nothing stored or forwarded on bastion

-L listens locally and connects out from the remote side. -R listens on the remote side and connects back through you. Both bind to loopback by default: -L needs -g or an explicit bind address to accept other clients, and -R needs GatewayPorts yes in the server’s sshd_config. ExitOnForwardFailure=yes makes ssh exit when the port is already in use instead of running without the tunnel.

pkill -f 'ssh -fN.*-L 5432'      # stop a backgrounded forward
ssh -O check bastion             # is a control master running for this host
ssh -O exit bastion              # close the master and every session using it

Forwards can be added to a live connection instead of reconnecting. With a control master running, ssh -O forward and ssh -O cancel change the master’s forwards; inside an interactive session, the ~C escape opens a command line that accepts the same -L/-R/-D syntax, and ~# lists active forwards.

ssh -O forward -L 8443:web.internal.example:443 bastion     # add to the running master
ssh -O cancel -L 8443:web.internal.example:443 bastion      # remove it
ssh -L /tmp/pg.sock:/var/run/postgresql/.s.PGSQL.5432 host  # Unix socket on both ends; psql -h /tmp
ssh -R 0:localhost:3000 host                                # port 0: server picks a free port and prints it
ssh -W db.internal.example:5432 bastion                     # stdin/stdout to a TCP port; what ProxyJump uses underneath

The server bounds what a client may forward. PermitOpen lists the destinations -L and -D may reach, PermitListen the ports -R may bind, and AllowTcpForwarding (yes, no, local, remote) switches each direction off entirely. Set them per group in a Match block so a bastion forwards only to the internal networks it exists for:

Match Group bastion-users
  AllowTcpForwarding local
  PermitOpen db.internal.example:5432 web.internal.example:443   # host:port entries; * wildcards a host or port, no CIDR
  PermitListen none
  PermitTTY no
  ForceCommand /usr/bin/false

Server configuration#

sshd reads /etc/ssh/sshd_config. The first value obtained for each keyword wins, and most distributions put Include /etc/ssh/sshd_config.d/*.conf at the top, so a drop-in overrides the same keyword in the main file. Put your settings in a drop-in such as /etc/ssh/sshd_config.d/10-hardening.conf so package updates do not conflict with them.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
AllowGroups ssh-users
MaxAuthTries 3
LoginGraceTime 20
X11Forwarding no
AllowAgentForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2

PermitRootLogin defaults to prohibit-password, which already blocks password logins as root. AllowGroups rejects everyone outside the group before authentication.

Validate, then reload. A reload re-reads config for new connections; existing sessions keep running.

sudo sshd -t                                        # syntax and key check, silent on success
sudo sshd -T | grep -i passwordauthentication       # effective value after drop-ins
sudo sshd -T -C user=alice,host=host.example.com,addr=198.51.100.7   # effective values for one connection, with Match applied
sudo systemctl reload sshd                          # unit is "ssh" on Debian and Ubuntu

Do not lock yourself out

Keep the current session open, reload, and verify a fresh login from a second terminal before closing anything. On a cloud instance, confirm console access first.

Match blocks apply only to connections that match, and must come after the global settings:

Match Group sftp-only
  ChrootDirectory /srv/sftp/%u
  ForceCommand internal-sftp
  AllowTcpForwarding no

The chroot directory and every parent must be owned by root and not writable by any other user, or sshd drops the session after authentication.

Since OpenSSH 9.8, PerSourcePenalties makes sshd temporarily refuse addresses that repeatedly fail authentication or crash the pre-auth process. This replaces much of what fail2ban was used for, and can also block a legitimate client that retried a bad key many times.

Troubleshooting#

ssh -vvv host 2>&1 | tail -40                          # client view: kex, key offers, auth methods
sudo journalctl -u sshd -f                             # server view: the actual rejection reason
ssh -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -i ~/.ssh/id_ed25519 host   # test one key only
ssh-keyscan -t ed25519 host | ssh-keygen -lf -         # compare against the fingerprint you expect
Message or symptomCauseCheck
Connection timed outRouting or firewall; SSH never startednc -vz host 22
Connection refusedNothing listening, or wrong portss -tlnp | grep sshd on the server
Permission denied (publickey)Key not in authorized_keys, wrong user, or permissions too openServer journal; ssh -v for which keys were offered
Too many authentication failuresAgent offered several keys before the right oneSet IdentitiesOnly yes and IdentityFile for that host
Host key verification failedHost rebuilt, IP reused, or a man-in-the-middleVerify the fingerprint out of band, then ssh-keygen -R host
Connection closed by remote host before authAllowUsers/AllowGroups, MaxStartups, PerSourcePenalties or fail2banServer journal
no matching host key type found / no matching key exchange methodOne side only offers algorithms the other has disabled (old device, or crypto policy)ssh -Q kex, ssh -vv for both offer lists
Hangs after expecting SSH2_MSG_KEX_ECDH_REPLYPath MTU problem: large key exchange packets droppedLower MTU on the path, or test -o KexAlgorithms=curve25519-sha256
Stalls on some networks onlyMiddlebox mishandling the DSCP mark (interactive traffic uses EF since OpenSSH 10.1)-o IPQoS=none
Slow login, fast once connectedClient trying GSSAPI (on by default in Fedora/RHEL client config), or server UseDNS yes with slow reverse DNSssh -vvv timestamps; -o GSSAPIAuthentication=no
Certificate invalid: name is not a listed principal in the journalCertificate principals do not include the login name, or AuthorizedPrincipalsFile lists other namesssh-keygen -L -f id-cert.pub, compare with the user in ssh -v
Certificate invalid: expiredValidity window passed, or clock skew between signer and serverssh-keygen -L -f id-cert.pub; timedatectl on both sides
bind: Address already in use and the tunnel silently missingLocal or remote port taken; without ExitOnForwardFailure ssh continues anywayss -tlnp | grep :5432; add -o ExitOnForwardFailure=yes
-R forward reachable only from the server’s loopbackGatewayPorts is no, or PermitListen excludes the portsudo sshd -T | grep -Ei 'gatewayports|permitlisten'
channel N: open failed: administratively prohibitedAllowTcpForwarding no or PermitOpen excludes the destinationsudo sshd -T -C user=alice | grep -Ei 'allowtcpforwarding|permitopen'
Bad owner or permissions on ~/.ssh/configConfig file writable by group or otherschmod 600 ~/.ssh/config
ControlSocket ... already exists or mux_client_request_session errorsStale control socket from a dead masterssh -O exit host or remove the socket in ControlPath
sign_and_send_pubkey: signing failed ... agent refused operationKey in the agent is locked, expired, or the FIDO token needs a touchssh-add -l; ssh-add -X if locked; watch the token
Session drops after exactly N minutes idleServer ClientAliveInterval/ClientAliveCountMax or a NAT timerSet ServerAliveInterval 30 client side; sudo sshd -T | grep clientalive
Chrooted SFTP user logs in then disconnectsChrootDirectory or a parent not owned by root, or group/world writablenamei -l /srv/sftp/alice; server journal bad ownership or modes for chroot directory

File transfer over SSH#

scp file host:/tmp/                          # one file; see the scp page for syntax
rsync -avzP --delete ./dir/ host:/srv/dir/   # incremental and resumable; --delete removes extra files on the target
sftp host                                    # interactive, or scripted with -b batchfile
ssh host 'tar czf - /var/log' > logs.tgz     # stream without staging a file
tar czf - ./dir | ssh host 'tar xzf - -C /srv'

rsync sends only differences and can resume, so use it for anything larger than a config file. See scp for copy syntax and the SFTP protocol change.

SFTP#

sftp is the interactive client for the SFTP subsystem that scp now uses underneath. It has local (l-prefixed) and remote commands, resumes transfers, and runs scripted through a batch file.

sftp -P 2222 alice@host.example.com          # -P: port (capital, unlike ssh)
sftp sftp://alice@host.example.com:2222/srv/  # URI form, starting in /srv
sftp -r host:/srv/logs ./logs                 # recursive download in one go; symlinks are not followed
sftp -a host:/srv/big.iso .                   # resume a partial download
sftp -l 20000 host:/srv/big.iso .             # limit to 20000 Kbit/s

Interactive commands that matter:

sftp> ls -l                        # remote listing; lls is local
sftp> cd /srv/app; lcd ~/out        # remote and local working directories; pwd and lpwd show them
sftp> get -p config.yaml            # -p keeps mtime and mode; -a resumes; -R recursive
sftp> put -R ./release/ /srv/app/   # upload a tree
sftp> reget big.iso                 # same as get -a
sftp> rename app.yaml app.yaml.bak  # rename is atomic on the server
sftp> rm /srv/app/old.log           # deletes on the server
sftp> df -h                         # free space on the remote filesystem
sftp> !ls -l                        # run a local shell command
sftp> bye

Batch mode reads commands from a file and aborts on the first failing command, which is what you want in automation. Prefix a command with - to ignore its failure. It requires non-interactive authentication (a key in the agent or an unencrypted key), because -b implies BatchMode yes.

cat > upload.sftp <<'EOF'
-mkdir /srv/app/releases
put -p ./release.tgz /srv/app/releases/release-2026-09-24.tgz
rename /srv/app/releases/release-2026-09-24.tgz /srv/app/releases/current.tgz
EOF
sftp -b upload.sftp -o ConnectTimeout=10 deploy@host.example.com
sftp -b - host <<< 'ls -l /srv/app'          # commands from stdin

Server side, Subsystem sftp internal-sftp runs SFTP inside sshd itself, which is required inside a ChrootDirectory because there is no /usr/libexec/openssh/sftp-server visible from the chroot. Add -l INFO (Subsystem sftp internal-sftp -l INFO) to log every file operation to the journal.

Oneliners#

# Run a command on many hosts, 8 at a time
printf '%s\n' host{1..20}.example.com | xargs -P8 -I{} ssh -o ConnectTimeout=5 -o BatchMode=yes {} 'uptime'

# Copy a public key without ssh-copy-id
ssh host 'umask 077; mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_ed25519.pub

# Persistent tunnel that reconnects (autossh is a separate package)
autossh -M 0 -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes -N -L 5432:db.internal.example:5432 bastion

# Reachability test from inside the network, through a bastion
ssh -J bastion host 'nc -zv db.internal.example 5432'

# Fingerprints of every key in authorized_keys
ssh-keygen -lf ~/.ssh/authorized_keys

# Show, then remove, a host's known_hosts entry after a rebuild
ssh-keygen -F host; ssh-keygen -R host

# Compare a local and remote file without copying
diff <(ssh host 'sha256sum < /etc/app.conf') <(sha256sum < /etc/app.conf)

# Mount a remote directory (sshfs is a separate package)
sshfs host:/srv/data /mnt/data -o reconnect,ServerAliveInterval=15

# Measure login time with and without connection reuse
time ssh -o ControlPath=none host true; time ssh host true

# Which host key algorithms, kex and ciphers this client supports
ssh -Q key; ssh -Q kex; ssh -Q cipher

# Host key fingerprints of every key type a server presents, for comparing out of band
ssh-keyscan -t ed25519,rsa,ecdsa host.example.com 2>/dev/null | ssh-keygen -lf -

# Pre-seed known_hosts for a batch of new hosts (only when the network path is trusted)
ssh-keyscan -t ed25519 host{1..5}.example.com 2>/dev/null >> ~/.ssh/known_hosts

# Hash an existing known_hosts in place; the .old backup is left beside it
ssh-keygen -H -f ~/.ssh/known_hosts

# Accept a rotated host key without dropping the rest of the file
ssh-keygen -R host.example.com && ssh -o StrictHostKeyChecking=accept-new host.example.com true

# One key, one host, no agent: rule out config and agent noise
ssh -F /dev/null -o IdentitiesOnly=yes -o IdentityAgent=none -i ~/.ssh/id_ed25519 alice@host

# Debug output with timestamps to a file, terminal stays clean
ssh -vvv -E /tmp/ssh-debug.log host true; grep -n 'Authenticat' /tmp/ssh-debug.log

# Print a certificate's identity, principals and validity
ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub

# Days until a certificate expires
ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub | awk '/Valid:/ {print $NF}'

# Run a local script on a remote host without copying it first
ssh host 'bash -s -- -v' < ./check.sh

# Run a command as root via sudo with a TTY for the password prompt
ssh -t host 'sudo systemctl restart my-app'

# Stream a remote log with a heartbeat that survives idle NAT timers
ssh -o ServerAliveInterval=15 host 'journalctl -fu my-app'

# Open a tmux session, reattaching if it exists
ssh -t host 'tmux new -A -s main'

# Interactive port forward added to a live session: type ~C then -L 8443:web.internal.example:443

# Background SOCKS proxy that dies when the terminal does not
ssh -fN -D 1080 -o ExitOnForwardFailure=yes bastion && curl --proxy socks5h://127.0.0.1:1080 https://intranet.example.com/

# List the forwards an existing master holds, then close it
ssh -O check bastion; ssh -O exit bastion

# Copy a directory tree preserving ownership when rsync is missing on the target
tar cf - -C /srv app | ssh host 'tar xpf - -C /srv'

# Remote disk usage across a fleet as CSV, with a timeout so a dead host does not block
for h in host{1..5}.example.com; do printf '%s,' "$h"; ssh -o BatchMode=yes -o ConnectTimeout=5 "$h" "df --output=pcent / | tail -1" || echo unreachable; done

# Effective sshd value for one user from one address, with Match blocks applied
sudo sshd -T -C user=deploy,addr=198.51.100.7 | grep -Ei 'forcecommand|permitopen|allowtcpforwarding'

# Failed logins by source address today
sudo journalctl -u sshd --since today | grep -oE 'Failed .* from [0-9a-f.:]+' | awk '{print $NF}' | sort | uniq -c | sort -rn | head

# Who is logged in over SSH right now, and from where
who --ips

# Sign a file with an SSH key and verify it with an allowed-signers list
ssh-keygen -Y sign -f ~/.ssh/id_ed25519 -n file release.tgz
ssh-keygen -Y verify -f allowed_signers -I alice@laptop -n file -s release.tgz.sig < release.tgz

Scripts#

Report which hosts in a list accept your key, with the OpenSSH version they run, without ever prompting.

#!/usr/bin/env bash
# usage: ssh-fleet-check hosts.txt
set -euo pipefail
hosts=${1:?hosts file required}
opts=(-o BatchMode=yes -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new -o LogLevel=ERROR)
printf 'host\tstatus\tsshd\n'
while IFS= read -r host || [[ -n $host ]]; do
  [[ -z $host || $host == \#* ]] && continue
  if ver=$(ssh "${opts[@]}" "$host" 'ssh -V 2>&1' 2>/dev/null); then   # client and server ship in one package
    printf '%s\tok\t%s\n' "$host" "$ver"
  elif nc -zw3 "$host" 22 2>/dev/null; then
    printf '%s\tauth-failed\t-\n' "$host"
  else
    printf '%s\tunreachable\t-\n' "$host"
  fi
done < "$hosts"

Issue short-lived user certificates from a CA, keeping the serial monotonic so a KRL can revoke by serial later.

#!/usr/bin/env bash
# usage: sign-user-key <pubkey> <identity> <principals> [validity]
set -euo pipefail
ca=${SSH_CA:-$HOME/ca/user_ca}
pub=${1:?public key}; ident=${2:?identity}; princ=${3:?principals}; valid=${4:-+8h}
serial_file=${ca}.serial
serial=$(( $(cat "$serial_file" 2>/dev/null || echo 0) + 1 ))
ssh-keygen -s "$ca" -I "$ident" -n "$princ" -V "$valid" -z "$serial" \
  -O clear -O permit-pty -O permit-port-forwarding "$pub"
printf '%s\n' "$serial" > "$serial_file"
printf '%s\t%s\t%s\t%s\t%s\n' "$(date -u +%FT%TZ)" "$serial" "$ident" "$princ" "$valid" >> "${ca}.log"
ssh-keygen -L -f "${pub%.pub}-cert.pub" | grep -E 'Serial|Valid|Principals'

Rotate a host’s authorized_keys from a Git-managed file, keeping a backup and refusing to install an empty or malformed set.

#!/usr/bin/env bash
# usage: push-authorized-keys keys.txt host [host...]
set -euo pipefail
keys=${1:?keys file}; shift
[[ -s $keys ]] || { echo "refusing to push an empty key file" >&2; exit 1; }
ssh-keygen -lf "$keys" >/dev/null                      # fails when the file holds no parseable public key
for host in "$@"; do
  ssh -o BatchMode=yes -o ConnectTimeout=5 "$host" 'umask 077; mkdir -p ~/.ssh
    [ -f ~/.ssh/authorized_keys ] && cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak
    cat > ~/.ssh/authorized_keys.new && mv ~/.ssh/authorized_keys.new ~/.ssh/authorized_keys' < "$keys"
  ssh -o BatchMode=yes -o ConnectTimeout=5 "$host" true \
    && printf '%s: ok (%s keys)\n' "$host" "$(grep -c '^ssh-\|^sk-' "$keys")" \
    || { printf '%s: login failed after push; backup is ~/.ssh/authorized_keys.bak on the host\n' "$host" >&2; exit 1; }
done

Further reading#