Software Engineering WikiSE Wiki

FreeIPA / Red Hat IdM

Identity, Kerberos, host enrolment, DNS and certificates in one domain, with the commands for enrolment, RBAC and replication problems.

Reviewed MarkdownEdit

On this page

FreeIPA (packaged and supported as Red Hat Identity Management) is a domain controller for Linux hosts. One enrolment gives a host a Kerberos identity, a certificate path, DNS records and a single policy source, so you stop managing local users, sudoers and TLS trust on each box. Reach for it when more than a handful of Linux hosts need shared accounts, single sign-on or host-based access control; a small static fleet is usually better served by configuration management alone.

Cheatsheet#

TaskCommand
Get a ticketkinit admin
Show ticketsklist
Server healthipactl status
Add a useripa user-add alice --first=Alice --last=Bloggs
Reset a passwordipa passwd alice
Add to a groupipa group-add-member ops --users=alice
Enrol a hostipa-client-install --mkhomedir --domain=example.test
Host keytabipa-getkeytab -s ipa-01.example.test -p host/web-01.example.test@EXAMPLE.TEST -k /etc/krb5.keytab
Service principalipa service-add HTTP/web-01.example.test
Request a certificateipa-getcert request -f /etc/pki/tls/certs/web.crt -k /etc/pki/tls/private/web.key -K HTTP/web-01.example.test
Certificate statusgetcert list
DNS recordipa dnsrecord-add example.test web --a-rec=192.0.2.5
Sudo ruleipa sudorule-add ops-all
HBAC testipa hbactest --user=alice --host=web-01.example.test --service=sshd
Replication statusipa-replica-manage list -v ipa-01.example.test
Client cache resetsss_cache -E then systemctl restart sssd

What the pieces are#

IdM runs several servers behind one command surface. 389 Directory Server holds the identity data, MIT Kerberos issues tickets, Dogtag signs certificates, BIND with an LDAP backend answers DNS, and SSSD on each client caches all of it and enforces policy locally. You administer the lot through the ipa CLI (or the web UI on port 443), which talks to the framework over a Kerberos-authenticated JSON-RPC API.

ComponentResponsibilityFails as
389-dsUsers, groups, hosts, policiesLDAP timeouts, replication conflicts
KDCTickets and service principalsClock skew, missing SPN, wrong realm
DogtagIssuing and renewing certificatesExpired CA subsystem certificates
BIND-DYNDB-LDAPForward and reverse DNSMissing PTR, stale SRV records
SSSD (client)Caching, HBAC, sudo, home directoriesStale cache, offline mode

Kerberos depends on DNS and on time. The client finds servers through _kerberos and _ldap SRV records, and a Kerberos exchange rejects any timestamp more than five minutes out. Those two facts, a missing SRV record and clock skew, are behind most enrolment and login failures. The default ticket lifetime is 24 hours and the maximum renewable age is seven days, both set per Kerberos policy.

Kerberos tickets#

A kinit proves identity once to the KDC and receives a ticket-granting ticket (TGT), cached in a credentials cache (ccache). Every later service access exchanges the TGT for a service ticket without the password, which is what single sign-on means here. The ccache location comes from KRB5CCNAME; a keyring cache (KEYRING:persistent:$UID, the Fedora and RHEL default) survives until logout, while a file cache (FILE:/tmp/krb5cc_$UID) is what most scripts and older tools expect.

kinit alice                                # password prompt; TGT valid for the policy lifetime
kinit -kt /etc/krb5.keytab host/web-01.example.test   # from a keytab, no password, for services and cron
kinit -R                                   # renew the current TGT without re-authenticating, up to the renew lifetime
klist -f                                   # flags: F forwardable, R renewable, and the renew-until time
klist -e                                   # encryption types; aes256-cts-hmac-sha1-96 or aes256-cts-hmac-sha384-192 expected, never RC4 (arcfour)
klist -kt /etc/krb5.keytab                 # principals and key version numbers (KVNO) held in a keytab
kdestroy                                   # discard the current ccache
kswitch -p alice@EXAMPLE.TEST              # switch between cached principals when a collection cache holds several

Service tickets are cached alongside the TGT; a stale service ticket after a keytab rotation shows as a KVNO mismatch even though kinit works, so kdestroy then re-request. kvno HTTP/web-01.example.test asks the KDC for a fresh service ticket and prints the version, which is the quick check that a service principal exists and its keytab is current.

kvno HTTP/web-01.example.test              # request a service ticket; fails if the SPN is missing
ipa krbtpolicy-mod --maxlife=8h --maxrenew=$((7*24))h   # tighten the default ticket lifetime domain-wide
ipa krbtpolicy-mod alice --maxlife=1h      # per-user override for a privileged account
KRB5_TRACE=/dev/stderr kvno HTTP/web-01.example.test    # every step, for diagnosing "no support for encryption type"

Delegation is the sharp edge. Unconstrained delegation lets a service impersonate the user to any other service and is a serious finding; prefer constrained delegation (ipa servicedelegationrule) that names the target services. Forwardable tickets (kinit -f) are what allow SSH with GSSAPI to carry your identity to the next host; disable forwarding for accounts that never need it.

Users and groups#

The user and group objects are the identities every other policy references. Disable a departed user rather than deleting: disabling revokes access at once while keeping the UID, group memberships and file ownership, so audit trails and on-disk permissions stay intact. Delete only when you are certain nothing on any host still refers to the UID.

ipa user-add alice --first=Alice --last=Bloggs --email=alice@example.test --shell=/bin/bash
ipa user-mod alice --title='Platform Engineer' --sshpubkey="$(cat ~/.ssh/id_ed25519.pub)"
ipa user-find --all --raw alice        # --raw shows LDAP attribute names, useful for scripting
ipa user-disable alice                 # blocks authentication; keeps the object and memberships
ipa user-del alice --preserve          # moves to the deleted-users container, recoverable with user-undel
ipa passwd alice                       # sets a one-time password the user must change at next login
ipa group-add ops --desc='Platform on-call'
ipa group-add-member ops --users=alice --groups=contractors   # groups can nest
ipa group-show ops --all

Password policy is per group and evaluated by priority (lower number wins when a user is in several). Set a stricter policy on privileged groups than on the global default.

ipa pwpolicy-add ops --maxlife=90 --minlength=14 --history=10 --priority=1
ipa pwpolicy-show ops

Groups have a type. A POSIX group has a GID and appears in getent group; a non-POSIX group exists only in the directory for RBAC or as a mail list; an external group holds references to identities from a trusted Active Directory and is how AD users receive IdM policy. Convert a plain group to POSIX with ipa group-mod --posix; you cannot go back. User private groups (one per-user group matching the UID) are created automatically unless the install disabled them.

ipa group-add app-owners --nonposix                       # RBAC only, no GID
ipa group-add-member ops --external='EXAMPLE\\ad-admins'  # AD group via a trust; needs --external on a group created with --external
ipa group-add ops-posix --posix                           # allocates a GID from the ID range
ipa idrange-find                                          # the UID/GID ranges IdM assigns from

Automember rules add a new user or host to groups automatically from an attribute, so joiners land in the right groups without a manual step. Rebuild membership after adding a rule; automember only fires on creation otherwise.

ipa automember-add ops --type=group
ipa automember-add-condition ops --type=group --key=title --inclusive-regex='Platform'
ipa automember-rebuild --type=group                       # apply to existing users; can be heavy on a large directory

Hosts and services#

A host must exist as an object before it can hold a keytab, and enrolment creates that object and its host/ principal. A service (a web server, a database) gets its own principal so it can accept Kerberos tickets or hold a certificate independently of the host.

ipa host-add web-01.example.test --ip-address=192.0.2.5
ipa-client-install --domain=example.test --realm=EXAMPLE.TEST \
  --principal=admin --mkhomedir --enable-dns-updates   # run on the client; needs root
ipa service-add HTTP/web-01.example.test
ipa service-allow-retrieve-keytab HTTP/web-01.example.test --hosts=web-01.example.test
ipa-getkeytab -s ipa-01.example.test -p HTTP/web-01.example.test -k /etc/httpd.keytab
klist -kt /etc/httpd.keytab                            # list principals and key versions in the keytab

Warning

By default ipa-getkeytab resets the principal’s key and writes the new one to the keytab, which invalidates every keytab already issued for that principal. Two hosts sharing a principal will knock each other offline. Use one principal per host per service, or pass -r to retrieve the existing key without regenerating it (retrieval must be permitted with service-allow-retrieve-keytab first).

ipa-client-install --uninstall         # clean removal on the client before re-enrolment
ipa host-del web-01.example.test       # then remove the stale host entry on a server

DNS#

IdM can run the domain’s DNS so that enrolment maintains the SRV, A and PTR records clients depend on. Use the integrated DNS when hosts come and go; keep an external DNS if you already operate one, but then you must publish the _kerberos and _ldap SRV records yourself.

ipa dnszone-add example.test --dynamic-update=TRUE            # allow enrolled hosts to update their records
ipa dnsrecord-add example.test web --a-rec=192.0.2.5
ipa dnsrecord-add 2.0.192.in-addr.arpa 5 --ptr-rec=web.example.test.   # note the trailing dot
ipa dnsrecord-find example.test
ipa dnsconfig-mod --forwarder=192.0.2.53 --forward-policy=only
dig +short SRV _kerberos._udp.example.test                   # what a client resolves to find the KDC

If the SRV records are wrong or missing, enrolment fails with an unhelpful message about the realm rather than about DNS. Check them first. See DNS for resolving the underlying lookup.

Certificates#

Dogtag issues X.509 certificates from the IdM CA, and certmonger tracks each one on the host, renewing it before expiry and running a post-save command to reload the service. A certificate issued outside this tracking is the one that expires unnoticed. See TLS for verifying the resulting chain.

ipa-getcert request -f /etc/pki/tls/certs/web.crt -k /etc/pki/tls/private/web.key \
  -K HTTP/web-01.example.test -D web-01.example.test        # -K principal, -D SAN dNSName
getcert list                      # tracked certificates: expiry, state, and the post-save command
getcert resubmit -i <request-id>  # force renewal now instead of waiting for the auto-renewal window
ipa cert-find --subject=web-01
ipa cert-show 12 --out=cert.pem
ipa cert-revoke 12 --revocation-reason=4    # reason 4 = superseded
ipa-cacert-manage renew            # renew the CA certificate; run on the CA renewal master only
ipa-certupdate                     # refresh the local trust store on every host after a CA change

Important

Keep at least two replicas with CA capability and confirm the CA renewal master is healthy before certificates approach expiry. If the CA subsystem certificates lapse, issuance and renewal stop for the whole domain.

Access control#

HBAC (host-based access control) decides who may reach which service on which host; sudo rules decide what they may then run as root. SSSD on the client evaluates both against cached data, so a policy change is not live until the cache refreshes. Service names (sshd, login) map to PAM services on the client, so an HBAC rule that gates SSH must name sshd.

ipa hbacrule-add ops-ssh
ipa hbacrule-add-user ops-ssh --groups=ops
ipa hbacrule-add-host ops-ssh --hostgroups=web-servers
ipa hbacrule-add-service ops-ssh --hbacsvcs=sshd
ipa hbactest --user=alice --host=web-01.example.test --service=sshd   # simulate before you rely on it
ipa sudorule-add ops-restart
ipa sudorule-add-user ops-restart --groups=ops
ipa sudorule-add-host ops-restart --hostgroups=web-servers
ipa sudorule-add-allow-command ops-restart --sudocmds='/bin/systemctl restart nginx'
ipa sudorule-mod ops-restart --sudoopt='!authenticate'    # no re-prompt for the password

Warning

The default allow_all HBAC rule permits every user on every host. Disable it only after replacement rules pass hbactest, or the next login locks everyone out.

ipa hbacrule-disable allow_all

HBAC service groups let one rule cover several PAM services. The distinction between sshd and login matters: sshd gates network SSH, login the local console, and a rule that only names sshd still lets someone at the keyboard in. sudo is a separate PAM service again, so an HBAC rule that omits it does not stop sudo from prompting, only the sudo rules do.

ipa hbacsvcgroup-add remote-access
ipa hbacsvcgroup-add-member remote-access --hbacsvcs=sshd,cockpit
ipa hbacrule-add-service ops-ssh --hbacsvcgroups=remote-access
ipa hbacrule-find --all | grep -A3 'Rule name: ops-ssh'

Sudo rules mirror HBAC in shape: subjects (users or groups), hosts (or host groups), and what they may run (command or command group), plus run-as and options. Build command groups so a rule references one named set rather than a list that drifts.

ipa sudocmd-add '/usr/bin/systemctl restart nginx'
ipa sudocmdgroup-add web-restart
ipa sudocmdgroup-add-member web-restart --sudocmds='/usr/bin/systemctl restart nginx'
ipa sudorule-add-allow-command ops-restart --sudocmdgroups=web-restart
ipa sudorule-mod ops-restart --runasusers=root --runasgroups=root
ipa sudorule-add-option ops-restart --sudooption='!authenticate'
ipa sudorule-add-option ops-restart --sudooption='logfile=/var/log/sudo-ops.log'
ssh alice@web-01 sudo -l                 # confirm from the client after sss_cache -E; the client compiles the rules, not the server

The client fetches sudo rules over LDAP through SSSD, not from /etc/sudoers. sssctl shows what the client actually resolved, which is the fastest way to tell a rule problem from a cache problem.

sssctl user-checks alice -s sudo         # what SSSD returns for sudo for this user
sudo -l -U alice                          # the compiled sudo policy, run as root on the client

Delegate administration with roles instead of adding people to admins. A role holds privileges, a privilege holds permissions, and a permission grants a specific right on specific attributes. This keeps a helpdesk able to reset passwords without being able to change group membership or policy.

ipa role-add 'Helpdesk'
ipa privilege-add 'Password Reset'
ipa permission-add 'Reset user password' --type=user --right=write --attrs=userPassword
ipa privilege-add-permission 'Password Reset' --permissions='Reset user password'
ipa role-add-privilege 'Helpdesk' --privileges='Password Reset'
ipa role-add-member 'Helpdesk' --groups=helpdesk

Replication#

Replicas hold full read-write copies of the directory, so any one can serve the whole domain and no single server is a point of failure. Run at least two, and at least two with CA capability. Replication is multi-master, which means a write on two replicas to the same entry can collide and produce a conflict that needs manual resolution.

ipa-replica-manage list -v ipa-01.example.test   # agreements and last-update time per peer
ipa-replica-manage force-sync --from ipa-01.example.test
ipa-csreplica-manage list                        # certificate-server (Dogtag) replication
ipa-healthcheck --failures-only                  # run on each server; the fastest triage

Replication conflicts appear as directory entries carrying nsds5ReplConflict. Resolve them by choosing the surviving entry and deleting the conflict copy.

ldapsearch -Y GSSAPI -b "dc=example,dc=test" "(nsds5ReplConflict=*)" \* nsds5ReplConflict

Keycloak and OIDC#

FreeIPA authenticates Linux hosts with Kerberos and LDAP; web and API single sign-on wants OIDC or SAML, which is Keycloak’s job. A common layout is Keycloak as the OIDC provider federated to FreeIPA over LDAP for the account source, so people have one password and applications speak OIDC. Keycloak calls its tenant a realm (unrelated to a Kerberos realm), and each application is a client.

Every realm exposes standard OIDC endpoints under /realms/<realm>/protocol/openid-connect/, discoverable at the well-known URL. Point applications at the discovery URL and let them read the rest.

# Discovery document: every endpoint, supported grant types and signing keys
curl -s https://sso.example.com/realms/example/.well-known/openid-configuration | jq '{authorization_endpoint, token_endpoint, userinfo_endpoint, end_session_endpoint, jwks_uri, grant_types_supported}'
EndpointPathUse
Authorization.../authBrowser redirect that starts the authorization-code flow
Token.../tokenExchange a code, refresh token or client credentials for tokens
UserInfo.../userinfoClaims about the user, given an access token
Logout.../logoutEnd the SSO session (RP-initiated logout)
JWKS.../certsPublic keys to verify a token signature

The flows, and when to use each:

  • Authorization code with PKCE: interactive users in a browser or mobile app. The app never sees the password; it receives a short-lived code and swaps it for tokens server-side (or with PKCE, safely in a public client). This is the default and the only one to use for people.
  • Client credentials: a service authenticating as itself, no user. Returns an access token for the client’s own service account.
  • Refresh token: swap a refresh token for a new access token without re-prompting, within the refresh token’s lifetime.
  • Device code: input-constrained devices; the user visits a URL and enters a code shown on the device.
  • Direct access grants (password): the app collects the password and sends it. Only for trusted first-party tooling or migration; disable it on every client that does not need it, because it defeats MFA and the SSO session.
# Client credentials: a service getting its own token
curl -s -X POST https://sso.example.com/realms/example/protocol/openid-connect/token \
  -d grant_type=client_credentials -d client_id=my-service -d client_secret="$CLIENT_SECRET" | jq -r .access_token

# Inspect a token you already hold (header and claims; does not verify the signature)
jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "$ACCESS_TOKEN"

# Verify and introspect a token server-side (needs a confidential client's credentials)
curl -s -X POST https://sso.example.com/realms/example/protocol/openid-connect/token/introspect \
  -u "my-service:$CLIENT_SECRET" -d token="$ACCESS_TOKEN" | jq '{active, sub, preferred_username, realm_access}'

kcadm.sh is the admin API client; it authenticates once and stores the token, then creates and edits realm objects. Store the admin password out of the command line.

kcadm.sh config credentials --server https://sso.example.com --realm master --user admin --password-stdin <<< "$KC_ADMIN_PASSWORD"
kcadm.sh create realms -s realm=example -s enabled=true
kcadm.sh create clients -r example -s clientId=my-app -s 'redirectUris=["https://my-app.example.com/*"]' \
  -s publicClient=false -s standardFlowEnabled=true -s directAccessGrantsEnabled=false -s 'attributes."pkce.code.challenge.method"=S256'
kcadm.sh get clients -r example --fields id,clientId | jq -r '.[] | [.clientId, .id] | @tsv'
kcadm.sh create clients/<client-id>/client-secret -r example        # generate a secret
kcadm.sh get clients/<client-id>/client-secret -r example -F value
kcadm.sh create users -r example -s username=alice -s enabled=true -s email=alice@example.test
kcadm.sh set-password -r example --username alice --temporary   # prompts; user must reset at first login
kcadm.sh add-roles -r example --uusername alice --rolename platform-admin

Roles arrive in the token under realm_access.roles (realm roles) or resource_access.<client>.roles (client roles); a client maps them into the token with a role mapper, and a group mapper adds groups. An application authorises on those claims. Map FreeIPA groups through the LDAP federation’s group mapper so IdM stays the source of truth for membership, and see Vault OIDC for the same claims driving Vault policy.

Client troubleshooting#

Most login problems are visible from the client with a ticket request and the SSSD logs. Trace the Kerberos exchange when the error is unclear, and check the clock first when nothing else explains it.

klist -e                                   # current tickets and their encryption types
kinit -V alice                             # verbose ticket request
KRB5_TRACE=/dev/stderr kinit alice         # every step of the AS/TGS exchange
id alice                                   # does the client resolve the user through SSSD at all
getent passwd alice
sss_cache -E && systemctl restart sssd     # clear the SSSD cache after a policy change
journalctl -u sssd -f                      # follow SSSD while you reproduce the failure
realm list                                 # domain membership from the client's point of view
chronyc tracking                           # clock offset; skew over five minutes breaks Kerberos
SymptomCauseCheck
Clock skew too greatTime drift beyond five minuteschronyc tracking; fix NTP before anything else
Server not found in Kerberos databaseMissing SPN, or a name with no matching principalipa service-find; confirm the exact principal name
KDC has no support for encryption typeMismatched enctypes after an upgradeRegenerate the keytab; check the principal’s krbEncType
Login works, sudo does notSudo rule unmatched, or the SSSD cache is stalesss_cache -E; verify sudorule host and user members
User exists in IdM but id failsSSSD domain disabled, or the host is not enrolledrealm list; sssctl domain-status example.test
Everything fails after a CA renewalClients hold the old CA trustipa-certupdate on each client
Intermittent failuresOne replica unhealthy and still in rotationipa-healthcheck --failures-only on every server
kinit works but a service rejects the ticketStale service ticket after a keytab rotation, KVNO mismatchkdestroy; kvno HTTP/host; compare klist -kt KVNO with the KDC
SSH single sign-on stops carrying identityTicket not forwardable, or GSSAPI delegation offklist -f for the F flag; kinit -f; GSSAPIDelegateCredentials yes
id slow or times out on the clientSSSD querying an unreachable or slow replicasssctl domain-status example.test; set ldap_uri failover; journalctl -u sssd
Automember rule added but existing users not in the groupAutomember only fires on creationipa automember-rebuild --type=group
Sudo rule shows in ipa sudorule-show but not sudo -l on the hostSSSD cache stale, or sudo not in an HBAC service allowed on the hostsss_cache -E; sssctl user-checks alice -s sudo
OIDC login loops or invalid redirect_uriThe client’s redirectUris does not match the app’s callback exactlykcadm.sh get clients/<id> redirectUris; add the exact URL
Token accepted by one app, rejected by anotherAudience (aud) or issuer (iss) mismatch, or clock skew to KeycloakDecode the token’s aud/iss; check the app’s expected issuer and NTP
Keycloak login fails for an IdM userLDAP federation bind or search base wrong, or the account is disabled in IdMKeycloak server log; ipa user-show; test the bind DN with ldapsearch
MFA bypassed for some loginsA client has direct access grants (password grant) enabledkcadm.sh get clients -r example --fields clientId,directAccessGrantsEnabled

Oneliners#

# Last successful Kerberos authentication per user
ipa user-find --all --raw | awk '/^ *uid:/ {u=$2} /^ *krbLastSuccessfulAuth:/ {print u, $2}'

# Every member of a group, flattened
ipa group-show ops --all --raw | grep -E '^ *member(user|group):'

# Hosts that exist but were never enrolled (no keytab)
ipa host-find | awk '/Host name:/ {h=$3} /Keytab: False/ {print h}'

# Tracked certificates with their expiry and status
getcert list | grep -E 'Request ID|status:|expires:|certificate:'

# All HBAC rules and who they apply to
ipa hbacrule-find --all --raw | grep -E '^ *(cn|memberuser|memberhost|memberservice):'

# Test access for a user before telling them it works
ipa hbactest --user=alice --host=web-01.example.test --service=sshd

# Replication agreements and their last update
ipa-replica-manage list -v ipa-01.example.test | grep -E 'last update|status'

# Confirm the DNS SRV records a client will use to find servers
dig +short SRV _ldap._tcp.example.test _kerberos._tcp.example.test

# Force a full client refresh after policy changes
sss_cache -E && systemctl restart sssd && id alice

# Users whose password expires in the next 14 days
ipa user-find --all --raw | awk '/^ *uid:/ {u=$2} /^ *krbPasswordExpiration:/ {print $2, u}' | awk -v d="$(date -u -d '+14 days' +%Y%m%d)" '$1 < d""000000Z'

# Disabled but not yet deleted accounts (offboarding follow-up)
ipa user-find --disabled --raw | grep -E '^ *uid:'

# Accounts that have never logged in
ipa user-find --all --raw | awk '/^ *uid:/ {u=$2; s=0} /^ *krbLastSuccessfulAuth:/ {s=1} /^$/ {if (u && !s) print u; u=""}'

# Every host and whether it is enrolled, one per line
ipa host-find --raw | awk '/^ *fqdn:/ {h=$2} /^ *has_keytab: True/ {print h, "enrolled"} /^ *has_keytab: False/ {print h, "NOT enrolled"}'

# Hosts that have not contacted the domain recently (stale enrolments)
ipa host-find --all --raw | awk '/^ *fqdn:/ {h=$2} /^ *krbLastSuccessfulAuth:/ {print $2, h}' | sort

# Tracked certificates expiring within 30 days on this host
getcert list | awk '/Request ID/ {id=$0} /expires:/ {print id, $0}' | grep -v "$(date -d '+30 days' +%Y-%m-%d)"

# Which principals a keytab holds and their key versions
klist -kt /etc/krb5.keytab | awk 'NR>3 {print $4, $1}' | sort -u

# Request a fresh service ticket to confirm an SPN resolves and the keytab is current
for s in HTTP ldap host; do printf '%-6s ' "$s"; kvno "$s/$(hostname -f)" 2>&1 | tail -1; done

# All group memberships for one user, resolved through SSSD (indirect groups included)
id -Gn alice | tr ' ' '\n' | sort

# HBAC rules that apply to a given host group
ipa hbacrule-find --all --raw | awk '/^ *cn:/ {r=$2} /web-servers/ {print r}'

# Members of the admins group (the accounts to watch most closely)
ipa group-show admins --all --raw | grep -E '^ *member' | sed 's/^ *//'

# Confirm every server is healthy in one pass
for s in ipa-01 ipa-02; do echo "== $s =="; ssh "$s" ipa-healthcheck --failures-only 2>/dev/null || echo 'healthcheck reported failures'; done

# Kerberos ticket lifetime remaining, in minutes
klist | awk '/krbtgt/ {print $3, $4}' | while read -r d t; do echo $(( ( $(date -d "$d $t" +%s) - $(date +%s) ) / 60 )) min left; done

# Decode a Keycloak access token's claims without verifying
jq -R 'split(".")[1] | @base64d | fromjson | {sub, preferred_username, exp, realm_access}' <<< "$ACCESS_TOKEN"

# Keycloak: list clients that still allow the password grant (an MFA gap)
kcadm.sh get clients -r example --fields clientId,directAccessGrantsEnabled | jq -r '.[] | select(.directAccessGrantsEnabled) | .clientId'

# Keycloak: users who have not set OTP but are in an MFA-required group
kcadm.sh get users -r example -q briefRepresentation=true --fields id,username | jq -r '.[] | [.id, .username] | @tsv'

# Verify the SRV records a joining client depends on, both protocols
for r in _ldap._tcp _kerberos._tcp _kerberos._udp; do printf '%-16s ' "$r"; dig +short SRV "$r.example.test" | head -1; done

# Confirm client clock is within Kerberos tolerance of a server
offset=$(chronyc tracking | awk '/Last offset/ {print $4}'); echo "offset ${offset}s (must be under 300)"

Scripts#

Offboarding: disable a user, remove them from privileged groups, kill live tickets by expiring the account, and print what was done for the audit trail. Disables rather than deletes so file ownership and audit history survive. Run on an IdM server with a fresh kinit admin.

#!/usr/bin/env bash
# usage: offboard.sh USERNAME   (disables the account and strips privileged group membership)
set -euo pipefail
user=${1:?username required}
klist -s || { echo 'no valid ticket; run kinit admin first' >&2; exit 1; }
ipa user-show "$user" >/dev/null || { echo "no such user: $user" >&2; exit 1; }

printf '== before ==\n'
ipa user-show "$user" --all --raw | grep -E '^ *(uid|nsaccountlock|memberof_group):' || true

# Remove from privileged groups; ignore "not a member" errors
for g in admins ops platform-admins helpdesk; do
  ipa group-remove-member "$g" --users="$user" 2>/dev/null && printf 'removed from %s\n' "$g" || true
done

ipa user-disable "$user"
# Expire the password so cached credentials and any keytab-less access stop working
ipa user-mod "$user" --setattr=krbPasswordExpiration=19700101000000Z >/dev/null

printf '\n== after ==\n'
ipa user-show "$user" --all --raw | grep -E '^ *(uid|nsaccountlock|memberof_group):' || true
printf '\nDisabled %s at %s. Object preserved; delete with `ipa user-del --preserve` only after confirming no host references the UID.\n' "$user" "$(date -u +%FT%TZ)"

Certificate expiry report across the fleet: asks each enrolled host for its certmonger-tracked certificates and flags any inside the warning window, so a stalled renewal is caught before an outage. Read-only; uses SSH.

#!/usr/bin/env bash
# usage: cert-report.sh hosts.txt [WARN_DAYS]   (one host per line; needs SSH and getcert on each)
set -euo pipefail
hosts=${1:?hosts file}; warn=${2:-21}
rc=0
while IFS= read -r h; do
  [[ $h =~ ^[[:space:]]*(#|$) ]] && continue
  # getcert prints "expires: 2026-12-01 ..." lines; compute days for each
  out=$(ssh -o ConnectTimeout=10 -o BatchMode=yes "$h" 'getcert list 2>/dev/null | grep -E "status:|expires:|certificate:"' 2>/dev/null) \
    || { printf 'FAIL %-30s unreachable\n' "$h"; rc=1; continue; }
  awk -v host="$h" -v warn="$warn" '
    /status:/   { status=$2 }
    /expires:/  { exp=$2 " " $3 }
    /certificate:/ {
      cmd="date -d \"" exp "\" +%s"; cmd | getline e; close(cmd)
      days=int((e - systime())/86400)
      if (status != "MONITORING" || days < warn)
        printf "%-4s %-30s %4d days  %s\n", (days<warn?"WARN":"ok"), host, days, status
    }' <<< "$out" || true
  grep -q 'WARN' <<< "$out" && rc=1 || true
done < "$hosts"
exit "$rc"

Bulk user import from a CSV (uid,first,last,email,group), idempotent: existing users are updated and re-added to the group rather than failing the run. Creates accounts, so review the CSV first.

#!/usr/bin/env bash
# usage: import-users.sh users.csv   (header: uid,first,last,email,group)
set -euo pipefail
csv=${1:?csv file}
klist -s || { echo 'run kinit admin first' >&2; exit 1; }

tail -n +2 "$csv" | while IFS=, read -r uid first last email group; do
  [[ -n $uid ]] || continue
  if ipa user-show "$uid" >/dev/null 2>&1; then
    ipa user-mod "$uid" --first="$first" --last="$last" --email="$email" >/dev/null
    printf 'updated %s\n' "$uid"
  else
    ipa user-add "$uid" --first="$first" --last="$last" --email="$email" --random >/dev/null
    printf 'created %s (random password set; user must reset)\n' "$uid"
  fi
  if [[ -n ${group:-} ]]; then
    ipa group-show "$group" >/dev/null 2>&1 || ipa group-add "$group" --desc="imported" >/dev/null
    ipa group-add-member "$group" --users="$uid" >/dev/null 2>&1 || true
  fi
done

Further reading#