# Argo CD

> Diagnose Argo CD Applications that will not sync or stay healthy, and control drift, sync order, pruning and multi-tenant access.

Canonical: https://www.wiki.jodisand.me/argocd/
Reviewed: 2026-09-24
Related: [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Helm](https://www.wiki.jodisand.me/helm/index.md), [Git](https://www.wiki.jodisand.me/git/index.md)


## Cheatsheet

Commands target Argo CD 3.x. The `argocd` CLI needs `argocd login` first; the `kubectl` forms work with cluster access alone.

| Task | Command |
| --- | --- |
| Application state and conditions | `argocd app get my-app` |
| Live vs desired, field by field | `argocd app diff my-app` |
| Sync now | `argocd app sync my-app` |
| Preview a sync | `argocd app sync my-app --dry-run` |
| Sync one resource | `argocd app sync my-app --resource apps:Deployment:my-app` |
| Re-read Git and re-render | `argocd app get my-app --hard-refresh` |
| Wait until synced and healthy | `argocd app wait my-app --sync --health --timeout 600` |
| Stop a running sync | `argocd app terminate-op my-app` |
| History and roll back | `argocd app history my-app`, then `argocd app rollback my-app <id>` |
| What Argo CD rendered | `argocd app manifests my-app --source git` |
| What exists in the cluster | `argocd app manifests my-app --source live` |
| Delete resources removed from Git | `argocd app sync my-app --prune` |
| Application YAML | `kubectl get application my-app -n argocd -o yaml` |
| Controller logs | `kubectl logs -n argocd statefulset/argocd-application-controller` |
| Render errors | `kubectl logs -n argocd deploy/argocd-repo-server --tail 100` |

## An Application that will not sync

Argo CD compares two states: desired (the manifests rendered from Git) and live (what the cluster has). Sync status (`Synced`, `OutOfSync`) is that comparison. Health (`Healthy`, `Progressing`, `Degraded`, `Suspended`, `Missing`, `Unknown`) is a separate assessment of the live objects only. An Application can be `Synced` and `Degraded`, or `OutOfSync` and `Healthy`.

```sh
argocd app get my-app                                   # sync, health, conditions, last operation
argocd app diff my-app                                  # exact fields that differ; exit 1 when there is a diff
argocd app get my-app --hard-refresh                    # bypass the rendered-manifest cache
kubectl get application my-app -n argocd -o jsonpath='{.status.conditions}' | jq
kubectl get application my-app -n argocd -o jsonpath='{.status.operationState.message}{"\n"}'
kubectl logs -n argocd deploy/argocd-repo-server --tail 100
```

| Symptom | Cause | Check or fix |
| --- | --- | --- |
| `ComparisonError` condition | Rendering failed: bad path, missing values file, Helm or Kustomize error | Read the condition message; it quotes the tool's stderr |
| `Unknown` sync status, no resources | Repo server cannot fetch the repository: credentials, host key, network | `argocd repo list`, repo-server logs |
| `OutOfSync` again straight after a sync | A controller or webhook mutates the object after apply | `argocd app diff`, then `ignoreDifferences` ([Diffing and drift](#diffing-and-drift)) |
| Sync finished, app stays `Progressing` | Health check waiting for rollout, a LoadBalancer address, or a custom health script | `argocd app get my-app` shows which resource is `Progressing` |
| Sync stuck `Running` | A hook Job never completes, or a wave never becomes healthy | `argocd app get my-app`, then `argocd app terminate-op my-app` |
| `metadata.annotations: Too long` | Client-side apply stores the whole object in an annotation; large CRDs exceed 262144 bytes | Add the `ServerSideApply=true` sync option |
| Resource `SharedResourceWarning` | Two Applications manage the same object | Remove it from one source |
| Deleted objects come back | Deleted by hand and restored by self-heal or the next sync | Delete from Git and sync with prune |
| Nothing happens on push | No webhook; polling runs every 120s plus up to 60s jitter | Configure the Git webhook, or `--refresh` |
| `permission denied` on sync | AppProject forbids the repository, destination or kind | `argocd proj get <project>` |
| Rollback refused | Automated sync is enabled on the Application | Disable auto-sync first, or revert in Git |

> [!NOTE]
> The application controller is a StatefulSet in current manifests. If yours runs as a Deployment, use `deploy/argocd-application-controller`.

## Application spec

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io      # deleting the Application deletes its resources
spec:
  project: platform
  source:
    repoURL: https://github.com/example/infra.git
    targetRevision: v1.4.2                         # branch, tag or commit SHA
    path: clusters/prod/my-app
    helm:
      valueFiles: [values.yaml, values.prod.yaml]
      parameters:
        - { name: image.tag, value: "1.4.2" }
  destination:
    server: https://kubernetes.default.svc
    namespace: my-namespace
  syncPolicy:
    automated:
      prune: true                                  # delete objects removed from Git
      selfHeal: true                               # revert manual changes in the cluster
      allowEmpty: false                            # refuse to sync to zero resources
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
      - PruneLast=true                             # prune after all other resources are healthy
    retry:
      limit: 5
      backoff: { duration: 15s, factor: 2, maxDuration: 5m }
  revisionHistoryLimit: 10
```

Pin `targetRevision` to a tag or SHA for production. With `main`, the next merge deploys itself, which is either the point of GitOps or an incident, depending on the repository.

`selfHeal: true` reverts a manual `kubectl edit` within about 5 seconds of the controller noticing it. Check for it before debugging why a change keeps vanishing. Without the `resources-finalizer`, deleting the Application leaves its resources running and unmanaged.

Argo CD 3.0 changed defaults that affect upgrades from 2.x: resource tracking uses the `argocd.argoproj.io/tracking-id` annotation instead of a label, `status` fields are ignored in diffs for all resources, and RBAC `update` and `delete` on an Application no longer extend to its resources. See the [3.0 upgrade notes](https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.14-3.0/).

## Diffing and drift

Argo CD diffs rendered manifests against live objects after normalisation. Fields written by other controllers show as permanent drift unless excluded.

```yaml
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers: ["/spec/replicas"]             # an HPA owns this
    - group: ""
      kind: Secret
      name: db
      jqPathExpressions: ['.data["ca.crt"]']
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      managedFieldsManagers: [cert-manager-cainjector]   # ignore fields this manager owns
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true              # also leave those fields alone during sync
```

`ignoreDifferences` only changes the comparison by default. During sync the full desired manifest is still applied, so an HPA's replica count is reset on every sync. `RespectIgnoreDifferences=true` stops that for resources that already exist.

`ServerSideApply=true` hands field ownership to the API server, which is the cleaner fix when several controllers legitimately write to one object and when manifests are too large for client-side apply.

```sh
argocd app diff my-app --local ./clusters/prod/my-app   # compare a working copy against live
argocd app manifests my-app --source git                # what Argo CD rendered
argocd app manifests my-app --source live               # what exists now
```

## App of apps and ApplicationSets

An Application whose source directory contains more Application manifests bootstraps a whole cluster from one object. An ApplicationSet generates Applications from a template and a generator, which avoids writing one file per cluster or per team.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: tenants, namespace: argocd }
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/example/infra.git
        revision: main
        directories: [{ path: tenants/* }]
  template:
    metadata: { name: '{{.path.basename}}' }
    spec:
      project: tenants
      source: { repoURL: https://github.com/example/infra.git, targetRevision: main, path: '{{.path.path}}' }
      destination: { server: https://kubernetes.default.svc, namespace: '{{.path.basename}}' }
      syncPolicy: { automated: { prune: true, selfHeal: true } }
  syncPolicy:
    applicationsSync: create-update                  # never delete generated Applications
    preserveResourcesOnDeletion: true                # keep workloads if an Application is deleted
```

Generators include `git` (directories or files), `cluster`, `list`, `matrix`, `merge`, `scmProvider` and `pullRequest`. By default (`sync` policy) a directory removed from Git deletes its Application, and with the resources finalizer, the tenant's workloads. `applicationsSync: create-update` prevents that while a pattern is being trialled. The controller's `--policy` flag overrides this field unless policy override is enabled.

## Sync phases, waves and hooks

A sync runs in phases: `PreSync`, `Sync`, `PostSync`, with `SyncFail` on failure. Within a phase, resources are ordered by wave (lowest first, default 0), then by kind (namespaces and CRDs before other resources, custom resources last), then by name. Argo CD applies a wave, waits until it is healthy, pauses 2 seconds (`ARGOCD_SYNC_WAVE_DELAY`), then moves to the next. Pruning runs in reverse wave order.

```yaml
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"             # before wave 0
```

```yaml
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync               # PreSync, Sync, PostSync, SyncFail, Skip
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
```

A `PreSync` Job that never completes blocks the sync indefinitely. Give hook Jobs `backoffLimit` and `activeDeadlineSeconds`. Hooks do not run during a selective sync (`--resource`). A PreSync hook runs before the Application's own ConfigMaps and Secrets exist, so give those an earlier wave and make them hooks too if the Job needs them.

## Projects and access

An AppProject restricts which repositories, destination clusters and namespaces, and resource kinds its Applications may use. It is the boundary that stops a team's Application from creating a ClusterRoleBinding.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: tenants, namespace: argocd }
spec:
  sourceRepos: ["https://github.com/example/infra.git"]
  destinations:
    - { server: https://kubernetes.default.svc, namespace: "tenant-*" }
  clusterResourceWhitelist: []                      # no cluster-scoped objects
  namespaceResourceBlacklist:
    - { group: "", kind: ResourceQuota }
```

The `default` project allows everything. Do not put tenant Applications in it.

## Health checks and custom resources

Built-in health checks cover Deployments, StatefulSets, DaemonSets, ReplicaSets, Services, Ingresses, PVCs, Jobs, CronJobs, Pods and HPAs, plus a bundled library of Lua checks for common CRDs. A custom resource with no health check contributes no health status, so a broken CR can leave an Application `Healthy`. The Application's health is the worst health of its direct resources.

```yaml
# argocd-cm ConfigMap; key format resource.customizations.health.<group>_<kind>
resource.customizations.health.example.com_Database: |
  hs = {}
  if obj.status ~= nil and obj.status.phase == "Ready" then
    hs.status = "Healthy"
    hs.message = "ready"
  else
    hs.status = "Progressing"
    hs.message = "waiting for status.phase=Ready"
  end
  return hs
```

## ApplicationSet generators

Each generator produces a list of parameter sets; the template is rendered once per set. Combining generators with `matrix` (cartesian product) and `merge` (join on a key) covers the usual "every app on every cluster" and "per-cluster overrides" cases without a second tool.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: platform-addons, namespace: argocd }
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - clusters:                                   # one set per registered cluster with these labels
              selector:
                matchLabels: { env: prod }
              values:
                region: '{{index .metadata.labels "topology.kubernetes.io/region"}}'
          - list:
              elements:
                - { addon: cert-manager, chart: cert-manager, version: v1.18.2 }
                - { addon: external-dns, chart: external-dns, version: 1.19.0 }
  template:
    metadata:
      name: '{{.addon}}-{{.name}}'                      # .name is the cluster name from the clusters generator
      labels: { addon: '{{.addon}}', cluster: '{{.name}}' }
    spec:
      project: platform
      sources:
        - repoURL: https://charts.example.com
          chart: '{{.chart}}'
          targetRevision: '{{.version}}'
          helm:
            releaseName: '{{.addon}}'
            valueFiles: ['$values/addons/{{.addon}}/values.yaml', '$values/addons/{{.addon}}/{{.name}}.yaml']
            ignoreMissingValueFiles: true
        - repoURL: https://github.com/example/infra.git
          targetRevision: main
          ref: values
      destination: { server: '{{.server}}', namespace: '{{.addon}}' }
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true, ServerSideApply=true]
  templatePatch: |                                     # per-set YAML merged over the template (2.10+)
    {{- if eq .addon "cert-manager" }}
    spec:
      syncPolicy:
        syncOptions: [CreateNamespace=true, ServerSideApply=true, Replace=true]
    {{- end }}
```

| Generator | Parameters produced | Typical use |
| --- | --- | --- |
| `list` | Elements written inline | Small fixed sets, bootstrapping |
| `clusters` | Every cluster Secret matching a selector: `name`, `server`, `metadata.labels`, `metadata.annotations` | Fleet-wide add-ons |
| `git` directories | One set per matching directory: `path.path`, `path.basename`, `path.segments` | Tenant or app folders |
| `git` files | One set per matching file, with the file's YAML or JSON keys as parameters | `config.json` per environment |
| `matrix` | Cartesian product of two generators | Apps x clusters |
| `merge` | Join of generators on `mergeKeys`, later generators override earlier | Base list plus per-cluster overrides |
| `scmProvider`, `pullRequest` | Repositories or open PRs from GitHub, GitLab, Gitea, Bitbucket, Azure DevOps | Review environments |
| `plugin` | Output of an HTTP plugin | Inventory from an external system |

```sh
argocd appset list
argocd appset get platform-addons                      # generated Applications and conditions
argocd appset generate platform-addons.yaml            # render the Applications locally without creating anything (2.13+)
kubectl get applicationset platform-addons -n argocd -o jsonpath='{.status.conditions}' | jq
kubectl logs -n argocd deploy/argocd-applicationset-controller --tail 100 | grep -i error
```

The `clusters` generator only sees clusters registered as Secrets with the `argocd.argoproj.io/secret-type: cluster` label. The in-cluster destination has no such Secret by default, so it is skipped until one is created for `https://kubernetes.default.svc` with the labels the selector expects. Progressive syncs (`spec.strategy.type: RollingSync`) sync generated Applications in labelled steps and remain gated behind the `ARGOCD_APPLICATIONSET_CONTROLLER_ENABLE_PROGRESSIVE_SYNCS` environment variable.

## App of apps in practice

The bootstrap Application points at a directory of Application (or ApplicationSet) manifests. Deleting the root with the finalizer cascades through every child; leave the finalizer off the root and on the children so a mistake removes the pointers, not the workloads.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd                       # no resources-finalizer here
spec:
  project: default                        # the root creates Applications, which are cluster-scoped from Argo CD's point of view
  source:
    repoURL: https://github.com/example/infra.git
    targetRevision: main
    path: clusters/prod/apps              # contains one Application manifest per component
  destination: { server: https://kubernetes.default.svc, namespace: argocd }
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [ApplyOutOfSyncOnly=true]
```

```sh
kubectl apply -n argocd -f clusters/prod/apps/root.yaml   # the only manual apply a cluster should ever need
argocd app sync root && argocd app wait root --health --timeout 300
argocd app list -o name | xargs -n1 -P4 argocd app wait --sync --health --timeout 900   # then every child
```

Give children a `sync-wave` when order matters (CRD-providing operators at `-2`, cluster add-ons at `-1`, workloads at `0`), since the root syncs its children in wave order and waits for each Application's health. The root itself must be allowed to create `Application` resources in the `argocd` namespace, which the `default` project permits and a restricted project must list under `namespaceResourceWhitelist`. Anything in the root's directory that is not an Application is applied too, which is a convenient place for AppProjects.

## Resource-level sync options

Annotations on individual manifests override the Application's sync behaviour for that object.

```yaml
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false                 # never prune this object (PVCs, namespaces)
    argocd.argoproj.io/sync-options: Delete=false                # keep it when the Application is deleted
    argocd.argoproj.io/sync-options: Replace=true                # kubectl replace instead of apply (immutable fields, large objects)
    argocd.argoproj.io/sync-options: Force=true,Replace=true     # delete and recreate; causes downtime
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true   # CR whose CRD arrives in the same sync
    argocd.argoproj.io/sync-options: Validate=false              # skip kubectl schema validation
    argocd.argoproj.io/compare-options: IgnoreExtraneous         # exists in the cluster, not in Git, do not report OutOfSync
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation    # delete the previous hook run before creating the new one (default)
```

Several options go in one annotation separated by commas; repeating the key in YAML keeps only the last. `Replace=true` on a Deployment resets `status` and can trigger a rollout; `Force=true` deletes first and is for objects with immutable fields such as Jobs and some StatefulSet changes. `IgnoreExtraneous` is for objects other tools create inside a namespace Argo CD manages.

## RBAC

Access is decided by Casbin policies in the `argocd-rbac-cm` ConfigMap plus per-project roles in AppProjects. Users come from the local accounts in `argocd-cm` or from SSO, whose group claims map to roles through `g` lines. `policy.default` applies to every authenticated user that no policy matches; set it to `role:readonly` or `''` (nothing) rather than leaving `role:admin`.

```yaml
# argocd-rbac-cm
data:
  policy.default: role:readonly
  scopes: '[groups, email]'                         # which OIDC claims the g lines match against
  policy.csv: |
    p, role:deployer, applications, get,    platform/*, allow
    p, role:deployer, applications, sync,   platform/*, allow
    p, role:deployer, applications, action/apps/Deployment/restart, platform/*, allow
    p, role:deployer, logs, get, platform/*, allow
    p, role:deployer, exec, create, platform/*, deny
    p, role:tenant-a, applications, *, tenant-a/*, allow
    p, role:tenant-a, applications, delete/*/Pod/*/*, tenant-a/*, allow   # 3.0+: sub-resource permissions, here pod deletion only
    p, role:tenant-a, repositories, get, https://github.com/example/tenant-a.git, allow
    g, my-org:platform-team, role:deployer
    g, my-org:tenant-a-devs, role:tenant-a
    g, alice@example.com, role:admin
```

```sh
argocd account can-i sync applications 'platform/my-app'        # what the logged-in identity may do
argocd account can-i delete applications 'tenant-a/*' --as-user bob   # impersonate (needs admin)
argocd admin settings rbac validate --policy-file policy.csv     # syntax check before applying
argocd admin settings rbac can my-org:tenant-a-devs get applications 'tenant-a/web' --policy-file policy.csv --default-role role:readonly
argocd proj role create tenants ci-deployer                       # project-scoped role for automation
argocd proj role add-policy tenants ci-deployer -a sync -p allow -o '*'
argocd proj role create-token tenants ci-deployer -e 720h         # prints a JWT for that role; store it in the CI secret store
argocd proj role list-tokens tenants ci-deployer
```

Objects in `policy.csv` are `<project>/<application>`; a policy on `*/*` spans projects. Application names are cluster-wide, so a tenant with `create` on `tenant-a/*` can still collide with names elsewhere unless AppProjects also constrain `sourceNamespaces`. The `exec` resource gates `argocd app exec`/terminal in the UI and should stay denied for anything but break-glass roles. Project role tokens are the right credential for CI: they can only touch the one project and expire.

## CLI operations

```sh
argocd login argocd.example.com --sso --grpc-web          # SSO through the browser; --grpc-web when behind a proxy that lacks HTTP/2
argocd login argocd.example.com --username admin --password "$ARGOCD_PASSWORD"   # local account; prefer --sso or a project token
argocd login argocd.example.com --auth-token "$ARGOCD_TOKEN"   # CI with a project role token; ARGOCD_AUTH_TOKEN and ARGOCD_SERVER also work
argocd context                                             # switch between servers
argocd version --short

argocd app create my-app --repo https://github.com/example/infra.git --path clusters/prod/my-app --dest-server https://kubernetes.default.svc --dest-namespace my-namespace --project platform --sync-policy automated --self-heal --auto-prune
argocd app set my-app --revision v1.5.0                    # change targetRevision; use with disabled auto-sync for a controlled rollout
argocd app set my-app --helm-set image.tag=1.5.0 --parameter foo=bar
argocd app unset my-app --parameter image.tag
argocd app patch my-app --type merge --patch '{"spec":{"syncPolicy":null}}'   # disable automated sync
argocd app sync my-app --async                             # return immediately; use app wait afterwards
argocd app sync my-app --force --replace                   # replace instead of apply; downtime for some kinds
argocd app sync my-app --apply-out-of-sync-only --prune
argocd app sync -l team=payments --retry-limit 3           # by label, several apps
argocd app wait my-app --operation                         # only until the current operation finishes
argocd app resources my-app                                # every managed object with sync and health status
argocd app resources my-app --orphaned                     # objects in the namespace that nothing manages (project must enable orphanedResources)
argocd app logs my-app --kind Deployment --name my-app -f --tail 100
argocd app actions list my-app --kind Deployment
argocd app actions run my-app restart --kind Deployment --resource-name my-app   # rollout restart without kubectl access
argocd app delete-resource my-app --kind Pod --resource-name my-app-7c9f-abcde   # delete one live object (RBAC: delete/*/Pod/*/*)
argocd app history my-app && argocd app rollback my-app 12
argocd app delete my-app --cascade=false                   # remove the Application, keep the workloads

argocd cluster add my-context --name prod --label env=prod  # creates a ServiceAccount and Secret; the cluster must be in the kubeconfig
argocd cluster list
argocd repo add https://github.com/example/infra.git --username git --password "$GIT_TOKEN"
argocd repo add git@github.com:example/infra.git --ssh-private-key-path ~/.ssh/argocd_deploy
argocd repocreds add https://github.com/example/ --username git --password "$GIT_TOKEN"   # credential template for a whole org
argocd proj list && argocd proj get platform
argocd admin export > argocd-backup.yaml                   # Applications, AppProjects, repos, clusters, settings (contains secrets)
argocd admin import - < argocd-backup.yaml
argocd admin app get-reconcile-results --refresh --l team=payments reconcile.yaml   # reconcile matching apps offline; writes sync and health per app to the file (note the double-dash --l)
```

`argocd app set` writes the Application spec, so the change is visible in Git-managed clusters as drift on the Application object itself if the Application is managed by an app-of-apps; use it for break-glass and put the real change in Git. `argocd cluster add` needs cluster-admin on the target and installs an `argocd-manager` ServiceAccount with cluster-admin; for a least-privilege registration, create the ServiceAccount and Secret by hand with a narrower ClusterRole.

## Troubleshooting

| Symptom | Cause | Check or fix |
| --- | --- | --- |
| ApplicationSet creates no Applications | Generator matched nothing: cluster selector, git path glob, or `goTemplate` render error | `kubectl get applicationset X -o jsonpath='{.status.conditions}'`; controller logs |
| `ErrApplicationNotAllowedToUseProject` on generated apps | Template's `project` does not permit the destination or repository | `argocd proj get`; add the destination and repo |
| ApplicationSet deleted its Applications and their workloads | Directory removed or generator changed with `applicationsSync: sync` and finalizers on children | Set `applicationsSync: create-update` or `preserveResourcesOnDeletion: true` for tenant sets |
| `templatePatch` ignored | `goTemplate: false`, or patch YAML invalid | `goTemplate: true`; `argocd appset generate` to test |
| Hook Job fails every second sync with `AlreadyExists` | `hook-delete-policy` missing and `generateName` not used | `BeforeHookCreation` (default in recent versions) or `HookSucceeded`; or use `generateName` |
| `error validating data ... unknown field` on sync | CRD not installed yet, or schema changed | `SkipDryRunOnMissingResource=true` on the CR, CRDs in an earlier wave; or `Validate=false` |
| `the object has been modified; please apply your changes` | Another controller updating the object during apply | Retry; `ServerSideApply=true` for shared objects |
| `PermissionDenied` for a user who is in the right group | `scopes` does not include the claim, or the group name differs (case, prefix) | `argocd account get-user-info` shows the groups Argo CD sees |
| Project role token rejected | Token expired, role was recreated (new JWT issued-at), or its policy lacks the action | `argocd proj role get PROJECT ROLE` lists policies and token IDs; issue a new token |
| `argocd login` fails with `transport: authentication handshake failed` | Ingress terminates TLS and does not forward gRPC | `--grpc-web`, or configure the ingress for gRPC |
| Controller CPU high, syncs slow | Too many Applications per controller shard, or a huge cluster | `argocd_app_reconcile` histogram on the controller's metrics port (8082); increase controller replicas and set `ARGOCD_CONTROLLER_REPLICAS` for sharding |
| `argocd app diff` exit 1 in CI but the UI shows Synced | `diff` reports changes from local rendering or ignoreDifferences not applied to `--local` | Use `argocd app diff --server-side-generate`, and compare `--revision` instead of `--local` |
| Application stuck `Deleting` | Finalizer waiting on a resource that cannot be deleted (finalizers on a PVC or CR, missing CRD) | `kubectl get <kind> -n ns -o jsonpath='{.metadata.finalizers}'`; remove those first, then the Application's finalizer as a last resort |

## Oneliners

```sh
# Every Application that is not synced or not healthy
argocd app list -o json | jq -r '.[] | select(.status.sync.status!="Synced" or .status.health.status!="Healthy") | [.metadata.name, .status.sync.status, .status.health.status] | @tsv'

# The same without the CLI
kubectl get applications -n argocd -o custom-columns='NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status'

# Which commit each Application is running
kubectl get applications -n argocd -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.sync.revision}{"\n"}{end}'

# Sync every Application in a project, four at a time
argocd app list -p platform -o name | xargs -n1 -P4 argocd app sync

# Applications with automated sync off
kubectl get applications -n argocd -o json | jq -r '.items[] | select(.spec.syncPolicy.automated==null) | .metadata.name'

# Hard refresh every Application after a repository credential change
kubectl get applications -n argocd -o name | xargs -I{} kubectl annotate {} -n argocd argocd.argoproj.io/refresh=hard --overwrite

# Follow the controller for one Application
kubectl logs -n argocd statefulset/argocd-application-controller -f | grep -E 'my-app|error'

# Initial admin password (prints a credential; rotate it and delete the Secret after first login)
kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d

# Applications by project with sync and health, as a table
argocd app list -o json | jq -r '.[] | [.spec.project, .metadata.name, .status.sync.status, .status.health.status, .spec.source.targetRevision // .spec.sources[0].targetRevision] | @tsv' | column -t

# Applications whose last operation failed, with the message
kubectl get applications -A -o json | jq -r '.items[] | select(.status.operationState.phase == "Failed" or .status.operationState.phase == "Error") | "\(.metadata.name)\t\(.status.operationState.message | .[0:120])"'

# Applications that have not been reconciled in the last 10 minutes (controller stuck or sharded away)
kubectl get applications -n argocd -o json | jq -r --arg t "$(date -u -d '10 minutes ago' +%FT%TZ)" '.items[] | select(.status.reconciledAt < $t) | "\(.metadata.name) \(.status.reconciledAt)"'

# Which resources inside an Application are OutOfSync
kubectl get application my-app -n argocd -o json | jq -r '.status.resources[] | select(.status == "OutOfSync") | "\(.kind)/\(.name) \(.namespace // "-")"'

# Which resources are unhealthy, with the health message
kubectl get application my-app -n argocd -o json | jq -r '.status.resources[] | select(.health.status != null and .health.status != "Healthy") | "\(.kind)/\(.name): \(.health.status) \(.health.message // "")"'

# Applications pointing at a branch instead of a tag or SHA
kubectl get applications -A -o json | jq -r '.items[] | select((.spec.source.targetRevision // .spec.sources[0].targetRevision) | test("^(main|master|develop|HEAD)$")) | .metadata.name'

# Applications without selfHeal, or without prune
kubectl get applications -A -o json | jq -r '.items[] | select(.spec.syncPolicy.automated.selfHeal != true) | .metadata.name'

# Applications in the default project (should be none in a multi-tenant install)
kubectl get applications -A -o json | jq -r '.items[] | select(.spec.project == "default") | .metadata.name'

# Disable automated sync on every Application in a project before maintenance (re-enable from Git afterwards)
argocd app list -p platform -o name | xargs -n1 -I{} argocd app patch {} --type merge --patch '{"spec":{"syncPolicy":{"automated":null}}}'

# Refresh (soft) every Application, staggered so the repo server is not flooded
kubectl get applications -n argocd -o name | while read -r a; do kubectl annotate "$a" -n argocd argocd.argoproj.io/refresh=normal --overwrite; sleep 0.5; done

# Sync all OutOfSync Applications in a project and wait for them
argocd app list -p platform -o json | jq -r '.[] | select(.status.sync.status == "OutOfSync") | .metadata.name' | xargs -n1 -P4 -I{} sh -c 'argocd app sync {} --async && argocd app wait {} --sync --health --timeout 600'

# Diff what a Git revision would deploy against live, without touching the Application
argocd app diff my-app --revision feature-branch

# Render the manifests for a revision to a file for review
argocd app manifests my-app --revision v1.5.0 > /tmp/my-app-v1.5.0.yaml

# Applications and the clusters they target, counted per cluster
kubectl get applications -A -o json | jq -r '.items[].spec.destination | .name // .server' | sort | uniq -c | sort -rn

# Registered clusters with connection state
argocd cluster list -o json | jq -r '.[] | [.name, .server, .connectionState.status, (.info.serverVersion // "-")] | @tsv'

# Repositories whose connection failed
argocd repo list -o json | jq -r '.[] | select(.connectionState.status != "Successful") | "\(.repo) \(.connectionState.message)"'

# Who can sync what: dump the effective RBAC policy
kubectl get cm argocd-rbac-cm -n argocd -o jsonpath='{.data.policy\.csv}'

# Groups Argo CD sees for the logged-in user (SSO debugging)
argocd account get-user-info

# Roll a Deployment inside an Application without kubectl access to the cluster
argocd app actions run my-app restart --kind Deployment --resource-name my-app

# Terminate every stuck operation in a project
argocd app list -p platform -o json | jq -r '.[] | select(.status.operationState.phase == "Running") | .metadata.name' | xargs -n1 argocd app terminate-op

# Server-side rendered diff, matching what the controller computes
argocd app diff my-app --server-side-generate

# Recent sync events across all Applications
kubectl get events -n argocd --field-selector reason=OperationCompleted --sort-by=.lastTimestamp | tail -20

```

## Scripts

Fleet status report: every Application with sync, health, revision, age of the last reconciliation and the last operation's result, sorted with the broken ones first. Read-only; uses `kubectl` only, so it works without `argocd login`.

```sh
#!/usr/bin/env bash
# usage: argocd-report.sh [namespace]   default argocd
set -euo pipefail

ns=${1:-argocd}
now=$(date +%s)
kubectl get applications -n "$ns" -o json | jq -r --argjson now "$now" '
  .items[]
  | {name: .metadata.name,
     project: .spec.project,
     sync: (.status.sync.status // "Unknown"),
     health: (.status.health.status // "Unknown"),
     rev: ((.status.sync.revision // "-") | .[0:8]),
     age_m: (if .status.reconciledAt then (($now - (.status.reconciledAt | sub("\\.[0-9]+"; "") | fromdate)) / 60 | floor) else -1 end),
     op: (.status.operationState.phase // "-")}
  | [(if .sync != "Synced" or .health != "Healthy" or (.op | IN("Failed", "Error")) then 0 else 1 end), .project, .name, .sync, .health, .rev, (.age_m | tostring) + "m", .op]
  | @tsv' \
| sort -k1,1n -k2,2 -k3,3 | cut -f2- \
| { printf 'PROJECT\tNAME\tSYNC\tHEALTH\tREVISION\tRECONCILED\tLAST-OP\n'; cat; } | column -t -s $'\t'

bad=$(kubectl get applications -n "$ns" -o json | jq '[.items[] | select((.status.sync.status // "") != "Synced" or (.status.health.status // "") != "Healthy")] | length')
printf '\n%s Applications need attention\n' "$bad"
(( bad == 0 ))
```

Controlled release: pin an Application to a new Git revision with automated sync disabled, sync it, wait for health, and restore the previous revision on failure. Suited to a CI job that holds a project role token.

```sh
#!/usr/bin/env bash
# usage: argocd-release.sh APP REVISION   (ARGOCD_SERVER and ARGOCD_AUTH_TOKEN in the environment)
set -euo pipefail

app=${1:?application} rev=${2:?revision}
timeout=${TIMEOUT:-600}
prev=$(argocd app get "$app" -o json | jq -r '.spec.source.targetRevision')
auto=$(argocd app get "$app" -o json | jq -c '.spec.syncPolicy.automated // null')

restore() {
  printf 'restoring %s to %s\n' "$app" "$prev" >&2
  argocd app set "$app" --revision "$prev"
  argocd app sync "$app" --async >/dev/null || true
  if [[ $auto != null ]]; then argocd app patch "$app" --type merge --patch "{\"spec\":{\"syncPolicy\":{\"automated\":$auto}}}" >/dev/null; fi
}

[[ $auto != null ]] && argocd app patch "$app" --type merge --patch '{"spec":{"syncPolicy":{"automated":null}}}' >/dev/null
argocd app set "$app" --revision "$rev"
argocd app diff "$app" --revision "$rev" || true          # exit 1 when there is a diff; shown for the log
if argocd app sync "$app" --timeout "$timeout" --retry-limit 2 && argocd app wait "$app" --sync --health --timeout "$timeout"; then
  [[ $auto != null ]] && argocd app patch "$app" --type merge --patch "{\"spec\":{\"syncPolicy\":{\"automated\":$auto}}}" >/dev/null
  printf 'released %s at %s\n' "$app" "$rev"
  exit 0
fi
argocd app get "$app" | sed -n '/^GROUP/,$p' >&2
restore
exit 1
```

Orphan and drift audit: for every Application, list live resources that are OutOfSync and, where the project enables orphaned resource monitoring, objects in the namespace that no Application manages. Read-only.

```sh
#!/usr/bin/env bash
# usage: argocd-audit.sh [project]
set -euo pipefail

proj=${1:-}
args=(-o name); [[ -n $proj ]] && args+=(-p "$proj")
rc=0
for app in $(argocd app list "${args[@]}"); do
  app=${app#argocd/}
  oos=$(argocd app resources "$app" 2>/dev/null | awk 'NR > 1 && $5 == "OutOfSync" {print $2 "/" $4}' | paste -sd, -)
  orphans=$(argocd app resources "$app" --orphaned 2>/dev/null | awk 'NR > 1 {print $2 "/" $4}' | paste -sd, -)
  if [[ -n $oos || -n $orphans ]]; then
    printf '%s\n' "$app"
    [[ -n $oos ]] && printf '  out of sync: %s\n' "$oos"
    [[ -n $orphans ]] && printf '  orphaned:    %s\n' "$orphans"
    rc=1
  fi
done
exit "$rc"
```

> [!WARNING] Removing the resources finalizer
> `kubectl patch application my-app -n argocd --type json -p '[{"op":"remove","path":"/metadata/finalizers"}]'` lets a stuck Application delete without cascading. Its resources stay in the cluster, unmanaged. Use it only when the cascade itself is failing and you will clean up the resources yourself.

## Further reading

- [Argo CD documentation](https://argo-cd.readthedocs.io/en/stable/): user guide, operator manual and CLI reference.
- [Sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/) and [sync phases and waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/): every option, annotation and ordering rule.
- [ApplicationSet generators](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/): parameters each generator exposes, `matrix`, `merge` and progressive syncs.
- [RBAC configuration](https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/): resources, actions, sub-resource syntax and policy testing commands.
- [Resource health](https://argo-cd.readthedocs.io/en/stable/operator-manual/health/): built-in checks and writing Lua health scripts.
- [argocd CLI reference](https://argo-cd.readthedocs.io/en/stable/user-guide/commands/argocd/): every command and flag.