Managing configuration drift and manual "hotfixes" in a GitOps-managed Kubernetes cluster.

All answers to this question.

This is the "drift" paradox. The whole point of GitOps is to prevent manual changes, but production emergencies don't always wait for a PR review. The best practice is to never disable the sync entirely. Instead, use the "Pause" feature in Argo CD or Flux. This stops the reconciliation temporarily while you fix the issue manually. However, you MUST have a post-incident process where that manual change is committed to Git and the "Pause" is removed. Some teams use a "Break-Glass" service account that triggers an alert the second someone uses 'kubectl' on a production namespace, ensuring the drift is documented immediately.

   Answered 2024-09-18 by Barbara Robinson

Have you tried using a "Fast-Track" PR process with a dedicated label that bypasses some of the longer CI checks for emergency fixes?

   Answered 2024-09-20 by Thomas Wright

• We haven't tried a fast-track PR yet. My worry is that "fast" usually means "untested." How do you ensure that an emergency PR doesn't accidentally introduce a syntax error that breaks the whole GitOps sync for the entire namespace? Is there a way to run a 'dry-run' sync before the PR is merged?

     Commented 2024-09-22 by Daniel White

We use 'Kustomize' overlays for different environments. If we need a hotfix, we change the 'prod' overlay and push. It takes 2 minutes, which is usually fast enough for us.

   Answered 2024-09-24 by Margaret Lewis

• 2 minutes is ideal! If the pipeline is that fast, there's almost no reason to ever touch 'kubectl' manually in the first place.

     Commented 2024-09-25 by Matthew Brown