What are the risks of using "GitOps" with ArgoCD for production deployments?

All answers to this question.

The biggest risk of GitOps is "Manual Override." If someone makes an emergency change directly via kubectl, ArgoCD will see the drift and immediately overwrite it with what is in Git. This can be dangerous during an active incident if the person fixing the issue forgets to update the repo. However, the benefit is that your Git repo is always a "Single Source of Truth." If your cluster is deleted, you can recreate the entire environment in minutes just by pointing ArgoCD at your repo. To mitigate Git downtime, ensure you have local cached copies of your manifests and use a highly available Git provider with redundant regions.

   Answered 2024-03-18 by Elizabeth Martinez

If ArgoCD automatically syncs every change, how do you handle "Secrets" like API keys? You obviously can't store them in plain text in a Git repository for GitOps to work.

   Answered 2024-04-05 by Richard Clark

• Richard, you should never put raw secrets in Git. The standard solution is to use "Sealed Secrets" (Bitnami) or an external secret operator that integrates with HashiCorp Vault or AWS Secrets Manager. You store a "Secret Reference" or an encrypted blob in Git, and a controller inside the Kubernetes cluster decrypts it or fetches the value at runtime. This allows you to maintain the GitOps flow for your configuration while keeping your sensitive credentials secure and compliant with industry standards like SOC2 or PCI-DSS.

     Commented 2024-04-12 by Paul Nelson

We found that GitOps actually reduced our "Config Drift" significantly because the system is self-healing. It’s much safer than the old "Push" based Jenkins deployments.

   Answered 2024-05-20 by Nancy White

• I agree with Nancy. The "Pull" model of ArgoCD is much more resilient. It’s better to have a system that constantly reconciles itself than one that only checks in during a deployment.

     Commented 2024-05-23 by Christopher Taylor