How to implement automated rollbacks in GitOps when a deployment fails health checks?

All answers to this question.

To achieve this, you need to look into Progressive Delivery tools like Argo Rollouts or Flagger. These act as "intelligent" controllers on top of your GitOps tool. Instead of a standard Deployment, you define a "Rollout" object. When you update the image tag in Git, the controller starts a Canary deployment. It monitors your metrics (via Prometheus or Datadog), and if it sees a latency spike, it automatically halts the rollout and reverts the traffic to the old version. The Git repo will still show the new version, so you’ll need a "Git-back" mechanism to sync the repo state with the cluster reality.

   Answered 2024-07-23 by Susan Martinez

Do you have enough traffic for Canary testing, or would a Blue-Green strategy be easier for your specific application architecture?

   Answered 2024-07-25 by David Johnson

• We have high traffic, but our app is stateful, which makes Blue-Green a bit difficult with database connections. We are leaning toward Canary. Does Flagger support Istio service mesh for the traffic shifting, and how hard is it to write the custom "Analysis" rules for the automated rollback logic?

     Commented 2024-07-27 by Richard Moore

You can also use Argo CD’s "Self-Heal" feature combined with a manual approval gate in your CI pipeline to prevent bad configs from reaching Git in the first place.

   Answered 2024-07-29 by Linda Davis

• Manual gates are safe, but they slow us down. Automated analysis with Flagger is definitely the goal for true Continuous Deployment.

     Commented 2024-07-30 by Ashley Thompson