How do we handle Gradient Vanishing in very deep Residual Networks for image segmentation?

All answers to this question.

Even with ResNet, you need to be very careful with your "He Initialization" and Batch Normalization placement. If you are using ReLU, the dying-neuron problem can masquerade as gradient vanishing. I highly recommend switching to an activation function like GELU or Swish, which provides a non-zero gradient for slightly negative values. Also, check your learning rate scheduler; sometimes a "Warm-up" period is required for very deep nets to prevent the early gradients from exploding or vanishing before the weights have found a reasonable starting manifold.

   Answered 2025-02-06 by Elizabeth Carter

Have you tried Layer Normalization instead of Batch Normalization? In some medical imaging tasks with small batch sizes, Batch Norm becomes unstable and can actually cause the gradients to oscillate or vanish prematurely.

   Answered 2025-02-08 by Richard Moore

• Richard, we are using a batch size of 2 because the 3D MRI scans are huge, so Batch Norm is definitely a suspect. We hadn't considered Layer Norm for a CNN-based ResNet, but it makes sense given our constraints. We'll also try Group Normalization, as it was specifically designed for small batch scenarios in computer vision. Hopefully, this stabilizes the internal covariate shift enough to get the gradients flowing again through those middle blocks.

     Commented 2025-02-11 by Jason Scott

Deep supervision can also help. Add auxiliary loss functions at intermediate layers to force the early parts of the network to learn useful features even if the final gradient is weak.

   Answered 2025-02-13 by Linda Thompson

• Auxiliary losses are a lifesaver for 100+ layer models. It provides a "shortcut" for the training signal to reach those distant early layers during the first few epochs.

     Commented 2025-02-16 by Elizabeth Carter