How do I handle "Feature Drift" in a production Machine Learning model?

All answers to this question.

Feature drift is the silent killer of production models. You definitely want to compare the distribution of your "inference" data against your "training" data. The Kolmogorov-Smirnov (K-S) test is excellent for continuous variables, while Population Stability Index (PSI) is a classic in the finance industry for categorical data. I recommend setting up a monitoring dashboard using tools like Prometheus or Evidently AI. If your PSI exceeds $0.2$, it’s a clear signal that the underlying population has shifted enough to require a model refresh or a deeper investigation into whether the world has changed (Concept Drift).

   Answered 2024-01-19 by Margaret Adams

Is it possible that the drift isn't in the data itself, but in the "target" variable? For example, are people defaulting on loans more often due to external economic factors?

   Answered 2024-01-25 by Steven Baker

• Steven, that would be "Concept Drift." If the relationship between the features (like income) and the target (default) has changed, then the model's logic is fundamentally broken. We are seeing exactly that—macroeconomic shifts are making our old weights obsolete. We’ve decided to implement a "Shadow Deployment" where we test a newly trained model on live data in the background. Once the shadow model consistently outperforms the legacy model on recent data, we'll swap them out. Thanks for pointing out that distinction.

     Commented 2024-01-30 by Elizabeth Hall

Automate your retraining! If you have a solid CI/CD pipeline, you can set it to automatically trigger a new training run whenever your drift metrics hit a certain threshold.

   Answered 2024-02-05 by Mary Nelson

• Automation is the dream, Mary. Just make sure you have human-in-the-loop validation before the new model actually goes live to avoid "catastrophic forgetting."

     Commented 2024-02-10 by Margaret Adams