Bringing AI into Production: Our Journey with the Change Reliability Indicator
In this talk, I will introduce the Change Reliability Indicator, a predictive scoring method that leverages historical data to flag deployment changes most likely to cause high-priority incidents. By identifying incident-inducing changes early, we aim to enable IT teams to take proactive measures before issues arise. I will also share our journey of bringing the model from idea to moving it to production at ING. Building a model begins with understanding user needs and organizational requirements, such as compliance. Beyond model accuracy, we invested in requirements engineering for end-users, rigorous monitoring, and careful presentation of predictions to ensure the model’s usefulness and trustworthiness. Turning a proof-of-concept into a production system requires a multidisciplinary effort of data scientists, engineers, and managers. Along the way, we learned valuable lessons about what it takes to deploy AI at scale. Through these insights, example code (where possible), and practical experiences, I aim to show that reliable AI is not only about model performance but also about building reliable solutions.
