Best Practices of Model Management
While companies around the globe compete to integrate data science, machine learning, and artificial intelligence into their core processes, a new competency is emerging: model management, or the practice of monitoring models in production through technologies and processes. Models provide the foundation for each of these advanced fields, and organizations are discovering the need to implement best practices to efficiently maintain models while simultaneously freeing up resources to develop new models.
A recent McKinsey study showed that organizations successfully leveraging models showed a 10% increase in profit margin compared to peers lagging behind the technology. Models accelerate breakthroughs and improve operations, making models invaluable assets to company stakeholders. Similar to other valuable assets, enterprises are discovering how to ensure their models’ continued success through model management.
As an emerging practice, it’s important to highlight some of the best practices and key capabilities of model management.
1. Track operational statistics
With the goal of producing a sharper competitive edge, understanding a model’s operational statistics provides insights about how to improve future models while also making adjustments to current models. Operational statistics may include the number of times called response time, outcomes, and requesting applications. Certain applications, such as recommendation engines or fraud alerts, may require immediate response times, while other circumstances may not be time-critical. A model may generate statistically reliable predictions, but if it underperforms when paired with specific applications, then it may not be a candidate for future projects. Keeping track of these factors allows teams to advance their products continuously and intervene when necessary.
2. Monitor business results
Companies can often provide data regarding how data science or AI-driven decisions improve their bottom line, but may not track the intricacies of the models generating those results. Model management requires monitoring revenue generated, customer growth, content engagement, or total fraud prevention, and being able to link it all back to the responsible model. In upgrading a model to improve accuracy, sales may unexpectedly drop due to minor tweaks in responses; model management allows the data science teams to navigate these waters without making drastic, costly decisions in an attempt at a clean slate. Models are also proprietary and represent value to investors and clients; being able to speak on behalf of a model’s capabilities and backing statements with data generates a new layer of value.
3. Score distributions
Model management involves taking a closer look at scores and how they relate to business decisions and results. Score distribution may provide insights into missed opportunities or blindspots. A model that scores mortgage applications, when compared to third-party data, may reveal an unnecessarily high emphasis on specific qualifications while overlooking others, ultimately losing potential business to competitors. As scores generate decisions and decisions create a strategy, it is critical to monitor each component and trace the results to the corresponding model. If features are added or removed, record the impact on the score distributions and subsequent results to become more empowered for future direction.
4. Explore unexpected variations
If any metrics—operational statistics, score distributions, or results—varies significantly from expectations or past performance, it may be necessary to alert the data science leaders. Model management allows companies to circumvent lost revenue or public humiliation caused by models gone awry. Eradicating the cause of such abnormalities could require reengineering features, refreshing the model, or updating integration points to include appropriate values. It’s always valuable to catch a bug before a customer or end-user does and improves stakeholders’ trust in model-first strategies.
5. Follow changes over time
Data evolves, systems and software change, user interfaces change, and business demands shift. Any countless number of factors could impact a model’s performance. Model management requires data scientists to monitor how this ever-changing environment affects the quality of predictions and reliability of the outcomes. Just like vehicles, a poorly-maintained model will eventually erode in some way until it’s obsolete, but through maintenance and upgrades, it could continue to perform its assigned task and achieve positive results.
Integrating model management
Companies like Amazon and Netflix built globally dominant brands around models that stretch beyond recommendation-engines into forecasting, merchandising, operational efficiency, translations, and more. Companies fight over the next data science unicorn while boardrooms formulate entire strategies around the capabilities of data science, including machine learning, AI, and IoT. The landscape looks entirely different than only ten years ago, and models—once a niche, hi-tech concept—are now front and center with the potential to provide more value than any other company asset.
Model management is not a secondary practice to be juggled in-between product launches and flashy presentations; it’s uniquely important for the success of a data science team in both the long and short term. Adequately providing time and resources is an important step to ensure product reliability and improve the quality of future projects.
To learn about automated and manageable pipelines, read our blog on key components of a data science factory.