Concept Drift vs Data Drift: Best Practices for Identifying and Addressing

Identifying concept drift requires monitoring changes in the underlying concept or relationship between variables, such as a shift in customer behavior or a change in market trends.

Data drift, on the other hand, refers to changes in the distribution of the data itself, such as a shift in the mean or variance of a variable.

Concept drift can occur due to changes in the underlying population, such as a change in demographics or an increase in data quality.

Monitoring the performance of machine learning models over time is key to detecting concept drift, as changes in model performance can indicate underlying shifts in the data.

Data drift, however, can occur due to changes in data collection methods or data quality, such as a change in sensor calibration or an increase in missing values.

Regularly reviewing and updating data pipelines and collection methods can help mitigate data drift.

Explore further: Concept Drift Detection

Concept drift is the most undesirable property of streaming data, making it very unpredictable. It leads to a degradation in the performance of mining techniques like classification or clustering, increasing the chances of misclassification. This is because the relationship between the target variable and input features changes over time.

For instance, an email spam classifier that predicts whether an email is spam or not based on the textual body of the email. The machine learning model learns a relationship between the target variable and the set of keywords that appears in a spam email.

These sets of keywords might not be constant, their pattern changes over time. This means that the model built on the old set of emails doesn’t work anymore on a new pattern of keywords.

Here are the three types of shifts that can lead to model decay:

This is why it becomes necessary to identify such drifts in the data to get efficient results with accuracy.

Concept drift refers to the changes in the underlying patterns or relationships within a dataset over time.

There are different types of concept drift patterns, one of which is gradual drift, where the change occurs slowly and steadily.

Explore further: Concept Redesigned Websites

Another type is sudden drift, where the change happens abruptly and without warning.

Concept drift can also be caused by a change in the underlying distribution of the data, such as a shift in the mean or variance.

This can be seen in production, where the patterns or relationships within a dataset change due to various factors, such as changes in user behavior or external events.

To detect concept drift, you need to set up ML model monitoring, which helps track how well your machine learning model is doing over time.

Machine learning model monitoring involves running reports on model quality, gathering metrics from the machine learning service, and displaying them on a live dashboard. You can also set up alerts to warn you if something goes wrong.

There are several metrics you can track to detect concept drift, including performance scores, classification confidence, and data drift.

Monitoring data drift introduces many challenges, including defining thresholds, as different quantification methods can create different scales of values.

The ideal concept drift handling system should be able to monitor the performance of the model over a long time, track relevant custom metrics, and detect data drift.

Here are some common methods for monitoring concept drift:

By monitoring concept drift, you can avoid the negative consequences of inaccurate predictions, such as lost revenue, reduced customer satisfaction, and increased operational costs.

Detecting concept drift is crucial to handling it effectively. Machine learning model monitoring helps track how well your machine learning model is doing over time.

You can track various metrics to detect concept drift, such as model performance, data quality, and alerts for unexpected changes. For example, if your model's performance drops below a certain level, you'll get a signal to retrain the model or dig deeper into the issue.

Evidently is an open-source Python library that helps implement testing and monitoring for production machine learning models. You can choose from 100+ pre-built checks and metrics to evaluate concept drift, including comparing model performance for past periods using reports like Regression Performance or Classification Performance.

Detecting concept drift is crucial to prevent it from affecting your machine learning model's performance. You can set up machine learning model monitoring to track how well your model is doing over time.

Machine learning model monitoring helps track how well your machine learning model is doing over time. You can use simple reports and one-off checks or real-time monitoring dashboards to do this.

To detect concept drift, you can track various metrics, including model performance, data distribution, and correlations between features. For example, you can use the Kolmogorov-Smirnov test to determine whether two samples come from the same distribution.

The Kolmogorov-Smirnov test is a non-parametric test that can be used to determine whether two samples come from the same distribution. This test can be used to detect data drift by comparing the distribution of the training data and the distribution of the test data.

Evidently is an open-source Python library that helps implement testing and monitoring for production machine learning models. You can use Evidently to evaluate concept drift in your models by comparing the model performance for past periods using reports like Regression Performance or Classification Performance.

A different take: Evidently Ai Data Drift

Some common metrics to track when detecting concept drift include:

These metrics can help you identify changes in your data distribution over time, which can indicate concept drift. By tracking these metrics regularly, you can detect concept drift early on and take corrective action to prevent it from affecting your model's performance.

Heuristics are a great way to detect concept drift. You can develop heuristics that reflect the model quality or correlate with it.

Tracking metrics like an average share of clicks on a recommendation block can be a useful heuristic. This can indicate if the model no longer shows relevant suggestions.

Domain experts can provide valuable input in developing meaningful heuristics. This is especially true when dealing with complex models and data.

The appearance of new categories in input features can be a sign of concept drift. For example, if a model forecasting future sales starts encountering entirely new product categories, it could be a warning sign.

It's essential to continuously monitor your model's performance and adjust your heuristics accordingly. This will help you catch concept drift before it's too late.

Concept drift can't be avoided for complex phenomena, so it's essential to have strategies in place to manage it. One approach is to periodically retrain the model to learn from historical data, as mentioned in Example 4. This involves detecting when a model needs retraining and then updating it with recent data.

Retraining a model can be costly, especially in a supervised setting, so it's often necessary to selectively create a sub-sample from the complete population and retrain on that. Feature dropping is another simple but effective method to deal with concept drift, as discussed in Example 5. This involves building multiple models where one feature is used at a time and monitoring the AUC-ROC response to determine if a feature is drifting.

Other remedies for concept drift include business rules and policies, human-in-the-loop decision-making, and pausing or stopping the model. For example, you can modify decision thresholds for classification models or use heuristics on top of model output, as seen in Example 2.

Readers also liked: Decision Tree Algorithm Machine Learning

Addressing concept drift requires a combination of strategies and techniques to manage its effects.

One approach is to detect concept drift through established model monitoring. This is crucial for knowing when to take action.

Sometimes, model retraining under concept drift is not possible due to a lack of new labeled data or other constraints. In these scenarios, business rules and policies can be modified to adjust model sensitivity. For example, decision thresholds can be changed to reduce false positives.

Human-in-the-loop decision-making can also be considered. This involves sending some or all of the data for manual decision-making when concept drift is detected.

Alternative models can be used to make decisions when the primary model is no longer reliable. This can include heuristics or other model types.

In some cases, it's best to simply pause or stop the model if its quality is unsatisfactory. Alternatively, you can choose to do nothing and accept a diminished performance.

Intriguing read: Pruning Decision Tree

Here are some possible remedies for preventing deterioration in prediction accuracy due to concept drift:

Periodic retraining is necessary for models that experience concept drift due to complex phenomena that are not governed by fixed laws of nature.

Feature dropping is a simple yet effective method to deal with concept drift. It involves building multiple models, each using a different feature while keeping the target variable unchanged.

The idea is to monitor the AUC-ROC response after prediction on test data. If the value of AUC-ROC for a particular feature goes beyond a certain threshold, that feature might be considered as drifting.

This method is widely used in the industry, and it's a good starting point for addressing concept drift. A common threshold for AUC-ROC is 0.8, but this can vary depending on the specific use case.

By dropping the drifting feature, you can prevent it from negatively impacting your model's performance. This can be a good solution, especially if the feature is not essential to the model's functionality.

Concept drift is a common problem in machine learning and data analysis, and it requires innovative solutions to address effectively. One approach is to develop dynamic systems that can adapt to changing data distributions.

Projects such as INFER, a computational intelligence platform developed by Bournemouth University, Evonik Industries, and the Research and Engineering Centre, aimed to create robust predictive systems from 2010 to 2014.

The INFER project was just one of many efforts to tackle concept drift. Another notable project was HaCDAIS, which focused on handling concept drift in adaptive information systems at Eindhoven University of Technology from 2008 to 2012.

Researchers at INESC Porto and the Laboratory of Artificial Intelligence and Decision Support in Portugal worked on KDUS, a project that explored knowledge discovery from ubiquitous streams.

These projects demonstrate the importance of addressing concept drift in various domains and the need for innovative solutions. ADEPT, a project from the University of Manchester and the University of Bristol, worked on adaptive dynamic ensemble prediction techniques.

Check this out: Elements of Statistical Learning Solutions

In 2022, the GAENARI project developed a C++ incremental decision tree algorithm that aims to minimize concept drifting damage.

Here are some key projects that have tackled concept drift:

Dealing with concept drift requires a structured approach. Data collection and preprocessing is the first step, which involves handling missing values, outliers, and label encoding for categorical variables.

To effectively detect concept drift, it's essential to analyze data points of adjacent windows. Accuracy metrics like accuracy, precision, recall, AUC-ROC response curve, and execution time are useful for this purpose.

Divide the data stream into a series of windows to facilitate data labeling. Assign a class label to individual data points based on the business context.

If concept drift is detected, it's crucial to follow an appropriate methodology to address it.

Data drift can be a major challenge in machine learning, but it's not the only one. One of the biggest challenges is ensuring that the model remains accurate over time.

Continuous monitoring is key to addressing data drift. By regularly checking the data, organizations can detect changes as they occur and address them quickly.

Data drift can lead to decreased model performance, which can have serious consequences. For example, in healthcare, a model that's no longer accurate can lead to misdiagnoses and incorrect treatment.

To mitigate the effects of data drift, organizations can use data cleansing techniques such as deduplication, standardization, and validation.

Here are some strategies that can help address data drift:

These strategies can help ensure that the machine learning model remains accurate and continues to perform well over time.

Measuring data drift isn't straightforward. You need to understand which distribution you want to test and check if it's drifting relative to the distribution you choose as your reference distribution.

To define the right drift metrics, you'll need to decide how to quantify the distance between these two distributions. This is a crucial step in detecting concept drift.

Defining the tested period and the reference distribution should be customized on a case-by-case basis. You should consider what constitutes the tested period and what distribution to use as a reference for comparison.

The most common use case for drift is to test for a training-serving skew, which can be referred to as a type of uncertainty measure. However, in many cases, the training dataset may not reflect the real-life distribution.

Model quality metrics, such as accuracy, precision, recall, or F1-score, are the most direct and reliable reflection of concept drift. A significant drop in these metrics over time can indicate the presence of concept drift.

The reference distribution is a crucial aspect of measuring concept drift, and it's essential to choose it wisely. In many cases, the training dataset may not reflect the real-life distribution, so comparing production distribution to the test dataset, which was untouched and represents the actual distribution, may be a better approach.

The most common use case for drift is to test for a training-serving skew, where the model in production is working under different circumstances than what was assumed when it was being trained. This can be referred to as a type of uncertainty measure.

For example, if you've rebalanced classes, run stratified sampling, or used other normalization methods, the training dataset may not accurately represent the real-life distribution. In these cases, comparing production distribution to the test dataset is a better approach.

In some cases, comparing the tested distribution to a sliding window reference distribution can be useful. This can help highlight whether the underlying data distribution has been changing over time.

Here are some common use cases for defining the reference distribution:

Ultimately, the choice of reference distribution depends on the specific use case and data.

Model quality metrics are the most direct and reliable reflection of concept drift. A significant drop in accuracy, precision, recall, or F1-score over time can indicate the presence of concept drift.

For classification problems, you can track metrics like accuracy, precision, recall, or F1-score. In scenarios like spam detection, you can gather user feedback, such as moving emails to spam or marking them as "not spam."

However, in some situations, you can't get the labels easily. For instance, if you predict future sales, you can only know the forecast error once that period is over.

You can utilize proxy metrics and heuristics as early warning signs of concept drift. They can tell you about a change in the environment or model behavior that might precede a drop in model quality.

Here's an interesting read: Confusion Matrix in Ai

Concept drift is a real challenge in data analysis, and it's essential to understand the methods used to investigate it.

For instance, researchers have identified various methods to investigate concept drift in big data streams, as mentioned in a reference paper.

The reference paper "Methods to Investigate Concept Drift in Big Data Streams" highlights the importance of understanding concept drift in real-world applications.

For your interest: Proximal Gradient Methods for Learning

Investigating concept drift requires a combination of statistical and machine learning techniques.

The paper suggests that statistical methods, such as hypothesis testing, can be used to detect concept drift.

Machine learning algorithms, such as ensemble methods, can also be employed to identify concept drift.

Researchers have found that concept drift can occur due to changes in the underlying distribution of the data.

These changes can be caused by various factors, including changes in the population, environment, or data collection process.

In reality, concept drift can have significant consequences on the accuracy and reliability of machine learning models.

The univariate approach to detecting data drift is a straightforward method that can be easily implemented. You can choose from a variety of statistical measures that can be applied to a single univariate distribution.

One of the advantages of the univariate approach is that it's simple to drill down to the specific feature that's drifting, making it easy to interpret the results. You can see the drift score of each feature and decide how it contributed to the overall drift score.

See what others are reading: Inception Score

Some of the benefits of the univariate approach include:

However, this approach can be impacted by redundancy, where correlated features can lead to multiple measurements of the same drift, making it less effective in high-dimensional datasets.

In the univariate approach, you can compute the drift on a per-feature basis and then use some aggregation function to get a single aggregated drift metric. This is a common practice, but it has its drawbacks.

Computing drift on a per-feature basis can lead to an overload of alerts due to the granularity of the measurement method, especially in scenarios of high dimensionality. Many machine learning models leverage dozens, hundreds, or even thousands of different features.

You can use the average drift level in all of the features to come up with a single score, but this might not accurately reflect the importance of each feature. Normalizing this according to the importance of each feature can help, but this information might not be available.

Another option is to use a multivariate method from the get-go, but this approach has its own set of pros and cons.

Recommended read: Bootstrap Method Machine Learning

The univariate approach has its advantages, and one of the biggest benefits is how simple it is to implement. You can choose from a variety of statistical measures that can be easily applied to a single univariate distribution.

One of the things I like about the univariate approach is how easy it is to drill down to the individual features that are causing a problem. This makes it easy to interpret the results and see exactly how each feature contributed to the overall drift score.

The univariate approach also allows you to easily adjust the weights of different features according to their importance. This means you can tailor the approach to the specific needs of your project.

Here are some of the key advantages of the univariate approach:

There are several software tools available for detecting concept drift in machine learning systems, including Frouros, an open-source Python library specifically designed for drift detection.

Some notable tools for concept drift detection include NannyML, which can detect univariate and multivariate distribution drift, and MOA, a free open-source software for mining data streams with concept drift.

If you're looking for a tool that can handle both data stream mining and concept drift, you might want to consider RapidMiner, formerly known as YALE, which is a free open-source software for knowledge discovery and data mining.

Here are some popular software tools for concept drift detection:

There are some amazing open-source software options available for detecting concept drift in machine learning systems. One such option is Frouros, a Python library designed specifically for drift detection.

Frouros is a powerful tool that can help you identify changes in your data streams. It's free and open-source, making it a great choice for anyone looking to get started with concept drift detection.

NannyML is another popular open-source library that's perfect for detecting univariate and multivariate distribution drift. It's also a Python library, making it easy to integrate with your existing workflows.

For those who prefer a more comprehensive platform, RapidMiner is a great option. It's a free, open-source software that offers a range of features, including data stream mining and tracking drifting concepts.

If you're looking for something a bit more specialized, EDDM (Early Drift Detection Method) is a free open-source implementation of drift detection methods in Weka. It's a great choice for those who want to dive deeper into the technical details of concept drift detection.

Here are some of the software options mentioned in this section:

Reviews of software and tools for data stream learning are plentiful, but it's essential to consider the challenges of benchmarking these algorithms with real-world data, as noted by Souza et al. in their 2020 study.

One of the most significant challenges is concept drift, which occurs when the underlying distribution of the data changes over time. This can be seen in the work of Gama et al., who surveyed the field of ensemble learning for data stream analysis and found that concept drift is a major issue.

Krawczyk et al. also highlighted the importance of ensemble learning in their 2017 survey, noting that it can help to adapt to changing data distributions.

To give you a better idea of the tools available, here are some notable reviews: