Product Quality Anomaly Detection | TIM User Story

Read how TIM enables users to quickly pinpoint pain points in their processes and to more effectively resolve them. TIM helps users to intelligently choose the batches to inspect and TIM’s guidance helps them to significantly reduce downtime and maintenance costs, to avoid costly waste and to improve product quality.

 

 

 

Use Case Description

 

1. Objective

The objective of this use case is to detect anomalies in quality measurements from a product produced in a roasting machine using temperature and humidity sensor data.

 

2. Goal

The goal of this use case is to improve product quality and ensure that the quality monitoring process is working properly.

 

3. Persona

The TIM user referenced in this use case description can be, but is not limited to, any of the following personas:

 

Nb	Persona	Description
1	Process Engineer	A process engineer can use the information outlined below to be guided towards relevant issues in such industrial processes.
2	Data Scientist	A data scientist can use this user story to easily experiment and build models with TIM that assist the process engineer and data analyst in their work.
3	Data Analyst	A data analyst can use the results from the analysis described below to create valuable insights and improvements to the processes.

 

4. Details

The qualities of the batches are measured by taking a sample every hour from each batch and analyze these thoroughly. Machine Learning will assist in the validation of these quality measurements and the process behind it. Anomaly detection by TIM will help the user in the following ways:

 

a) Normal Behavior

The user generates a dataset that can be divided in training and testing datasets. The training data should contain values for which the quality measurements are thoroughly analyzed and validated. TIM will then be able to accurately learn the relations between the sensor data and the quality measurements and describe that in a normal behavior model.

This normal behavior model will help the user understand the effect of different temperature and humidity values on the quality of the products. Furthermore, this normal behavior model can be used to detect anomalies using the following perspectives to validate the quality measurements.

 

b) Residual perspective

The residual perspective will detect global anomalies where the difference between the expected value of the KPI (the normal behavior) by TIM and the measured value of the KPI, is significant. Four situations can then arise:

If the measured value is significantly lower than the normal behavior, this could indicate that the quality measurement process has underestimated the quality of the batch. This underestimation could lead to the produced batch being unqualified for sale and therefore unnecessarily go to waste. The anomaly detected by TIM can notify the user of this and trigger a re-analysis of the quality of the batch. If the new quality is indeed better than initially measured we have a true positive detection (TP), if the quality is reconfirmed as low we have a false positive detection (FP).

If the measured value is significantly higher than the normal behavior, this could indicate that the quality measurement process has overestimated the quality of the batch. This overestimation could lead to the produced batch mistakenly being qualified for sale and which can lead to customer dissatisfaction. The anomaly detected by TIM can notify the user of this and trigger a re-analysis of the quality of the batch. If the new quality is indeed worse than initially measured we have a true negative detection (TN), if the quality is reconfirmed as high, we have a false negative detection (FN). The performance of the models by TIM are monitored with the TP, TN, FP & FN rates and the goal is to maximize the TP, TN an minimize the FP, FN.

In any scenario, the residual perspective helps validate the quality of individual batches.

 

c) Imbalance perspective

The imbalance perspective will detect anomalies when the there is an unexplained structural change in the quality measurements. The quality measurements can either be continuously significantly higher or lower than what the normal behavior model from TIM would expect based on the sensor data.

If the measured values are continuously better or worse than the normal behavior, this could indicate that the quality of the batches is continuously resp. overestimated or underestimated, e.g. by a change in the process, and the process then needs to be validated. It could also indicate a change in the machinery, e.g. a maintenance. In this case the normal behavior model needs to be retrained to have the new data pattern reflected in the normal behavior.

In either scenario, the imbalance perspective assists the user in validating the process.

 

d) Fluctuation perspective

The fluctuation perspective will detect anomalies when the fluctuations in the quality measurements change significantly.

If the measured samples are typically between a certain range of quality, but at some point the range increases and samples of both significantly higher & lower quality are measured, TIM will detect anomalies with the fluctuation perspective. This could indicate that the equipment is not running as smoothly as usual. The user can then do a maintenance check on the equipment and again continue to produce batches of the quality between a desired range.

In either scenario, the fluctuation perspective assists the user in monitoring the performance of the equipment.

Here is an overview of the scenarios described above:

 

Nb	Perspective	Scenario	Value
1	Residual	KPI <<< Normal behavior	Reduce waste
2	Residual	KPI >>> Normal behavior	Avoid bad quality product
3	Imbalance	Process validation	Assist in process maintenance
4	Imbalance	Model retraining	Assist in model maintenance
5	Fluctuation	Predictive maintenance	Monitor equipment health

 

* Note that the uses of the perspectives are not limited to what is described in the use case.

 

Dataset description

This dataset is processed and prepared for time series analysis with TIM. The original dataset contained minutely measurements for the predictor candidates and hourly measurements for the KPI value “Quality”. The predictor candidate values where then aggregated by mean and aligned with the KPI to generate the final dataset.

 

Nb	Column	Type	Description
1	date_time	Timestamp	Hourly sampled From: 04/01/2015 00:05 to: 03/05/2018 23:05
2	quality	KPI	Quality measurement from a sample taken from a batch of product produced in the past hour. Unit:  unkown
3	T_data_1_1	Predictor candidate	Temperature sensor 1 from machine 1
4	T_data_1_2	Predictor candidate	Temperature sensor 2 from machine 1
5	T_data_1_3	Predictor candidate	Temperature sensor 3 from machine 1
6	T_data_2_1	Predictor candidate	Temperature sensor 1 from machine 2
7	T_data_2_2	Predictor candidate	Temperature sensor 2 from machine 2
8	T_data_2_3	Predictor candidate	Temperature sensor 3 from machine 2
9	T_data_3_1	Predictor candidate	Temperature sensor 1 from machine 3
10	T_data_3_2	Predictor candidate	Temperature sensor 2 from machine 3
11	T_data_3_3	Predictor candidate	Temperature sensor 3 from machine 3
12	T_data_4_1	Predictor candidate	Temperature sensor 1 from machine 4
13	T_data_4_2	Predictor candidate	Temperature sensor 2 from machine 4
14	T_data_4_3	Predictor candidate	Temperature sensor 3 from machine 4
15	T_data_5_1	Predictor candidate	Temperature sensor 1 from machine 5
16	T_data_5_2	Predictor candidate	Temperature sensor 2 from machine 5
17	T_data_5_3	Predictor candidate	Temperature sensor 3 from machine 5
18	H_data	Predictor candidate	Humidity sensor
19	AH_data	Predictor candidate	Ambient humidity sensor

 

 

User Story

This section outlines the steps a TIM user would take to analyze the described dataset and experiment with TIM. The objective is to find predictive value in the data and the optimal configuration in TIM to extract that value.

 

1. Input Data Engineering

Before using TIM for anomaly detection, the TIM user prepares the dataset as outlined in the data properties section of the TIM documentation.

Link to the documentation: Input data properties – TIM Documentation (tangent.works)

Next, the dataset is uploaded to TIM for analysis.

 

 

2. Detection model build

A model is built on the training data using default settings with the perspectives “Residual”, ”Imbalance” & “Fluctuation” turned on. In case where the training data is thoroughly analyzed and validated, the sensitivity of each of the perspectives should be set to 0. In this dataset that is not the case and we will therefore us default TIM settings. With this TIM will automatically determine the sensitivity for us, therefore in sample anomalies will be detected by TIM.

 

 

 

3. Model Inspection

The insights from TIM can help us understand the way the model is built up by inspecting which features have been created by TIM. When the normal behavior closely matches the measured values, a useful model is built which accurately describes the patterns in the data. From the model zoo browser and we can see these patterns and learn which sensors are actually relevant in analyzing the quality of the produced batches.

The graph above shows a close alignment of the normal behavior and the measured values. This can also be validated by accuracy metrics provided by TIM. We can now assume we have a model of a high quality and inspect it further. In de image below we see a complex model where most temperature sensors and both humidity sensors have predictive value.

From here, a translation layer needs to be developed between the features generated by TIM and the use case specific knowledge to get relevant business insights from the data. The creation of this translation layer is always very use case specific is typically done together by the owner of the use case and a TIM expert. This is however out of scope for this use case description.

 

 

 

4. In Sample Anomaly inspection

The anomalies detected by TIM can now be more closely analyzed by the user according to the scenarios outlined above. In the example below we zoom in to a section of the data where anomalies are detected by all perspectives.

 

 

 

5. Scenario Analysis

The overview table of above is repeated here. We can use this to understand the anomalies that have been detected.

 

Nb	Perspective	Scenario	Value
1	Residual	KPI <<< Normal behavior	Reduce waste
2	Residual	KPI >>> Normal behavior	Avoid bad quality product
3	Imbalance	Process validation	Assist in process maintenance
4	Imbalance	Model retraining	Assist in model maintenance
5	Fluctuation	Predictive maintenance	Monitor equipment health

 

 

a) Residual

If we only select the Residual perspective in the anomaly indicator chart, we can visualize the anomalies detected by this perspective separately. Here, we clearly see that the measured values are significantly higher than the expected normal behavior. TIM expected these batches to be of significantly lower quality however the quality measurement indicates otherwise.

The user will now need to reinspect this batch and make sure they are not mis-qualifying a bad batch as one of higher quality. They can also dig deeper in why TIM labeled this batch as low quality using Root Cause Analysis (RCA), which we will show in the next section.

 

 

 

b) Imbalance

The imbalance perspective is close to the threshold line for a longer period of time and slightly crosses it once before moving down again. In this case, the normal behavior was consistently lower than the KPI for an extended period of time which could indicate that something has changed in the system structurally.

However, the imbalance indicator drops again, therefore we could conclude that it was not a structural change that sticked but rather a temporary shift in the data. This can serve as a warning to the user but no real action would be required other than continuing to monitor the imbalance perspective.

 

 

 

c) Fluctuation

The big drop in normal behavior also triggers the fluctuation perspective to detect anomalies. The big movements that are expected by TIM based on the sensor data are not reflected in the measured values. Depending on the severity of the fluctuations and the intensity of the anomalies, the user can choose to do a maintenance check and hopefully remove the cause of these fluctuations.

 

 

 

6. Root Cause Analysis (RCA)

To identify the cause of the drop in normal behavior which is causing these anomalies we apply RCA on the normal behavior line and inspect the results below.

You can immediately see that right before the anomalies start to occur a big delta is seen in one of the sensors. In this case the sensor called “T_data_3_3” indicated in bright green is the largest contributor to the drop in normal behavior.

 

 

If we visualize this predictor candidate in the graph, you can clearly see a spike in that sensor.

 

 

If the batch is indeed confirmed to be of lower quality than initially measured, we can use the information above to pinpoint the cause of this degradation in quality to machine number 3. The user can then use this information to specifically maintain this particular machine rather than having to check all the rest as well.

 

 

7. Business Value

Read how TIM enables users to quickly pinpoint pain points in their processes and to more effectively resolve them. TIM helps users to intelligently choose the batches to inspect and TIM’s guidance helps them to significantly reduce downtime and maintenance costs, to avoid costly waste and to improve product quality.