energy_fault_detector.threshold_selectors

Threshold selection methods

class AdaptiveThresholdSelector(gamma=0.2, nn_size=10, nn_epochs=300, nn_learning_rate=0.001, nn_batch_size=128, smoothing_parameter=1, early_stopping=True, patience=3, validation_split=0.25, verbose=0, groupby_level='auto')[source]

Bases: ThresholdSelector

Adaptive threshold calculation based on NN regression and mutual information.

Parameters:

  • gamma (float) – Determines the sensitivity; added to the SVR model output (the expected anomaly score).

  • nn_size (int) – NN hyperparameter determining the size of the hidden layer of the NN.

  • nn_epochs (int) – NN hyperparameter for the number of epochs during training.

  • nn_learning_rate (float) – NN hyperparameter for the learning rate of the optimizer during training.

  • nn_batch_size (int) – Number of samples per gradient update.

  • smoothing_parameter (int) – Parameter for score smoothing; determines the length of segments in the smoothing function. A value of 1 practically disables smoothing (default is 1).

  • early_stopping (bool) – If True, the early stopping callback will be used in the fit method.

  • patience (int) – Parameter for early stopping. If early stopping is used, training will end if more than patience epochs in a row have not shown an improved loss. (default is 3)

  • verbose (int) – Determines the amount of console output during training: 0=silent, 1=progress bar, 2=one line per epoch.

Configuration example:

train:
  threshold_selector:
    name: AdaptiveThresholdSelector
    params:
      gamma: 0.2
      nn_size: 10
      nn_epochs: 100
      nn_learning_rate: 0.001
      nn_batch_size: 128
      smoothing_parameter: 3
      early_stopping: True
      patience: 3
      verbose: 0
fit(scaled_ae_input, anomaly_score, normal_index=None)[source]

Trains an NN model with the autoencoder input as input and the corresponding anomaly_score as targets.

Parameters:

  • scaled_ae_input (Union[ndarray, DataFrame]) – Standardized sensor data (autoencoder input).

  • anomaly_score (Union[ndarray, Series]) – Anomaly scores based on deviations of the autoencoder.

  • normal_index (Series) – Labels indicating whether each sample is normal (True) or anomalous (False). Optional; if not provided, assumes all data represents normal behavior.

Returns:

The instance of this class after fitting the model.

Return type:

AdaptiveThresholdSelector

predict(x, scaled_ae_input)[source]

Predicts the status (normal or anomalous) of each sample based on the trained NN model.

Parameters:

  • x (Union[ndarray, Series]) – Anomaly scores based on deviations of the autoencoder.

  • scaled_ae_input (Union[ndarray, DataFrame]) – Standardized sensor data (autoencoder input).

Returns:

A tuple containing a boolean array indicating the predicted status of each

sample and the corresponding adaptive thresholds.

Return type:

Tuple[ndarray, ndarray]

set_fit_request(*, anomaly_score: bool | None | str = '$UNCHANGED$', normal_index: bool | None | str = '$UNCHANGED$', scaled_ae_input: bool | None | str = '$UNCHANGED$') → AdaptiveThresholdSelector

Request metadata passed to the fit method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

anomaly_scorestr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for anomaly_score parameter in fit.

normal_indexstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for normal_index parameter in fit.

scaled_ae_inputstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for scaled_ae_input parameter in fit.

Returns

selfobject

The updated object.

set_predict_request(*, scaled_ae_input: bool | None | str = '$UNCHANGED$', x: bool | None | str = '$UNCHANGED$') → AdaptiveThresholdSelector

Request metadata passed to the predict method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to predict if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to predict.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

scaled_ae_inputstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for scaled_ae_input parameter in predict.

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in predict.

Returns

selfobject

The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → AdaptiveThresholdSelector

Request metadata passed to the score method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

selfobject

The updated object.

class FDRSelector(target_false_discovery_rate=0.2)[source]

Bases: ThresholdSelector

Find a threshold given a target false discovery rate (FDR).

Parameters:

target_false_discovery_rate (float) – The target FDR to fit the threshold to. Defaults to 0.2.

threshold

Scores above the threshold are classified as anomalies, while scores below are classified as normal.

Type:

float

actual_false_discovery_rate_

The actual FDR (the nearest threshold to the target) after fitting.

Type:

float

Example Configuration:

train:
  threshold_selector:
    name: FDRSelector
    params:
      target_false_discovery_rate: 0.2
fit(x, y=None)[source]

Finds a threshold given the specified false discovery rate.

Parameters:

  • x (Union[ndarray, Series]) – Array with calculated anomaly scores.

  • y (Series) – Labels indicating whether each sample is normal (True) or anomalous (False). Required for FDR threshold calculation.

Returns:

The instance of this class after fitting the threshold.

Return type:

FDRSelector

set_fit_request(*, x: bool | None | str = '$UNCHANGED$') → FDRSelector

Request metadata passed to the fit method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in fit.

Returns

selfobject

The updated object.

set_predict_request(*, x: bool | None | str = '$UNCHANGED$') → FDRSelector

Request metadata passed to the predict method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to predict if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to predict.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in predict.

Returns

selfobject

The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → FDRSelector

Request metadata passed to the score method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

selfobject

The updated object.

class FbetaSelector(beta=0.5, eps=1e-06, quantile=1.0)[source]

Bases: ThresholdSelector

Find a threshold via searching for the optimal fbeta score amongst all options.

Parameters:

  • beta (float) – beta weights the importance between precision and recall in the fbeta score. For example beta=2.0 means recall is twice as important as precision and beta=0.5 means the opposite. Defaults to 0.5.

  • eps (float) – small number, to ensure the optimal threshold is just below the anomaly score resulting in the optimal fbeta score.

  • quantile (float) – optional parameter that can introduce score smoothing. This parameter specifies a quantile which is used during self.fit to neglect all normal anomaly-scores that are greater than the quantile of all normal anomaly-scores. It must be a float between 0 and 1, where 1 practically disables the score smoothing.

threshold

scores above the threshold is classified as anomaly, below is classified as normal.

Type:

float

Configuration example:

train:
  threshold_selector:
    name: FbetaSelector
    params:
      beta: 0.5
      eps: 0.000001
      quantile: 1.
fit(x, y=None)[source]

Selects a threshold based on fbeta-scores.

Parameters:

  • x (Union[ndarray, Series]) – numpy array or pandas Series with calculated anomaly scores

  • y (Series) – series of labels indicating whether sample is normal (True) or anomalous (False) Required for F_beta Threshold!

Return type:

FbetaSelector

mark_normal_outliers(anomaly_score, normal_index)[source]

Marks all elements of anomaly_score that are normal according to normal_index which have an anomaly_score that is above the 99% quantile of all normal anomaly_scores

Parameters:

  • anomaly_score (ndarray) – array containing a time series of anomaly scores.

  • normal_index (ndarray) – boolean values indicating whether an element of anomaly_score is normal or not

Notes

If precision/recall cannot be calculated, the threshold is set to the maximum score.

Returns:

Boolean array which is true for all samples that are either normal and below quantile

or not normal.

Return type:

array

set_fit_request(*, x: bool | None | str = '$UNCHANGED$') → FbetaSelector

Request metadata passed to the fit method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in fit.

Returns

selfobject

The updated object.

set_predict_request(*, x: bool | None | str = '$UNCHANGED$') → FbetaSelector

Request metadata passed to the predict method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to predict if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to predict.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in predict.

Returns

selfobject

The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → FbetaSelector

Request metadata passed to the score method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

selfobject

The updated object.

class QuantileThresholdSelector(quantile=0.99)[source]

Bases: ThresholdSelector

Find a threshold by defining a specified quantile of the given anomaly scores.

Parameters:

quantile (float) – The quantile of the scores to be computed. Defaults to 0.99.

threshold

Scores above the threshold are classified as anomalies, while scores below are classified as normal.

Type:

float

Example Configuration:

train:
  threshold_selector:
    name: QuantileThresholdSelector
    params:
      quantile: 0.99
fit(x, y=None)[source]

Sets the threshold to the chosen quantile of the provided anomaly scores.

Parameters:

  • x (Union[ndarray, Series]) – Array containing calculated anomaly scores.

  • y (Series) – Labels indicating whether each sample is normal (True) or anomalous (False). Optional; if not provided, it is assumed that all data represents normal behavior.

Returns:

The instance of this class after setting the threshold.

Return type:

QuantileThresholdSelector

Notes

UserWarning: If a suitable threshold cannot be found, the threshold is set to the maximum score.

set_fit_request(*, x: bool | None | str = '$UNCHANGED$') → QuantileThresholdSelector

Request metadata passed to the fit method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in fit.

Returns

selfobject

The updated object.

set_predict_request(*, x: bool | None | str = '$UNCHANGED$') → QuantileThresholdSelector

Request metadata passed to the predict method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to predict if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to predict.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in predict.

Returns

selfobject

The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → QuantileThresholdSelector

Request metadata passed to the score method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

selfobject

The updated object.