energy_fault_detector.data_preprocessing.duplicate_value_to_nan

class DuplicateValuesToNan(value_to_replace=0.0, n_max_duplicates=144, features_to_exclude=None, groupby_level='auto')[source]

Bases: DataTransformer

Replaces duplicate values with NaN.

In many data sets, zero can mean NaN, so we replace these duplicated values if they continue over n_max_duplicates times. The class can also be used for other values to replace.

Example:

value_to_replace = 0
n_max_duplicates = 2
Input: [0, 0, 0, 1, 2, 1, 3, 5, 1, 0, 0, 0, 0, 0, 7]
Output: [0, 0, np.nan, 1, 2, 1, 3, 5, 1, 0, 0, np.nan, np.nan, np.nan, 7]
Parameters:

  • value_to_replace (float) – The value to replace with NaN (default: 0.).

  • n_max_duplicates (int) – The maximum number of duplicates allowed before replacing with NaN (default: 144).

  • features_to_exclude (List[str]) – List of features to not transform. Defaults to None. Some sensors simply do not change for a while and that is ok.

feature_names_in_

list of column names in input.

feature_names_out_

list of columns in output.

Initialize the DuplicateValuesToNan transformer.

Parameters:

  • value_to_replace (float) – The value to replace with NaN (default: 0.).

  • n_max_duplicates (int) – The maximum number of duplicates allowed before replacing with NaN (default: 144).

fit(x, y=None)[source]

Set feature names in and out.

Parameters:

  • x (Union[array, DataFrame]) – The input data as a numpy array or pandas DataFrame.

  • y (Optional[array]) – The target data as a numpy array (optional).

Returns:

The fitted DuplicateValuesToNan transformer.

Return type:

DuplicateValuesToNan

get_feature_names_out(input_features=None)[source]

Returns the list of feature names in the output.

Return type:

List[str]

inverse_transform(x)[source]

Not implemented for data replacer (not useful)

Return type:

DataFrame

set_fit_request(*, x: bool | None | str = '$UNCHANGED$') → DuplicateValuesToNan

Request metadata passed to the fit method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in fit.

Returns

selfobject

The updated object.

set_inverse_transform_request(*, x: bool | None | str = '$UNCHANGED$') → DuplicateValuesToNan

Request metadata passed to the inverse_transform method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to inverse_transform if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to inverse_transform.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in inverse_transform.

Returns

selfobject

The updated object.

set_transform_request(*, x: bool | None | str = '$UNCHANGED$') → DuplicateValuesToNan

Request metadata passed to the transform method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to transform if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to transform.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in transform.

Returns

selfobject

The updated object.

transform(x)[source]

Replace any value that is duplicated more than self.n_max_duplicates with NaN.

Parameters:

x (DataFrame) – The input data as a pandas DataFrame.

Return type:

DataFrame

Returns:

The transformed data with duplicate values replaced with NaN.