energy_fault_detector.data_preprocessing.data_preprocessor

Generic class for building a preprocessing pipeline.

class DataPreprocessor(steps=None)[source]

Bases: Pipeline, SaveLoadMixin

A data preprocessing pipeline that allows for configurable steps based on the extended pipeline.

When no steps are provided, a default pipeline is created which removes features that are constant or binary and contain more 5% missing values. Afterward, remaining missing values are imputed with the mean and the features are scaled with the StandardScaler.

Parameters:

steps (Optional[List[Dict[str, Any]]]) –

Optional list of step specifications. Each item is a dict with:

  • name: registered step name (see STEP_REGISTRY).

  • enabled: optional bool (default True).

  • params: dict of constructor arguments for the step.

  • step_name: optional explicit pipeline name (defaults to name).

Notes

Enforced ordering in steps mode:

  1. NaN introducing steps first (DuplicateValuesToNan, CounterDiffTransformer),

  2. ColumnSelector (if present),

  3. Other steps

  4. SimpleImputer placed before scaler (always present; mean strategy by default),

  5. Scaler always last (StandardScaler by default).

  6. TimestampTransformer (if present).

Configuration example:

train:
  data_preprocessor:
    steps:
    - name: column_selector
      params:
        max_nan_frac_per_col: 0.05
        features_to_exclude: ['exclude_this_feature']
    - name: counter_diff_transformer
      step_name: counter_flow
      params:
        counters: ['flow_total_m3']
        compute_rate: True
        fill_first: 'zero'
    - name: counter_diff_transformer
      step_name: counter_energy
      params:
        counters: ['energy_total_kwh']
        compute_rate: False
        fill_first: 'zero'
        reset_strategy: 'rollover',
        rollover_values:
          'energy_total_kwh': 100000.0
NAME_ALIASES: Dict[str, str] = {'angle_transform': 'angle_transformer', 'counter_diff': 'counter_diff_transformer', 'counter_diff_transform': 'counter_diff_transformer', 'duplicate_value_to_nan': 'duplicate_to_nan', 'duplicate_values_to_nan': 'duplicate_to_nan', 'imputer': 'simple_imputer', 'minmax': 'minmax_scaler', 'standard': 'standard_scaler', 'standardize': 'standard_scaler', 'standardscaler': 'standard_scaler', 'timestamp_features': 'timestamp_transformer', 'timestamp_transform': 'timestamp_transformer'}
STEP_REGISTRY = {'angle_transformer': <class 'energy_fault_detector.data_preprocessing.angle_transformer.AngleTransformer'>, 'column_selector': <class 'energy_fault_detector.data_preprocessing.column_selector.ColumnSelector'>, 'counter_diff_transformer': <class 'energy_fault_detector.data_preprocessing.counter_diff_transformer.CounterDiffTransformer'>, 'duplicate_to_nan': <class 'energy_fault_detector.data_preprocessing.duplicate_value_to_nan.DuplicateValuesToNan'>, 'low_unique_value_filter': <class 'energy_fault_detector.data_preprocessing.low_unique_value_filter.LowUniqueValueFilter'>, 'minmax_scaler': <class 'sklearn.preprocessing._data.MinMaxScaler'>, 'simple_imputer': <class 'sklearn.impute._base.SimpleImputer'>, 'standard_scaler': <class 'sklearn.preprocessing._data.StandardScaler'>, 'timestamp_transformer': <class 'energy_fault_detector.data_preprocessing.timestamp_transformer.TimestampTransformer'>}
fit(X, y=None, **fit_params)[source]

Fit all transformers in the pipeline.

Parameters:

  • X (DataFrame) – Input DataFrame.

  • y – Target variable (optional).

  • **fit_params – Parameters to pass to the fit method of each step. Use step_name__parameter format (e.g., column_selector__protected_features=[‘feature1’]). Special parameter ‘protected_features’ can be used to validate that these features are present in the output after fitting.

Returns:

self

Raises:

ValueError – If protected_features are specified but not all are present in output.

fit_transform(x, **kwargs)[source]

Fit and transform in one step.

Parameters:

  • x (DataFrame) – Input DataFrame.

  • **kwargs (Any) – Parameters to pass to the fit method of each step. Use step_name__parameter format (e.g., column_selector__protected_features=[‘feature1’]).

Return type:

DataFrame

Returns:

Transformed DataFrame with the same index as input.

inverse_transform(x, **kwargs)[source]

Inverse-transform scaler and angles (other transforms are not reversed).

Parameters:

x (DataFrame) – The transformed data.

Return type:

DataFrame

Returns:

DataFrame with inverse scaling and angle back-transformation.

set_inverse_transform_request(*, x: bool | None | str = '$UNCHANGED$') → DataPreprocessor

Request metadata passed to the inverse_transform method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to inverse_transform if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to inverse_transform.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in inverse_transform.

Returns

selfobject

The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → DataPreprocessor

Request metadata passed to the score method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

selfobject

The updated object.

set_transform_request(*, x: bool | None | str = '$UNCHANGED$') → DataPreprocessor

Request metadata passed to the transform method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to transform if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to transform.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters

xstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for x parameter in transform.

Returns

selfobject

The updated object.

transform(x, **kwargs)[source]

Apply pipeline steps to the input DataFrame.

Parameters:

x (DataFrame) – Input DataFrame.

Return type:

DataFrame

Returns:

DataFrame with the same index as input.