data_preprocessing.ffill_imputer

class ForwardFillImputer(ffill_limit='15min', categorical_features=None)[source]

Bases: DataTransformer

Impute missing values using time-based forward fill with a maximum gap duration.

Forward-fills NaN values from the last valid observation, but only when the elapsed time between the last valid value and the current row does not exceed ffill_limit. Rows where the gap is too large remain NaN and are subsequently dropped.

The transformer assumes time-series data with a temporal index (DatetimeIndex or TimedeltaIndex). During fit, it categorizes features as numerical or categorical, identifies and logs non-declared categorical features, and converts ffill_limit to a Timedelta. During transform, it forward-fills missing values (invalidating fills that exceed the time threshold), drops duplicate rows, and removes any rows still containing NaN values.

Dropping duplicate rows after forward-filling is necessary because forward-filling propagates the last valid value into consecutive NaN rows. If the underlying signal did not change between observations, the filled rows become exact duplicates of the source row. Removing these redundant rows avoids introducing identical training samples that carry no additional information and would otherwise bias the reconstruction error.

feature_names_in_

List of input feature names observed during fitting.

Type:

List[str]

feature_names_out_

List of output feature names (same as input features unless some were dropped).

Type:

List[str]

ffill_limit_

Fitted ffill_limit converted to a Timedelta.

Type:

pd.Timedelta

categorical_features

List of user-specified categorical feature names (matched by exact name).

Type:

List[str]

numerical_columns

List of identified numerical column names after filtering out non-declared categoricals.

Type:

List[str]

categorical_columns

List of identified categorical column names (from categorical_features).

Type:

List[str]

non_declared_categorical_features

List of columns detected as object-type (categorical) but not declared as such.

Type:

List[str]

Initializes the ForwardFillImputer with a forward-fill time limit and optional categorical features.

Parameters:

  • ffill_limit (Union[str, Timedelta]) – Maximum time gap to forward-fill, expressed as a pandas-compatible Timedelta string (e.g. "15min", "1h") or a Timedelta instance. NaNs whose elapsed time from the last valid observation exceeds this limit are left unfilled and the corresponding rows are dropped. Default is "15min".

  • categorical_features (Optional[List[str]]) – List of column names to treat as categorical features. Columns matching any of these (by exact name) are treated as categorical; others as numerical. Non-declared object-type columns in numerical slots are automatically detected and excluded from processing. Default is None (interpreted as empty list).

fit(x, y=None)[source]

Fits the imputer to the input data by identifying feature types and storing metadata.

Populates numerical_columns, categorical_columns, and non_declared_categorical_features based on the input DataFrame’s structure and user-provided categorical feature hints. Converts ffill_limit to a Timedelta and stores it as ffill_limit_.

Parameters:

  • x (DataFrame) – Input feature DataFrame. Must contain all features used during transform.

  • y (Optional[Series]) – Target variable. Ignored; included for compatibility with the scikit-learn API.

Returns:

The fitted transformer instance (self), for method chaining.

Return type:

ForwardFillImputer

Raises:

  • TypeError – If x is not a pandas DataFrame.

  • ValueError – If ffill_limit cannot be converted to a Timedelta.

get_feature_names_out(input_features=None)[source]

Return output feature names for downstream transformers.

Return type:

List[str]

inverse_transform(x)[source]

Returns the input DataFrame with columns reordered to match the original input feature order.

This method is included for compatibility with the DataTransformer/scikit-learn API. Since forward-fill imputation is not invertible (lost NaNs and duplicates cannot be recovered), only column reordering is performed—no actual value inversion occurs.

Parameters:

x (DataFrame) – Transformed DataFrame with columns matching feature_names_out_.

Returns:

DataFrame with columns reordered to match feature_names_in_ (original order).

Return type:

DataFrame

Raises:

  • TypeError – If x is not a pandas DataFrame.

  • ValueError – If the transformer has not been fitted (via check_is_fitted).

transform(x)[source]

Applies time-based forward-fill imputation and cleaning pipeline to input data.

The steps executed are:

  1. Forward-fill all NaN values.

  2. Invalidate fills where the elapsed time from the last valid observation exceeds ffill_limit_ (set those values back to NaN).

  3. Drop duplicate rows (exact duplicates created by forward-filling where the underlying signal did not change).

  4. Drop any rows still containing NaN values (dropna(how="any")).

Unlike the previous resampling-based approach, the original (possibly irregular) timestamps are preserved — no synthetic rows are introduced.

Parameters:

x (DataFrame) – Input feature DataFrame, with the same column names as used in fit().

Returns:

Cleaned and imputed DataFrame, containing only the columns from get_feature_names_out().

Return type:

DataFrame

Raises:

  • TypeError – If x is not a pandas DataFrame, or if its index is not temporal (DatetimeIndex/TimedeltaIndex).

  • ValueError – If x is missing columns seen during fit(), or if numerical columns cannot be converted to float.

  • ValueError – If the imputer has not been fitted (via check_is_fitted).