Functions

Pure DataFrame anonymization and pseudonymization functions.

All functions are pure: (DataFrame, **kwargs) -> DataFrame. No Spark actions are triggered — lazy evaluation is preserved throughout.

Methods encrypt and fake require optional extras:

pip install ‘Scamander-MDIF[anonymization]’

scamander.anonymization.functions.encrypt_column(df: DataFrame, column: str, encrypt_key: str) → DataFrame

Encrypt column values with AES-256-GCM (reversible pseudonymization).

Each value is encrypted with a fresh random 12-byte nonce and the result is base-64 encoded as <nonce><ciphertext+tag>. Decryption is possible with the same key using standard AES-256-GCM.

Requires the cryptography package:

pip install 'Scamander-MDIF[anonymization]'

Parameters:

  • df – Input DataFrame.

  • column – Column to encrypt. Values are cast to UTF-8 string.

  • encrypt_key – Base-64 encoded 32-byte AES-256 key. Generate one with: import os, base64; base64.b64encode(os.urandom(32)).decode()

Returns:

DataFrame with column replaced by base-64 encoded ciphertext.

Veroorzaakt:

ImportError – When cryptography is not installed.

Example:

import os, base64
key = base64.b64encode(os.urandom(32)).decode()
encrypt_column(df, "national_id", encrypt_key=key)
scamander.anonymization.functions.fake_column(df: DataFrame, column: str, faker_provider: str) → DataFrame

Replace column values with synthetic data from Faker.

Values are replaced regardless of their original content — nulls remain null. The replacement is not deterministic: each call produces different synthetic values.

Requires the faker package:

pip install 'Scamander-MDIF[anonymization]'

Parameters:

  • df – Input DataFrame.

  • column – Column to replace with synthetic data.

  • faker_provider – Faker provider method name. Examples: "name", "email", "phone_number", "address", "ssn", "company".

Returns:

DataFrame with column replaced by Faker-generated strings.

Veroorzaakt:

  • ImportError – When faker is not installed.

  • ValueError – When faker_provider is not a valid Faker method.

Example:

fake_column(df, "full_name", faker_provider="name")
scamander.anonymization.functions.generalize_column(df: DataFrame, column: str, bins: list[float], labels: list[str]) → DataFrame

Replace numeric values with categorical bucket labels (k-anonymity helper).

Buckets are half-open on the right [low, high) except for the last bucket which is fully closed [low, high]. Values outside all bins are set to null.

Parameters:

  • df – Input DataFrame.

  • column – Numeric column to generalize.

  • bins – Ordered bin boundaries (N+1 values define N buckets).

  • labels – Bucket labels (N strings, one per bucket).

Returns:

DataFrame with column replaced by the string bucket label.

Example:

generalize_column(df, "age",
    bins=[0.0, 18.0, 65.0, 120.0],
    labels=["child", "adult", "senior"])
scamander.anonymization.functions.generalize_date_column(df: DataFrame, column: str, date_unit: str) → DataFrame

Truncate a date or timestamp column to month or year granularity.

Uses Spark’s date_trunc and returns the first instant of the chosen period:

  • "month" — 2024-03-15 → 2024-03-01

  • "year" — 2024-03-15 → 2024-01-01

DateType columns remain DateType after truncation (date_trunc returns TimestampType internally; this function casts back so that downstream schema expectations are not broken).

Parameters:

  • df – Input DataFrame.

  • column – Date or timestamp column to generalise.

  • date_unit – Truncation granularity — "month" or "year".

Returns:

DataFrame with column truncated to the chosen granularity, preserving the original column type.

Example:

# "2024-03-15" -> "2024-03-01"
generalize_date_column(df, "birth_date", date_unit="month")
scamander.anonymization.functions.hash_column(df: DataFrame, column: str, hash_key: str | None = None) → DataFrame

Replace a column with its SHA-256 hash (one-way pseudonymization).

When hash_key is supplied, HMAC-SHA256 is used so that output can only be reproduced by parties that know the same secret key. Without a key, plain SHA-256 is applied via a native Spark function (no UDF overhead).

Parameters:

  • df – Input DataFrame.

  • column – Column to hash. Values are cast to string before hashing.

  • hash_key – Optional HMAC secret string. Omit for plain SHA-256.

Returns:

DataFrame with column replaced by the hex digest string.

Example:

# Plain SHA-256
hash_column(df, "email")

# HMAC-SHA256 with a shared secret
hash_column(df, "email", hash_key=platform.get_secret("hmac-key"))
scamander.anonymization.functions.mask_column(df: DataFrame, column: str, mask_char: str = '*', visible_prefix: int = 0, visible_suffix: int = 0) → DataFrame

Replace column characters with a masking character, preserving optional edges.

Parameters:

  • df – Input DataFrame.

  • column – Column to mask. Values are cast to string.

  • mask_char – Single character used as the mask. Default "*".

  • visible_prefix – Number of leading characters to keep unchanged.

  • visible_suffix – Number of trailing characters to keep unchanged.

Returns:

DataFrame with column masked.

Examples:

# Full mask: "john.doe@example.com" → "********************"
mask_column(df, "email")

# Keep last 4: "4111111111111234" → "************1234"
mask_column(df, "card_number", visible_suffix=4)

# Keep first 1 and last 4: "john.doe@example.com" → "j***************com"
mask_column(df, "email", visible_prefix=1, visible_suffix=4)
scamander.anonymization.functions.redact_column(df: DataFrame, column: str) → DataFrame

Replace all values in a column with null, preserving the original type.

Parameters:

  • df – Input DataFrame.

  • column – Column to redact.

Returns:

DataFrame with column set to null for every row.

Example:

redact_column(df, "birth_date")