Testing API¶

Tests and result types for comparing source and target score distributions.

Public functions¶

`samesame.shift.test_shift(source, target, *, n_resamples=9999, rng=None, weights=None)` ¶

Test whether the source and target score distributions differ.

The test uses ROC AUC to measure how well the score separates target from source, then permutes the group labels to form a two-sided null.

Parameters:

Name	Type	Description	Default
`source`	`ArrayLike`	Scores for the source (reference) group — e.g., training data or a past deployment.	required
`target`	`ArrayLike`	Scores for the target group — e.g., the current deployment or other population under evaluation.	required
`n_resamples`	`int`	Number of label permutations. Default `9999`. Use `999` while exploring and `19999` for finer resolution below `0.001`.	`9999`
`rng`	`int \| Generator \| RandomState \| None`	Random state for reproducibility. Pass `np.random.default_rng(12345)` or an `int` seed. Default `None`.	`None`
`weights`	`ImportanceWeights \| None`	Per-observation importance weights from :class:`samesame.weights.ImportanceWeights`. Omit to compare the full source and target samples; supply to focus on common support. Weights are normalized internally (each group's weights sum to its `n`; inactive groups stay at `1`) (see :func:`samesame.weights.domain_weights`).	`None`

Returns:

Type	Description
`ShiftResult`	Observed AUC, two-sided p-value, and null distribution. The null is formed by permuting group labels while keeping scores and weights fixed.

See Also

test_harm : Directional test when you can declare the harmful tail. samesame.weights.domain_weights : Build weights from P(target|x). samesame.weights.ImportanceWeights : Container for per-group weights.

Notes

The p-value doubles the smaller tail (capped at 1) and adds +1 smoothing so it is never exactly zero (Phipson & Smyth, 2010).
Interpret statistic relative to 0.5 (chance; 0.8 or 0.2 both signal strong separation) and pvalue as evidence against exchangeability — not as harm or business impact.
For honest p-values, scores from a fitted model must be out of sample. In-sample predictions can inflate separation because the scoring model has memorized its inputs.

References

Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be zero. Stat. Appl. Genet. Mol. Biol. 9(1):Article 39.

Examples:

>>> import numpy as np
>>> import samesame as ss
>>> rng = np.random.default_rng(12345)
>>> source = rng.normal(0, 1, size=300)
>>> target = rng.normal(0.6, 1, size=300)
>>> res = ss.test_shift(source, target, rng=rng)
>>> 0.5 < res.statistic <= 1.0
True
>>> res.pvalue < 0.01
True

Source code in src/samesame/shift.py

def test_shift(
    source: ArrayLike,
    target: ArrayLike,
    *,
    n_resamples: int = 9999,
    rng: Seed = None,
    weights: ImportanceWeights | None = None,
) -> ShiftResult:
    """
    Test whether the source and target score distributions differ.

    The test uses ROC AUC to measure how well the score separates target from
    source, then permutes the group labels to form a two-sided null.

    Parameters
    ----------
    source : ArrayLike
        Scores for the source (reference) group — e.g., training data or a
        past deployment.
    target : ArrayLike
        Scores for the target group — e.g., the current deployment or other
        population under evaluation.
    n_resamples : int, optional
        Number of label permutations. Default ``9999``. Use ``999`` while
        exploring and ``19999`` for finer resolution below ``0.001``.
    rng : int | np.random.Generator | np.random.RandomState | None, optional
        Random state for reproducibility. Pass ``np.random.default_rng(12345)``
        or an ``int`` seed. Default ``None``.
    weights : ImportanceWeights | None, optional
        Per-observation importance weights from
        :class:`samesame.weights.ImportanceWeights`. Omit to compare the full
        source and target samples; supply to focus on common support. Weights
        are normalized internally (each group's weights sum to its ``n``;
        inactive groups stay at ``1``) (see
        :func:`samesame.weights.domain_weights`).

    Returns
    -------
    ShiftResult
        Observed AUC, two-sided p-value, and null distribution. The null is
        formed by permuting group labels while keeping scores and weights
        fixed.

    See Also
    --------
    test_harm : Directional test when you can declare the harmful tail.
    samesame.weights.domain_weights : Build weights from ``P(target|x)``.
    samesame.weights.ImportanceWeights : Container for per-group weights.

    Notes
    -----
    * The p-value doubles the smaller tail (capped at ``1``) and adds ``+1``
      smoothing so it is never exactly zero (Phipson & Smyth, 2010).
    * Interpret ``statistic`` relative to ``0.5`` (chance; ``0.8`` or
      ``0.2`` both signal strong separation) and ``pvalue`` as evidence
      against exchangeability — not as harm or business impact.
    * For honest p-values, scores from a fitted model must be out of
      sample. In-sample predictions can inflate separation because the
      scoring model has memorized its inputs.

    References
    ----------
    Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be
        zero. *Stat. Appl. Genet. Mol. Biol.* 9(1):Article 39.

    Examples
    --------
    >>> import numpy as np
    >>> import samesame as ss
    >>> rng = np.random.default_rng(12345)
    >>> source = rng.normal(0, 1, size=300)
    >>> target = rng.normal(0.6, 1, size=300)
    >>> res = ss.test_shift(source, target, rng=rng)
    >>> 0.5 < res.statistic <= 1.0
    True
    >>> res.pvalue < 0.01
    True
    """
    statistic, pvalue, null_distribution = _permutation_test(
        source,
        target,
        metric=_auc_metric,
        alternative="two-sided",
        n_resamples=n_resamples,
        rng=rng,
        weights=weights,
    )
    return ShiftResult(
        statistic=statistic, pvalue=pvalue, null_distribution=null_distribution
    )

`samesame.shift.test_harm(source, target, *, worse, n_resamples=9999, rng=None, weights=None)` ¶

Test whether target moved toward the declared harmful tail.

A small test_shift p-value says something changed. This test asks the narrower question: after orienting the score so larger means worse (worse="lower" flips the sign internally), does target put more mass beyond thresholds the source rarely exceeds? Formally it is the weighted AUC ∫ TPR·(1−FPR)² dFPR of Kamulete (2022); thresholds the source rarely exceeds get more weight, so the test leans into the harmful tail. A small p-value is evidence for that directional movement — not for arbitrary shift.

Decide worse from what the score means before looking at results; do not pick the direction that gives the smaller p-value.

Parameters:

Name	Type	Description	Default
`source`	`ArrayLike`	Scores for the source (reference) group — e.g., training data or a past deployment.	required
`target`	`ArrayLike`	Scores for the target group — e.g., the current deployment or other population under evaluation.	required
`worse`	`('higher', 'lower')`	Which tail is harmful. `"higher"` when larger scores mean harm (e.g., predicted risk, prediction error, or outlier score); `"lower"` when smaller scores mean harm (e.g., confidence via `LogitGap`). Accepts a plain string or :class:`Worse`.	`'higher'`
`n_resamples`	`int`	Number of label permutations. Default `9999`. Use `999` while exploring and `19999` for finer resolution below `0.001`.	`9999`
`rng`	`int \| Generator \| RandomState \| None`	Random state for reproducibility. Pass `np.random.default_rng(12345)` or an `int` seed. Default `None`.	`None`
`weights`	`ImportanceWeights \| None`	Per-observation importance weights from :class:`samesame.weights.ImportanceWeights`. Omit to compare the full source and target samples; supply to focus on common support. Weights are normalized internally (each group's weights sum to its `n`; inactive groups stay at `1`) (see :func:`samesame.weights.domain_weights`).	`None`

Returns:

Type	Description
`HarmfulShiftResult`	Observed weighted AUC, one-sided p-value, declared `worse`, and null distribution. The null is formed by permuting group labels while keeping scores and weights fixed.

See Also

test_shift : Broad, two-sided screen when any change matters. samesame.weights.domain_weights : Build weights from P(target|x). Worse : The "higher" / "lower" choice in plain language.

Notes

One-sided greater alternative with +1 smoothing (never zero) (Phipson & Smyth, 2010).
Compare the statistic to null_distribution and the score's own scale, not to 0.5. See :doc:How the harm test works <../explanation/harmful-shift-statistic> for the ROC intuition and the ∫ TPR·(1−FPR)² dFPR form.

References

Kamulete, V. M. (2022). Test for non-negligible adverse shifts. Proceedings of the 38th UAI, PMLR 180:959-968. arXiv:2107.02990. Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be zero. Stat. Appl. Genet. Mol. Biol. 9(1):Article 39.

Examples:

>>> import numpy as np
>>> import samesame as ss
>>> rng = np.random.default_rng(12345)
>>> source = rng.normal(0.20, 0.07, size=300)
>>> target = rng.normal(0.28, 0.07, size=300)  # higher risk = worse
>>> res = ss.test_harm(source, target, worse="higher", rng=rng)
>>> res.pvalue < 0.05
True

Source code in src/samesame/shift.py

def test_harm(
    source: ArrayLike,
    target: ArrayLike,
    *,
    worse: Worse | str,
    n_resamples: int = 9999,
    rng: Seed = None,
    weights: ImportanceWeights | None = None,
) -> HarmfulShiftResult:
    """
    Test whether target moved toward the declared harmful tail.

    A small ``test_shift`` p-value says *something* changed. This test asks
    the narrower question: after orienting the score so larger means worse
    (``worse="lower"`` flips the sign internally), does target put more
    mass beyond thresholds the source rarely exceeds? Formally it is the
    weighted AUC ``∫ TPR·(1−FPR)² dFPR`` of Kamulete (2022); thresholds
    the source rarely exceeds get more weight, so the test leans into the
    harmful tail. A small p-value is evidence for that directional movement
    — not for arbitrary shift.

    Decide ``worse`` from what the score means before looking at results;
    do not pick the direction that gives the smaller p-value.

    Parameters
    ----------
    source : ArrayLike
        Scores for the source (reference) group — e.g., training data or a
        past deployment.
    target : ArrayLike
        Scores for the target group — e.g., the current deployment or other
        population under evaluation.
    worse : {'higher', 'lower'} or Worse
        Which tail is harmful. ``"higher"`` when larger scores mean harm
        (e.g., predicted risk, prediction error, or outlier score);
        ``"lower"`` when smaller scores mean harm (e.g.,
        confidence via ``LogitGap``). Accepts a plain string or
        :class:`Worse`.
    n_resamples : int, optional
        Number of label permutations. Default ``9999``. Use ``999`` while
        exploring and ``19999`` for finer resolution below ``0.001``.
    rng : int | np.random.Generator | np.random.RandomState | None, optional
        Random state for reproducibility. Pass ``np.random.default_rng(12345)``
        or an ``int`` seed. Default ``None``.
    weights : ImportanceWeights | None, optional
        Per-observation importance weights from
        :class:`samesame.weights.ImportanceWeights`. Omit to compare the full
        source and target samples; supply to focus on common support. Weights
        are normalized internally (each group's weights sum to its ``n``;
        inactive groups stay at ``1``) (see
        :func:`samesame.weights.domain_weights`).

    Returns
    -------
    HarmfulShiftResult
        Observed weighted AUC, one-sided p-value, declared ``worse``, and
        null distribution. The null is formed by permuting group labels
        while keeping scores and weights fixed.

    See Also
    --------
    test_shift : Broad, two-sided screen when any change matters.
    samesame.weights.domain_weights : Build weights from ``P(target|x)``.
    Worse : The ``"higher"`` / ``"lower"`` choice in plain language.

    Notes
    -----
    * One-sided ``greater`` alternative with ``+1`` smoothing (never zero)
      (Phipson & Smyth, 2010).
    * Compare the statistic to ``null_distribution`` and the score's own
      scale, not to ``0.5``. See :doc:`How the harm test works
      <../explanation/harmful-shift-statistic>` for the ROC intuition and
      the ``∫ TPR·(1−FPR)² dFPR`` form.

    References
    ----------
    Kamulete, V. M. (2022). Test for non-negligible adverse shifts.
        *Proceedings of the 38th UAI*, PMLR 180:959-968. arXiv:2107.02990.
    Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be
        zero. *Stat. Appl. Genet. Mol. Biol.* 9(1):Article 39.

    Examples
    --------
    >>> import numpy as np
    >>> import samesame as ss
    >>> rng = np.random.default_rng(12345)
    >>> source = rng.normal(0.20, 0.07, size=300)
    >>> target = rng.normal(0.28, 0.07, size=300)  # higher risk = worse
    >>> res = ss.test_harm(source, target, worse="higher", rng=rng)
    >>> res.pvalue < 0.05
    True
    """
    worse_enum = _coerce_worse(worse)

    metric = _harm_metric_factory(worse_enum)

    statistic, pvalue, null_distribution = _permutation_test(
        source,
        target,
        metric=metric,
        alternative="greater",
        n_resamples=n_resamples,
        rng=rng,
        weights=weights,
    )
    return HarmfulShiftResult(
        statistic=statistic,
        pvalue=pvalue,
        worse=worse_enum,
        null_distribution=null_distribution,
    )

Public result classes¶

`samesame.shift.ShiftResult` `dataclass` ¶

Result of :func:test_shift — a two-sided permutation result.

The statistic is ROC AUC ∫ TPR dFPR — how well the score separates target from source (0.5 is chance; values farther from 0.5 signal stronger separation, in either direction). The p-value is evidence against label exchangeability — not business impact, causality, an effect size, or the probability the null is true.

Parameters:

Name	Type	Description	Default
`statistic`	`float`	Observed ROC AUC.	required
`pvalue`	`float`	Two-sided permutation p-value.	required
`null_distribution`	`NDArray[float64]`	Permutation null distribution of the statistic.	required

Attributes:

Name	Type	Description
`statistic`	`float`	Observed ROC AUC, weighted when `weights` were supplied.
`pvalue`	`float`	Two-sided permutation p-value with +1 smoothing (always > 0; doubling the smaller tail, capped at 1) (Phipson & Smyth, 2010).
`null_distribution`	`NDArray[float64]`	Null distribution of the statistic (length `n_resamples`), produced by permuting group labels while keeping scores and weights fixed.

See Also

test_harm : When you can name the harmful tail in advance. samesame.weights.domain_weights : If poor overlap is a real concern.

References

Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be zero. Stat. Appl. Genet. Mol. Biol. 9(1):Article 39.

Source code in src/samesame/shift.py

@dataclass(frozen=True)
class ShiftResult:
    """
    Result of :func:`test_shift` — a two-sided permutation result.

    The statistic is ROC AUC ``∫ TPR dFPR`` — how well the score
    separates target from source (``0.5`` is chance; values farther from
    ``0.5`` signal stronger separation, in either direction). The p-value
    is evidence against label exchangeability — not business impact,
    causality, an effect size, or the probability the null is true.

    Parameters
    ----------
    statistic : float
        Observed ROC AUC.
    pvalue : float
        Two-sided permutation p-value.
    null_distribution : NDArray[np.float64]
        Permutation null distribution of the statistic.

    Attributes
    ----------
    statistic : float
        Observed ROC AUC, weighted when ``weights`` were supplied.
    pvalue : float
        Two-sided permutation p-value with +1 smoothing (always > 0;
        doubling the smaller tail, capped at 1) (Phipson & Smyth, 2010).
    null_distribution : NDArray[np.float64]
        Null distribution of the statistic (length ``n_resamples``),
        produced by permuting group labels while keeping scores and
        weights fixed.

    See Also
    --------
    test_harm : When you can name the harmful tail in advance.
    samesame.weights.domain_weights : If poor overlap is a real concern.

    References
    ----------
    Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be
        zero. *Stat. Appl. Genet. Mol. Biol.* 9(1):Article 39.
    """

    statistic: float
    pvalue: float
    null_distribution: NDArray[np.float64]

    def __repr__(self) -> str:
        return (
            f"{type(self).__name__}("
            f"statistic={_fmt(self.statistic)}, pvalue={_fmt(self.pvalue)})"
        )

`samesame.shift.HarmfulShiftResult` `dataclass` ¶

Bases: ShiftResult

Result of :func:test_harm — a one-sided tail result.

One-sided tail result. The statistic is the weighted AUC ∫ TPR·(1−FPR)² dFPR of Kamulete (2022) after orienting the score so larger means worse (worse="lower" flips the sign); it leans into thresholds the source rarely exceeds. Read it against null_distribution and the score's own scale. See :doc:How the harm test works <../explanation/harmful-shift-statistic> for the ROC intuition.

Parameters:

Name	Type	Description	Default
`statistic`	`float`	Observed harmful-shift statistic.	required
`pvalue`	`float`	One-sided permutation p-value.	required
`null_distribution`	`NDArray[float64]`	Permutation null distribution of the statistic.	required
`worse`	`Worse`	Declared harmful direction tested by the result.	required

Attributes:

Name	Type	Description
`statistic`	`float`	Observed harmful-shift statistic.
`pvalue`	`float`	One-sided (`greater`) permutation p-value with +1 smoothing (always > 0) (Phipson & Smyth, 2010).
`null_distribution`	`NDArray[float64]`	Null distribution of the statistic (length `n_resamples`), produced by permuting group labels while keeping scores and weights fixed.
`worse`	`Worse`	The declared harmful direction that was tested.

See Also

test_shift : Broad screen when any change matters. Worse : "higher" vs "lower" in plain language.

References

Kamulete, V. M. (2022). Test for non-negligible adverse shifts. Proceedings of the 38th UAI, PMLR 180:959-968. arXiv:2107.02990. Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be zero. Stat. Appl. Genet. Mol. Biol. 9(1):Article 39.

Source code in src/samesame/shift.py

@dataclass(frozen=True, repr=False)
class HarmfulShiftResult(ShiftResult):
    """
    Result of :func:`test_harm` — a one-sided tail result.

    One-sided tail result. The statistic is the weighted AUC
    ``∫ TPR·(1−FPR)² dFPR`` of Kamulete (2022) after orienting the score
    so larger means worse (``worse="lower"`` flips the sign); it leans
    into thresholds the source rarely exceeds. Read it against
    ``null_distribution`` and the score's own scale. See
    :doc:`How the harm test works <../explanation/harmful-shift-statistic>`
    for the ROC intuition.

    Parameters
    ----------
    statistic : float
        Observed harmful-shift statistic.
    pvalue : float
        One-sided permutation p-value.
    null_distribution : NDArray[np.float64]
        Permutation null distribution of the statistic.
    worse : Worse
        Declared harmful direction tested by the result.

    Attributes
    ----------
    statistic : float
        Observed harmful-shift statistic.
    pvalue : float
        One-sided (``greater``) permutation p-value with +1 smoothing
        (always > 0) (Phipson & Smyth, 2010).
    null_distribution : NDArray[np.float64]
        Null distribution of the statistic (length ``n_resamples``),
        produced by permuting group labels while keeping scores and
        weights fixed.
    worse : Worse
        The declared harmful direction that was tested.

    See Also
    --------
    test_shift : Broad screen when any change matters.
    Worse : ``"higher"`` vs ``"lower"`` in plain language.

    References
    ----------
    Kamulete, V. M. (2022). Test for non-negligible adverse shifts.
        *Proceedings of the 38th UAI*, PMLR 180:959-968. arXiv:2107.02990.
    Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be
        zero. *Stat. Appl. Genet. Mol. Biol.* 9(1):Article 39.
    """

    worse: Worse

    def __repr__(self) -> str:
        return (
            f"{type(self).__name__}("
            f"statistic={_fmt(self.statistic)}, pvalue={_fmt(self.pvalue)}, "
            f"worse={self.worse.value!r})"
        )

Public enum¶

`samesame.shift.Worse` ¶

Bases: StrEnum

Polarity that defines which tail is harmful for :func:test_harm.

Choose worse from the score's definition (e.g., risk is higher-is-worse, confidence via LogitGap is lower-is-worse) and pre-register it; do not pick the direction that gives the smaller p-value after seeing the data. A plain string "higher" / "lower" is accepted wherever this enum is; the two forms are interchangeable.

Attributes:

Name	Type	Description
`HIGHER`	`Worse`	Larger scores mean more harm (e.g., predicted risk, prediction error, or outlier score).
`LOWER`	`Worse`	Smaller scores mean more harm (e.g., confidence via `LogitGap`; lower is worse).

See Also

samesame.shift.test_harm : The test that consumes this choice.

Examples:

>>> from samesame import Worse
>>> Worse("higher") == Worse.HIGHER
True
>>> Worse("lower") == Worse.LOWER
True

Source code in src/samesame/shift.py

class Worse(StrEnum):
    """
    Polarity that defines which tail is harmful for :func:`test_harm`.

    Choose ``worse`` from the score's definition (e.g., risk is higher-is-worse,
    confidence via ``LogitGap`` is lower-is-worse) and pre-register it; do not
    pick the direction that gives the smaller p-value after seeing the data.
    A plain string ``"higher"`` / ``"lower"`` is accepted wherever this
    enum is; the two forms are interchangeable.

    Attributes
    ----------
    HIGHER : Worse
        Larger scores mean more harm (e.g., predicted risk, prediction
        error, or outlier score).
    LOWER : Worse
        Smaller scores mean more harm (e.g., confidence via ``LogitGap``;
        lower is worse).

    See Also
    --------
    samesame.shift.test_harm : The test that consumes this choice.

    Examples
    --------
    >>> from samesame import Worse
    >>> Worse("higher") == Worse.HIGHER
    True
    >>> Worse("lower") == Worse.LOWER
    True
    """

    HIGHER = "higher"
    LOWER = "lower"

Testing API¶

Public functions¶

samesame.shift.test_shift(source, target, *, n_resamples=9999, rng=None, weights=None) ¶

samesame.shift.test_harm(source, target, *, worse, n_resamples=9999, rng=None, weights=None) ¶

Public result classes¶

samesame.shift.ShiftResult dataclass ¶

samesame.shift.HarmfulShiftResult dataclass ¶

Public enum¶

samesame.shift.Worse ¶

`samesame.shift.test_shift(source, target, *, n_resamples=9999, rng=None, weights=None)` ¶

`samesame.shift.test_harm(source, target, *, worse, n_resamples=9999, rng=None, weights=None)` ¶

`samesame.shift.ShiftResult` `dataclass` ¶

`samesame.shift.HarmfulShiftResult` `dataclass` ¶

`samesame.shift.Worse` ¶