Testing API¶

Each test compares one score per observation across two samples: a reference (source) and a sample under review (target). test_shift uses ROC AUC to test rank-based separation: do target scores tend to be higher or lower than source scores? It can miss changes in distribution shape that do not change this ranking pattern. test_harm tests whether target moves toward the score tail declared harmful with worse before testing.

For guidance on when to use each test, see Benign vs harmful shift. For the weighted-AUC statistic, see How the harm test works.

Public functions¶

`samesame.shift.test_shift(source, target, *, n_resamples=9999, rng=None, weights=None)` ¶

Test for rank-based separation between source and target scores.

The two-sided test uses ROC AUC and permutes group labels. ROC AUC measures how often a randomly chosen target score exceeds a randomly chosen source score, with ties counting half. Values away from 0.5 indicate separation in either direction.

Parameters:

Name	Type	Description	Default
`source`	`ArrayLike`	Scores for the source (reference) group — e.g., training data or a past deployment.	required
`target`	`ArrayLike`	Scores for the target group — e.g., the current deployment or other population under evaluation.	required
`n_resamples`	`int`	Number of label permutations. Default `9999`. Use `999` while exploring and `19999` for finer resolution below `0.001`.	`9999`
`rng`	`int \| Generator \| RandomState \| None`	Random state for reproducibility. Pass `np.random.default_rng(12345)` or an `int` seed. Default `None`.	`None`
`weights`	`ImportanceWeights \| None`	Per-observation importance weights from :class:`samesame.weights.ImportanceWeights`. Omit to compare the full source and target samples; supply to focus on common support. Weights are normalized internally (each group's weights sum to its `n`; inactive groups stay at `1`) (see :func:`samesame.weights.domain_weights`).	`None`

Returns:

Type	Description
`ShiftResult`	Observed AUC, two-sided p-value, and null distribution. The null is formed by permuting group labels while keeping scores and weights fixed.

See Also

test_harm : Directional test when you can declare the harmful tail. samesame.weights.domain_weights : Build weights from P(target|x). samesame.weights.ImportanceWeights : Container for per-group weights.

Notes

The p-value doubles the smaller tail (capped at 1) and adds +1 smoothing so it is never exactly zero (Phipson & Smyth, 2010).
Interpret statistic relative to 0.5 (chance; 0.8 or 0.2 both signal strong separation) and pvalue as evidence against exchangeability — not as harm or business impact.
This rank-based test is not an omnibus test: distributions can differ in shape while ROC AUC remains 0.5.
For honest p-values, scores from a fitted model must be out of sample. In-sample predictions can inflate separation because the scoring model has memorized its inputs.

References

Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be zero. Stat. Appl. Genet. Mol. Biol. 9(1):Article 39.

Examples:

>>> import numpy as np
>>> import samesame as ss
>>> rng = np.random.default_rng(12345)
>>> source = rng.normal(0, 1, size=300)
>>> target = rng.normal(0.6, 1, size=300)
>>> res = ss.test_shift(source, target, rng=rng)
>>> 0.5 < res.statistic <= 1.0
True
>>> res.pvalue < 0.01
True

Source code in src/samesame/shift.py

def test_shift(
    source: ArrayLike,
    target: ArrayLike,
    *,
    n_resamples: int = 9999,
    rng: Seed = None,
    weights: ImportanceWeights | None = None,
) -> ShiftResult:
    """
    Test for rank-based separation between source and target scores.

    The two-sided test uses ROC AUC and permutes group labels. ROC AUC measures
    how often a randomly chosen target score exceeds a randomly chosen source
    score, with ties counting half. Values away from ``0.5`` indicate
    separation in either direction.

    Parameters
    ----------
    source : ArrayLike
        Scores for the source (reference) group — e.g., training data or a
        past deployment.
    target : ArrayLike
        Scores for the target group — e.g., the current deployment or other
        population under evaluation.
    n_resamples : int, optional
        Number of label permutations. Default ``9999``. Use ``999`` while
        exploring and ``19999`` for finer resolution below ``0.001``.
    rng : int | np.random.Generator | np.random.RandomState | None, optional
        Random state for reproducibility. Pass ``np.random.default_rng(12345)``
        or an ``int`` seed. Default ``None``.
    weights : ImportanceWeights | None, optional
        Per-observation importance weights from
        :class:`samesame.weights.ImportanceWeights`. Omit to compare the full
        source and target samples; supply to focus on common support. Weights
        are normalized internally (each group's weights sum to its ``n``;
        inactive groups stay at ``1``) (see
        :func:`samesame.weights.domain_weights`).

    Returns
    -------
    ShiftResult
        Observed AUC, two-sided p-value, and null distribution. The null is
        formed by permuting group labels while keeping scores and weights
        fixed.

    See Also
    --------
    test_harm : Directional test when you can declare the harmful tail.
    samesame.weights.domain_weights : Build weights from ``P(target|x)``.
    samesame.weights.ImportanceWeights : Container for per-group weights.

    Notes
    -----
    * The p-value doubles the smaller tail (capped at ``1``) and adds ``+1``
      smoothing so it is never exactly zero (Phipson & Smyth, 2010).
    * Interpret ``statistic`` relative to ``0.5`` (chance; ``0.8`` or
      ``0.2`` both signal strong separation) and ``pvalue`` as evidence
      against exchangeability — not as harm or business impact.
    * This rank-based test is not an omnibus test: distributions can differ
      in shape while ROC AUC remains ``0.5``.
    * For honest p-values, scores from a fitted model must be out of
      sample. In-sample predictions can inflate separation because the
      scoring model has memorized its inputs.

    References
    ----------
    Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be
        zero. *Stat. Appl. Genet. Mol. Biol.* 9(1):Article 39.

    Examples
    --------
    >>> import numpy as np
    >>> import samesame as ss
    >>> rng = np.random.default_rng(12345)
    >>> source = rng.normal(0, 1, size=300)
    >>> target = rng.normal(0.6, 1, size=300)
    >>> res = ss.test_shift(source, target, rng=rng)
    >>> 0.5 < res.statistic <= 1.0
    True
    >>> res.pvalue < 0.01
    True
    """
    statistic, pvalue, null_distribution = _permutation_test(
        source,
        target,
        metric=_auc_metric,
        alternative="two-sided",
        n_resamples=n_resamples,
        rng=rng,
        weights=weights,
    )
    return ShiftResult(
        statistic=statistic, pvalue=pvalue, null_distribution=null_distribution
    )

`samesame.shift.test_harm(source, target, *, worse, n_resamples=9999, rng=None, weights=None)` ¶

Test whether target moved toward the declared harmful tail.

A small test_shift p-value is evidence of rank-based separation. This test asks a narrower question: after orienting the score so larger means worse (worse="lower" flips the sign internally), does target put more mass beyond high thresholds the source rarely exceeds? Formally, it is Kamulete's weighted AUC ∫ TPR·(1−FPR)² dFPR, which gives more weight to those thresholds. A small p-value is evidence for that directional movement—not for arbitrary shift.

Decide worse from what the score means before looking at results; do not pick the direction that gives the smaller p-value.

Parameters:

Name	Type	Description	Default
`source`	`ArrayLike`	Scores for the source (reference) group — e.g., training data or a past deployment.	required
`target`	`ArrayLike`	Scores for the target group — e.g., the current deployment or other population under evaluation.	required
`worse`	`('higher', 'lower')`	Which tail is harmful. `"higher"` when larger scores mean harm (e.g., predicted risk, prediction error, or outlier score); `"lower"` when smaller scores mean harm (e.g., confidence via `LogitGap`). Accepts a plain string or :class:`Worse`.	`'higher'`
`n_resamples`	`int`	Number of label permutations. Default `9999`. Use `999` while exploring and `19999` for finer resolution below `0.001`.	`9999`
`rng`	`int \| Generator \| RandomState \| None`	Random state for reproducibility. Pass `np.random.default_rng(12345)` or an `int` seed. Default `None`.	`None`
`weights`	`ImportanceWeights \| None`	Per-observation importance weights from :class:`samesame.weights.ImportanceWeights`. Omit to compare the full source and target samples; supply to focus on common support. Weights are normalized internally (each group's weights sum to its `n`; inactive groups stay at `1`) (see :func:`samesame.weights.domain_weights`).	`None`

Returns:

Type	Description
`HarmfulShiftResult`	Observed weighted AUC, one-sided p-value, declared `worse`, and null distribution. The null is formed by permuting group labels while keeping scores and weights fixed.

See Also

test_shift : Two-sided screen for rank-based score separation. samesame.weights.domain_weights : Build weights from P(target|x). Worse : The "higher" / "lower" choice in plain language.

Notes

One-sided greater alternative with +1 smoothing (never zero) (Phipson & Smyth, 2010).
Compare the statistic to null_distribution and the score's own scale, not to 0.5. See :doc:How the harm test works <../explanation/harmful-shift-statistic> for the ROC intuition and the ∫ TPR·(1−FPR)² dFPR form.

References

Kamulete, V. M. (2022). Test for non-negligible adverse shifts. Proceedings of the 38th UAI, PMLR 180:959-968. arXiv:2107.02990. Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be zero. Stat. Appl. Genet. Mol. Biol. 9(1):Article 39.

Examples:

>>> import numpy as np
>>> import samesame as ss
>>> rng = np.random.default_rng(12345)
>>> source = rng.normal(0.20, 0.07, size=300)
>>> target = rng.normal(0.28, 0.07, size=300)  # higher risk = worse
>>> res = ss.test_harm(source, target, worse="higher", rng=rng)
>>> res.pvalue < 0.05
True

Source code in src/samesame/shift.py

def test_harm(
    source: ArrayLike,
    target: ArrayLike,
    *,
    worse: Worse | str,
    n_resamples: int = 9999,
    rng: Seed = None,
    weights: ImportanceWeights | None = None,
) -> HarmfulShiftResult:
    """
    Test whether target moved toward the declared harmful tail.

    A small ``test_shift`` p-value is evidence of rank-based separation.
    This test asks a narrower question: after orienting the score so larger
    means worse (``worse="lower"`` flips the sign internally), does target
    put more mass beyond high thresholds the source rarely exceeds?
    Formally, it is Kamulete's weighted AUC
    ``∫ TPR·(1−FPR)² dFPR``, which gives more weight to those thresholds.
    A small p-value is evidence for that directional movement—not for
    arbitrary shift.

    Decide ``worse`` from what the score means before looking at results;
    do not pick the direction that gives the smaller p-value.

    Parameters
    ----------
    source : ArrayLike
        Scores for the source (reference) group — e.g., training data or a
        past deployment.
    target : ArrayLike
        Scores for the target group — e.g., the current deployment or other
        population under evaluation.
    worse : {'higher', 'lower'} or Worse
        Which tail is harmful. ``"higher"`` when larger scores mean harm
        (e.g., predicted risk, prediction error, or outlier score);
        ``"lower"`` when smaller scores mean harm (e.g.,
        confidence via ``LogitGap``). Accepts a plain string or
        :class:`Worse`.
    n_resamples : int, optional
        Number of label permutations. Default ``9999``. Use ``999`` while
        exploring and ``19999`` for finer resolution below ``0.001``.
    rng : int | np.random.Generator | np.random.RandomState | None, optional
        Random state for reproducibility. Pass ``np.random.default_rng(12345)``
        or an ``int`` seed. Default ``None``.
    weights : ImportanceWeights | None, optional
        Per-observation importance weights from
        :class:`samesame.weights.ImportanceWeights`. Omit to compare the full
        source and target samples; supply to focus on common support. Weights
        are normalized internally (each group's weights sum to its ``n``;
        inactive groups stay at ``1``) (see
        :func:`samesame.weights.domain_weights`).

    Returns
    -------
    HarmfulShiftResult
        Observed weighted AUC, one-sided p-value, declared ``worse``, and
        null distribution. The null is formed by permuting group labels
        while keeping scores and weights fixed.

    See Also
    --------
    test_shift : Two-sided screen for rank-based score separation.
    samesame.weights.domain_weights : Build weights from ``P(target|x)``.
    Worse : The ``"higher"`` / ``"lower"`` choice in plain language.

    Notes
    -----
    * One-sided ``greater`` alternative with ``+1`` smoothing (never zero)
      (Phipson & Smyth, 2010).
    * Compare the statistic to ``null_distribution`` and the score's own
      scale, not to ``0.5``. See :doc:`How the harm test works
      <../explanation/harmful-shift-statistic>` for the ROC intuition and
      the ``∫ TPR·(1−FPR)² dFPR`` form.

    References
    ----------
    Kamulete, V. M. (2022). Test for non-negligible adverse shifts.
        *Proceedings of the 38th UAI*, PMLR 180:959-968. arXiv:2107.02990.
    Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be
        zero. *Stat. Appl. Genet. Mol. Biol.* 9(1):Article 39.

    Examples
    --------
    >>> import numpy as np
    >>> import samesame as ss
    >>> rng = np.random.default_rng(12345)
    >>> source = rng.normal(0.20, 0.07, size=300)
    >>> target = rng.normal(0.28, 0.07, size=300)  # higher risk = worse
    >>> res = ss.test_harm(source, target, worse="higher", rng=rng)
    >>> res.pvalue < 0.05
    True
    """
    worse_enum = _coerce_worse(worse)

    metric = _harm_metric_factory(worse_enum)

    statistic, pvalue, null_distribution = _permutation_test(
        source,
        target,
        metric=metric,
        alternative="greater",
        n_resamples=n_resamples,
        rng=rng,
        weights=weights,
    )
    return HarmfulShiftResult(
        statistic=statistic,
        pvalue=pvalue,
        worse=worse_enum,
        null_distribution=null_distribution,
    )

Public result classes¶

`samesame.shift.ShiftResult` `dataclass` ¶

Result of :func:test_shift — a two-sided permutation result.

The statistic is ROC AUC ∫ TPR dFPR — how well the score separates target from source (0.5 is chance; values farther from 0.5 signal stronger separation, in either direction). The p-value is evidence against label exchangeability — not business impact, causality, an effect size, or the probability the null is true.

Parameters:

Name	Type	Description	Default
`statistic`	`float`	Observed ROC AUC.	required
`pvalue`	`float`	Two-sided permutation p-value.	required
`null_distribution`	`NDArray[float64]`	Permutation null distribution of the statistic.	required

Attributes:

Name	Type	Description
`statistic`	`float`	Observed ROC AUC, weighted when `weights` were supplied.
`pvalue`	`float`	Two-sided permutation p-value with +1 smoothing (always > 0; doubling the smaller tail, capped at 1) (Phipson & Smyth, 2010).
`null_distribution`	`NDArray[float64]`	Null distribution of the statistic (length `n_resamples`), produced by permuting group labels while keeping scores and weights fixed.

See Also

test_harm : When you can name the harmful tail in advance. samesame.weights.domain_weights : If poor overlap is a real concern.

References

Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be zero. Stat. Appl. Genet. Mol. Biol. 9(1):Article 39.

Source code in src/samesame/shift.py

@dataclass(frozen=True)
class ShiftResult:
    """
    Result of :func:`test_shift` — a two-sided permutation result.

    The statistic is ROC AUC ``∫ TPR dFPR`` — how well the score
    separates target from source (``0.5`` is chance; values farther from
    ``0.5`` signal stronger separation, in either direction). The p-value
    is evidence against label exchangeability — not business impact,
    causality, an effect size, or the probability the null is true.

    Parameters
    ----------
    statistic : float
        Observed ROC AUC.
    pvalue : float
        Two-sided permutation p-value.
    null_distribution : NDArray[np.float64]
        Permutation null distribution of the statistic.

    Attributes
    ----------
    statistic : float
        Observed ROC AUC, weighted when ``weights`` were supplied.
    pvalue : float
        Two-sided permutation p-value with +1 smoothing (always > 0;
        doubling the smaller tail, capped at 1) (Phipson & Smyth, 2010).
    null_distribution : NDArray[np.float64]
        Null distribution of the statistic (length ``n_resamples``),
        produced by permuting group labels while keeping scores and
        weights fixed.

    See Also
    --------
    test_harm : When you can name the harmful tail in advance.
    samesame.weights.domain_weights : If poor overlap is a real concern.

    References
    ----------
    Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be
        zero. *Stat. Appl. Genet. Mol. Biol.* 9(1):Article 39.
    """

    statistic: float
    pvalue: float
    null_distribution: NDArray[np.float64]

    def __repr__(self) -> str:
        return (
            f"{type(self).__name__}("
            f"statistic={_fmt(self.statistic)}, pvalue={_fmt(self.pvalue)})"
        )

`samesame.shift.HarmfulShiftResult` `dataclass` ¶

Bases: ShiftResult

Result of :func:test_harm — a one-sided permutation result.

The statistic is Kamulete's weighted AUC ∫ TPR·(1−FPR)² dFPR after orienting the score so larger means worse (worse="lower" flips the sign). It gives more weight to thresholds the source rarely exceeds. Read it against null_distribution and the score's own scale, not against 0.5. See :doc:How the harm test works <../explanation/harmful-shift-statistic> for the ROC intuition.

Parameters:

Name	Type	Description	Default
`statistic`	`float`	Observed harmful-shift statistic.	required
`pvalue`	`float`	One-sided permutation p-value.	required
`null_distribution`	`NDArray[float64]`	Permutation null distribution of the statistic.	required
`worse`	`Worse`	Declared harmful direction tested by the result.	required

Attributes:

Name	Type	Description
`statistic`	`float`	Observed harmful-shift statistic.
`pvalue`	`float`	One-sided (`greater`) permutation p-value with +1 smoothing (always > 0) (Phipson & Smyth, 2010).
`null_distribution`	`NDArray[float64]`	Null distribution of the statistic (length `n_resamples`), produced by permuting group labels while keeping scores and weights fixed.
`worse`	`Worse`	The declared harmful direction that was tested.

See Also

test_shift : Two-sided screen for rank-based score separation. Worse : "higher" vs "lower" in plain language.

References

Kamulete, V. M. (2022). Test for non-negligible adverse shifts. Proceedings of the 38th UAI, PMLR 180:959-968. arXiv:2107.02990. Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be zero. Stat. Appl. Genet. Mol. Biol. 9(1):Article 39.

Source code in src/samesame/shift.py

@dataclass(frozen=True, repr=False)
class HarmfulShiftResult(ShiftResult):
    """
    Result of :func:`test_harm` — a one-sided permutation result.

    The statistic is Kamulete's weighted AUC
    ``∫ TPR·(1−FPR)² dFPR`` after orienting the score so larger means worse
    (``worse="lower"`` flips the sign). It gives more weight to thresholds
    the source rarely exceeds. Read it against ``null_distribution`` and
    the score's own scale, not against ``0.5``. See
    :doc:`How the harm test works <../explanation/harmful-shift-statistic>`
    for the ROC intuition.

    Parameters
    ----------
    statistic : float
        Observed harmful-shift statistic.
    pvalue : float
        One-sided permutation p-value.
    null_distribution : NDArray[np.float64]
        Permutation null distribution of the statistic.
    worse : Worse
        Declared harmful direction tested by the result.

    Attributes
    ----------
    statistic : float
        Observed harmful-shift statistic.
    pvalue : float
        One-sided (``greater``) permutation p-value with +1 smoothing
        (always > 0) (Phipson & Smyth, 2010).
    null_distribution : NDArray[np.float64]
        Null distribution of the statistic (length ``n_resamples``),
        produced by permuting group labels while keeping scores and
        weights fixed.
    worse : Worse
        The declared harmful direction that was tested.

    See Also
    --------
    test_shift : Two-sided screen for rank-based score separation.
    Worse : ``"higher"`` vs ``"lower"`` in plain language.

    References
    ----------
    Kamulete, V. M. (2022). Test for non-negligible adverse shifts.
        *Proceedings of the 38th UAI*, PMLR 180:959-968. arXiv:2107.02990.
    Phipson, B., Smyth, G. K. (2010). Permutation P-values should never be
        zero. *Stat. Appl. Genet. Mol. Biol.* 9(1):Article 39.
    """

    worse: Worse

    def __repr__(self) -> str:
        return (
            f"{type(self).__name__}("
            f"statistic={_fmt(self.statistic)}, pvalue={_fmt(self.pvalue)}, "
            f"worse={self.worse.value!r})"
        )

Public enum¶

`samesame.shift.Worse` ¶

Bases: StrEnum

Direction that defines which tail is harmful for :func:test_harm.

Choose worse from the score's definition (e.g., risk is higher-is-worse, confidence via LogitGap is lower-is-worse) and declare it before testing; do not pick the direction that gives the smaller p-value after seeing the data. A plain string "higher" / "lower" is accepted wherever this enum is; the two forms are interchangeable.

Attributes:

Name	Type	Description
`HIGHER`	`Worse`	Larger scores mean more harm (e.g., predicted risk, prediction error, or outlier score).
`LOWER`	`Worse`	Smaller scores mean more harm (e.g., confidence via `LogitGap`; lower is worse).

See Also

samesame.shift.test_harm : The test that consumes this choice.

Examples:

>>> from samesame import Worse
>>> Worse("higher") == Worse.HIGHER
True
>>> Worse("lower") == Worse.LOWER
True

Source code in src/samesame/shift.py

class Worse(StrEnum):
    """
    Direction that defines which tail is harmful for :func:`test_harm`.

    Choose ``worse`` from the score's definition (e.g., risk is higher-is-worse,
    confidence via ``LogitGap`` is lower-is-worse) and declare it before
    testing; do not pick the direction that gives the smaller p-value after
    seeing the data.
    A plain string ``"higher"`` / ``"lower"`` is accepted wherever this
    enum is; the two forms are interchangeable.

    Attributes
    ----------
    HIGHER : Worse
        Larger scores mean more harm (e.g., predicted risk, prediction
        error, or outlier score).
    LOWER : Worse
        Smaller scores mean more harm (e.g., confidence via ``LogitGap``;
        lower is worse).

    See Also
    --------
    samesame.shift.test_harm : The test that consumes this choice.

    Examples
    --------
    >>> from samesame import Worse
    >>> Worse("higher") == Worse.HIGHER
    True
    >>> Worse("lower") == Worse.LOWER
    True
    """

    HIGHER = "higher"
    LOWER = "lower"

Testing API¶

Public functions¶

samesame.shift.test_shift(source, target, *, n_resamples=9999, rng=None, weights=None) ¶

samesame.shift.test_harm(source, target, *, worse, n_resamples=9999, rng=None, weights=None) ¶

Public result classes¶

samesame.shift.ShiftResult dataclass ¶

samesame.shift.HarmfulShiftResult dataclass ¶

Public enum¶

samesame.shift.Worse ¶

`samesame.shift.test_shift(source, target, *, n_resamples=9999, rng=None, weights=None)` ¶

`samesame.shift.test_harm(source, target, *, worse, n_resamples=9999, rng=None, weights=None)` ¶

`samesame.shift.ShiftResult` `dataclass` ¶

`samesame.shift.HarmfulShiftResult` `dataclass` ¶

`samesame.shift.Worse` ¶