Statistics and survival

Experiments (sample size, power, assignment, reading an A/B test), two-sample tests, bootstrap intervals of an AUC, and survival analysis: Kaplan-Meier, concordance and Cox models with start-stop rows.

Experiment

public static class Experiment

Experiment design and reading: how many rows an A/B test needs (two proportions as R's power.prop.test; means as power.t.test, by the shifted t), its power, the smallest effect it can see, a reproducible (stratified) assignment of rows to arms, the effect with its interval, and CUPED (an outcome adjusted by the same measure before the test).

Methods

static double SampleSize(double baseline, double lift, double alpha = 0.05, double power = 0.8, double ratio = 1, bool twoSided = true)

Rows per arm (the control's; the treated arm has ratio times as many) to see a change of a rate from baseline to baseline + lift (lift as a difference: 0.02 = two points) with the power given, at significance alpha. R: power.prop.test(p1 = baseline, p2 = baseline + lift, power = 0.8) for ratio 1.

static double Power(double rows, double baseline, double lift, double alpha = 0.05, double ratio = 1, bool twoSided = true)

The chance a test with rows per arm (the control's) sees a change of a rate from baseline by lift. R: power.prop.test(n, p1, p2).

static double MinimumEffect(double rows, double baseline, double alpha = 0.05, double power = 0.8, double ratio = 1, bool decrease = false)

The smallest change of a rate (from baseline; down when decrease) a test with rows per arm sees with the power given: a test smaller than this effect needs cannot tell it from noise.

static double SampleSizeMeans(double standardDeviation, double delta, double alpha = 0.05, double power = 0.8, bool twoSided = true)

Rows per arm to see a difference delta between two means whose rows vary by standardDeviation (Welch's t on equal arms; the noncentral t approximated by the shifted central t, within one row of R's power.t.test).

static Table Assign(Table rows, string key, string[]? arms = null, double[]? shares = null, string[]? stratify = null, int seed = 1, string column = "Arm")

Assigns each row to an arm, reproducibly: the same key and seed always give the same arm, whatever the rows' order or which other rows there are. Within each stratum (a column's values: region, plan) the arms get their shares as exactly as the rows allow. Adds a column named arm.

• rows: The rows to assign (customers, shops).
• key: The column that names a row.
• arms: The arms' names (default control and treatment).
• shares: Each arm's share (default equal); they need not add to 1.
• stratify: Columns to balance the arms within (each combination of their values a stratum).
• seed: Another seed makes another assignment.
• column: The new column's name.

static Effect Compare(Table rows, string arm, string outcome, string control = "control", string? treated = null, double confidence = 0.95)

The effect of the treatment on an outcome: the rows' arm column, the control arm's name and the treated arm's (by default the other one). A 0/1 outcome is compared as two proportions, any other as two means (Welch).

static Effect Compare(double[] treated, double[] control, double confidence = 0.95)

The effect of the treatment: the treated rows' outcomes against the control rows' (missing values left out).

static Table Cuped(Table rows, string outcome, string before, string? column = null)

CUPED (Deng et al., 2013): the outcome less theta times (the same measure before the test less its mean), theta = cov(before, outcome) / var(before) over all rows. The effect is the same; its interval narrower by as much as before explains of the outcome. Adds a column "outcome (CUPED)"; rows without a before value keep their outcome.

static Metric SampleRatio(IReadOnlyList<long> counts, IReadOnlyList<double> shares)

The sample ratio mismatch test: whether the rows each arm got are what the design's shares would give by chance (a chi-square test). A p-value under 0.001 says the split went wrong somewhere (an arm's rows lost, a bug in the assignment), and the test's results are not to be trusted until that is found. Experiment.SampleRatio([3157, 1575, 1575, 1184], [0.42, 0.21, 0.21, 0.16]).

static Table Balance(Table rows, string arm, string control, IEnumerable<string> covariates)

Whether the arms were alike before the treatment: for each covariate (a number; a text column a 0/1 per value) and each arm, its mean, control's mean and the standardized mean difference (the difference over the two arms' pooled standard deviation). Imbalanced: |SMD| above 0.1, the usual line. Experiment.Balance(rows, "Arm", "Control", ["tenure", "Plan"]).

Effect

public sealed record Effect(
    string Method,
    int ControlRows,
    double ControlMean,
    int TreatedRows,
    double TreatedMean,
    double Difference,
    double Low,
    double High,
    double PValue,
    double Confidence)

A comparison of two groups (an A/B test's arms, treated and control): each one's rows and mean, the difference with its confidence interval and p-value (two-sided), and the method: two proportions (an outcome of 0 and 1, Wald's interval and the pooled z test, as R's prop.test without continuity correction) or Welch's t (any number).

Properties

double Relative { get; }

The difference relative to the control's mean (0.1 = 10 % more).

bool Significant { get; }

Whether the interval leaves out 0.

Also: string Method { get; init; }, int ControlRows { get; init; }, double ControlMean { get; init; }, int TreatedRows { get; init; }, double TreatedMean { get; init; }, double Difference { get; init; }, double Low { get; init; }, double High { get; init; }, double PValue { get; init; }, double Confidence { get; init; }.

KsTest

public sealed record KsTest(
    double Statistic,
    double PValue,
    int Reference,
    int Current)

Kolmogorov-Smirnov's two-sample test: the largest distance between the two samples' cumulative shares, and its p-value (asymptotic).

Properties

Its properties: double Statistic { get; init; }, double PValue { get; init; }, int Reference { get; init; }, int Current { get; init; }.

Bootstrap

public static class Bootstrap

Confidence intervals of an AUC and of the difference between two models' AUCs, by a bootstrap that resamples clusters (outlets, products, customers) rather than rows: rows of the same cluster are not independent (a product's monthly snapshots, an outlet's orders), and an interval that resamples rows is then too narrow.

The method: each resample draws as many clusters as there are, uniformly with replacement (xoshiro256** from the seed), and takes all of each drawn cluster's rows, as often as it was drawn; its AUC is the Mann-Whitney statistic of those rows (ties count half, as Metrics.Auc). A resample without rows with (or without) the outcome has no AUC and is left out (counted in Skipped). The percentile interval is the replicates' quantiles at (1 - confidence) / 2 and (1 + confidence) / 2 (R's quantile type 7); BCa (Efron's bias-corrected and accelerated, the default) moves those levels by z0 (the normal quantile of the share of replicates below the estimate) and the acceleration a (from the jackknife that leaves out one cluster at a time). Without a cluster column each row is its own cluster (the ordinary bootstrap). Beside it, DeLong's interval treats the rows as independent (R's pROC::ci.auc and roc.test with method "delong"): the ratio of the two standard errors is what the clusters cost.

Clusters are put in order of their key (text order), so the rows' order does not change the result: the same rows, clusters, replicates and seed give the same interval on any computer. 2,000 replicates (the default) are enough for a 95 % BCa interval to the second decimal; ten thousand for the third.

Methods

static AucInterval Auc(double[] score, double[] outcome, Column? cluster = null, int replicates = 2000, double confidence = 0.95, string method = "bca", int seed = 1, string name = "AUC")

The AUC of a score with its confidence interval from a cluster bootstrap (each row its own cluster when no cluster is given), and DeLong's interval for the rows taken as independent beside it. Rows where the score or the outcome is missing are left out.

• score: The model's scores (higher: the outcome more likely).
• outcome: What happened: 1 or 0 (true or false).
• cluster: Each row's cluster (an outlet, a product, a customer: rows of one are not independent), as a column of the same rows (test["Variantnr"]); none: each row is its own.
• replicates: How many resamples: 2,000 by default, ten thousand for a third decimal.
• confidence: 0.95 for a 95 % interval.
• method: "bca" (bias-corrected and accelerated, the default) or "percentile".
• seed: The resamples' seed: the same seed gives the same interval.
• name: What it is called in the result.

static AucInterval Auc(Table rows, string score, string outcome, string? cluster = null, int replicates = 2000, double confidence = 0.95, string method = "bca", int seed = 1)

The AUC of a column of scores, with its interval from a cluster bootstrap (Auc(score, outcome, cluster) on the table's columns).

• rows: The rows scored (a backtest's Scores, the test rows with a column of scores).
• score: The score column.
• outcome: The outcome column (1 or 0).
• cluster: The cluster column (the outlet, the product); none: each row is its own.
• replicates: How many resamples (2,000).
• confidence: 0.95 for a 95 % interval.
• method: "bca" or "percentile".
• seed: The resamples' seed.

static AucInterval AucDifference(double[] score, double[] versus, double[] outcome, Column? cluster = null, int replicates = 2000, double confidence = 0.95, string method = "bca", int seed = 1, string name = "AUC difference")

The difference between two models' AUCs on the same rows (AUC of score minus AUC of versus), with its confidence interval from a cluster bootstrap (both scored on each resample, so their correlation counts) and a p-value of no difference; DeLong's paired interval and test for the rows taken as independent beside it. Rows where either score or the outcome is missing are left out.

• score: The first model's scores (the new one).
• versus: The second model's scores of the same rows (the simple rule, the version in use).
• outcome: What happened: 1 or 0.
• cluster: Each row's cluster, as a column of the same rows; none: each row is its own.
• replicates: How many resamples (2,000).
• confidence: 0.95 for a 95 % interval.
• method: "bca" or "percentile".
• seed: The resamples' seed.
• name: What it is called in the result.

static AucInterval AucDifference(Table rows, string score, string versus, string outcome, string? cluster = null, int replicates = 2000, double confidence = 0.95, string method = "bca", int seed = 1)

The difference between two score columns' AUCs (the first's minus the second's), with its interval from a cluster bootstrap.

• rows: The rows scored by both.
• score: The first model's score column.
• versus: The second model's score column.
• outcome: The outcome column (1 or 0).
• cluster: The cluster column; none: each row is its own.
• replicates: How many resamples (2,000).
• confidence: 0.95 for a 95 % interval.
• method: "bca" or "percentile".
• seed: The resamples' seed.

static AucInterval DeLong(double[] score, double[] outcome, double confidence = 0.95, string name = "AUC")

DeLong's interval of an AUC with the rows taken as independent (R's pROC::ci.auc(method = "delong"), clipped to 0 and 1): right when every row is its own unit, too narrow when rows come in clusters.

static AucInterval DeLongDifference(double[] score, double[] versus, double[] outcome, double confidence = 0.95, string name = "AUC difference")

DeLong's paired test of two AUCs on the same rows taken as independent (R's pROC::roc.test(paired = TRUE, method = "delong")): the difference, its interval and p-value.

static IEnumerable<int[]> Draws(int clusters, int replicates, int seed)

The resamples a bootstrap of this many clusters draws from a seed, in order: for each, how often each cluster was drawn (cluster k is the interval's ClusterKeys[k]). With them the interval can be checked elsewhere (R, Python): each resample's AUC is that of the rows, each as often as its cluster was drawn.

AucInterval

public sealed record AucInterval(
    string Name,
    double Estimate,
    double Low,
    double High,
    double Confidence,
    string Method)

An AUC (or the difference of two models' AUCs on the same rows) with its confidence interval: from a bootstrap that resamples clusters (Bootstrap.Auc, Bootstrap.AucDifference), with DeLong's interval for the rows taken as independent beside it (AucInterval.Independent), so what the clusters change shows.

Properties

string Name { get; init; }

What is measured: "AUC of score", "AUC of model − AUC of rule".

double Estimate { get; init; }

The AUC (or the difference) on the rows themselves.

double Low { get; init; }

The interval's low end, by Method.

double High { get; init; }

The interval's high end.

double Confidence { get; init; }

0.95 for a 95 % interval.

string Method { get; init; }

"BCa" or "percentile" (a bootstrap's), "DeLong" (rows taken as independent).

double StandardError { get; init; }

The standard error: the bootstrap replicates' standard deviation (n - 1), or DeLong's.

double PValue { get; init; }

A difference's two-sided p-value of no difference: the difference over its standard error, against the normal (as R's pROC::roc.test with method "bootstrap" or "delong"). NaN for one AUC.

int Replicates { get; init; }

How many resamples were drawn (a bootstrap's; 0 for DeLong).

int Seed { get; init; }

The seed the resamples were drawn from: the same rows, clusters and seed give the same interval, on any computer.

string ResampledBy { get; init; }

What was resampled: "rows", or the cluster column's name.

int Clusters { get; init; }

How many clusters were resampled (the rows, when each row is its own).

int Rows { get; init; }

How many rows were measured (rows with a score and an outcome).

double PercentileLow { get; init; }

The percentile interval's low end (the replicates' quantile at (1 - confidence) / 2; R's type 7).

double PercentileHigh { get; init; }

The percentile interval's high end.

double BcaLow { get; init; }

The BCa interval's low end (NaN when it cannot be computed: see AucInterval.Warnings).

double BcaHigh { get; init; }

The BCa interval's high end.

double BiasCorrection { get; init; }

BCa's bias correction z0: the normal quantile of the share of replicates below the estimate.

double Acceleration { get; init; }

BCa's acceleration a, from the jackknife that leaves out one cluster at a time.

int Skipped { get; init; }

Resamples without a row with (or without) the outcome: they have no AUC and are left out.

IReadOnlyList<double> Values { get; init; }

Each resample's AUC (or difference), in the order they were drawn (NaN for a skipped one).

IReadOnlyList<string> ClusterKeys { get; init; }

The clusters in the order the resamples number them (their keys in text order; without clusters, the rows' positions): Bootstrap.Draws(Clusters, Replicates, Seed) gives each resample's count of each, to check the interval elsewhere.

AucInterval? Independent { get; init; }

DeLong's interval with the rows taken as independent (what an interval that ignores the clusters says).

IReadOnlyList<string> Warnings { get; init; }

What to know before reading it: few clusters, BCa not computable, resamples left out.

bool Significant { get; }

Whether the interval leaves out the value of no information (0.5 for one AUC, 0 for a difference).

Survival

public static class Survival

Survival analysis in a notebook, as R's survival package computes it: how long until an event (a customer leaves, a policy lapses), the rows still without one counted as censored (they lasted at least so long). Harrell's concordance, Kaplan-Meier, Breslow's baseline hazard, and Cox's proportional hazards model: the model a boosted one (LightGbm.TrainSurvival) has to beat.

Methods

static double Concordance(double[] risk, double[] time, double[] evt)

Harrell's concordance: among pairs of rows in which the row with the shorter time had the event, how often that row was given the higher risk (a tie in risk counts half; two events at the same time make no pair). 0.5 is no better than chance; 1 orders every pair right.

• risk: Each row's risk (higher: the event sooner): a raw score, a linear predictor.
• time: Each row's duration (to its event, or to now).
• evt: 1 (true) for a row whose stretch ended with the event, 0 for one still running.

static (double[] Time, double[] Surv) KaplanMeier(double[] time, double[] evt)

Kaplan-Meier's curve: at each distinct time, the share still without the event.

static double At((double[] Time, double[] Surv) curve, double t)

A step curve's value at t: its last value at or before t (1 before its first time).

static (double[] Time, double[] CumHaz) Breslow(double[] score, double[] time, double[] evt, double offset)

Breslow's estimate of the cumulative baseline hazard at each event time, for risks exp(score - offset): at a time with d events it adds d divided by the sum of the risks of the rows still at risk. A row's survival is then exp(-H0(t) exp(score - offset)).

static CoxModel Cox(Table rows, IEnumerable<string> features, string duration, string evt, double ridge = 0.0001, string? entry = null, string ties = "breslow")

Cox's proportional hazards on a table's columns, as R's coxph: a number standardized (missing at its mean), text a column per value but the most common (up to 30 values; missing in none). Ties as coxph(ties = "breslow") by default, or Efron's (coxph's own default). With entry, each row is a stretch of time from its entry to its duration (R's Surv(start, stop, event)): a customer's rows, a month each with what held then (covariates that change in time), or rows that came under observation late (delayed entry). The simple model a boosted one has to beat; also what explains in hazard ratios, with their standard errors, intervals and p-values (ToTable).

• rows: The rows learned from.
• features: Its columns that may explain the time.
• duration: How long each row lasted, until its event or until now (with entry: the end of the row's stretch).
• evt: 1 (true) for a row whose stretch ended with the event, 0 for one still running.
• ridge: A small penalty on the standardized coefficients, so a column that separates the events still converges (0: none, as coxph; the standard errors include it).
• entry: Each row's start: it is at risk at a time t when entry < t ≤ duration. None: from 0.
• ties: "breslow" (the default) or "efron": how events at the same time share the risk set.

CoxModel

public sealed class CoxModel

A fitted Cox model: its terms and coefficients (a number's per its own unit), their standard errors, the rows and events it learned from, and how (ties, entries).

Properties

IReadOnlyList<double> Beta { get; }

The coefficients on the standardized columns (a number's per standard deviation).

IReadOnlyList<double> StdErrors { get; }

The standard errors of CoxModel.Beta (on the standardized columns), from the inverse information at the optimum.

IReadOnlyList<double> Coefficients { get; }

Each term's coefficient per its own unit (a number's per one of it; a value's against the reference): R's coef(fit).

IReadOnlyList<double> UnitStdErrors { get; }

The coefficients' standard errors per their own unit: R's sqrt(diag(vcov(fit))).

int Rows { get; }

The rows (with entries: the stretches of time) it learned from.

double LogLikelihood { get; }

The partial log-likelihood at the coefficients (without the ridge): R's fit$loglik[2].

string? Entry { get; }

The column of each row's entry (start/stop rows), or null (every row from 0).

string Ties { get; }

"breslow" or "efron".

IReadOnlyList<(string Term, double HazardRatio)> HazardRatios { get; }

Each term's hazard ratio: a number's per unit of it, a value's against the reference.

Also: IReadOnlyList<CoxTerm> Terms { get; }, int Events { get; }.

Methods

double[] Predict(Table rows)

Each row's linear predictor (its log relative hazard): higher, the event sooner.

Table ToTable()

The model as a table, a row a term, as R's summary(coxph): Term, Hazard ratio (per unit, or against the reference value), Coefficient, Std error, Lower and Upper (the hazard ratio's 95 % interval) and P-value (Wald's, two-sided).

Also: string ToString().

CoxTerm

public sealed record CoxTerm(
    string Column,
    string? Level,
    string? Reference,
    double Mean,
    double Scale)

A column of a Cox model's design: a number (its mean and scale) or a text value against a reference.

Properties

Its properties: string Column { get; init; }, string? Level { get; init; }, string? Reference { get; init; }, double Mean { get; init; }, double Scale { get; init; }, string Name { get; }.

SurvivalBaseline

public sealed record SurvivalBaseline(
    string Duration,
    string Event,
    double Offset,
    double[] Time,
    double[] CumHaz,
    double MedianTime)

What a survival model keeps to give each row its curve: the duration and event columns it learned from, the offset of its scores (their mean on the training rows: a risk of 1 is the average row's) and Breslow's baseline.

Properties

Its properties: string Duration { get; init; }, string Event { get; init; }, double Offset { get; init; }, double[] Time { get; init; }, double[] CumHaz { get; init; }, double MedianTime { get; init; }.

Methods

double SurvivalAt(double score, double t)

A row's survival at t from its raw score: exp(-H0(t) exp(score - offset)).

Also: JsonObject ToJson(), static SurvivalBaseline? FromJson(JsonNode? n).

Last updated 6 November 2026 · OpenBI Studio 0.3.78