Statistics and survival

Experiments from their size to their reading, intervals for a model’s quality, survival curves and Cox models, and the statistics OAX has for measures. The examples use the made-up customers of Synthetic.Retention, where a retention offer was given at random.

Experiments

Before a test, how many rows each arm needs to see a change, at the usual 5 % significance and 80 % power:

// rows per arm to see churn fall from 10 % to 8 %, at the usual 5 % and 80 % power
Math.Ceiling(Experiment.SampleSize(baseline: 0.10, lift: -0.02))

3,213

Experiment.Power and Experiment.MinimumEffect answer the same question the other ways round, and Experiment.Assign splits rows into arms, stratified when asked. After it, Experiment.Compare reads it: a 0/1 outcome as two proportions, any other as two means:

var month = Synthetic.Retention(customers: 4000, months: 3, seed: 5).Where(r => r.Date("as_of") == new DateTime(2024, 2, 1))
    .With("arm", r => r.Num("treated") == 1 ? "offer" : "control");
Experiment.Compare(month, arm: "arm", outcome: "churned", control: "control")

two proportions: treated 0.071 (1,816 rows) against control 0.098 (1,837): difference -0.027 (-27.5%, 95% interval -0.045 to -0.0089), p = 0.003

When a measure taken before the test explains much of the outcome, Experiment.Cuped takes it out, and the interval narrows without moving the effect:

// spend in the month before explains most of this month's: CUPED narrows the interval without moving the effect
var adjusted = Experiment.Cuped(month, outcome: "monthly_spend", before: "spend_before");
Display(Experiment.Compare(month, "arm", "monthly_spend"), Experiment.Compare(adjusted, "arm", "monthly_spend (CUPED)"));

Welch's t: treated 38.7966 (1,816 rows) against control 38.5166 (1,837): difference 0.28 (+0.7%, 95% interval -1.418 to 1.978), p = 0.746: not distinguishable from no effect

Welch's t: treated 38.7394 (1,816 rows) against control 38.5731 (1,837): difference 0.1663 (+0.4%, 95% interval -0.2459 to 0.5786), p = 0.429: not distinguishable from no effect

Experiment.SampleRatio checks that the arms got the shares they should, and Experiment.Balance that their rows look alike before the treatment.

How sure is a number

An AUC from a test set is an estimate. Bootstrap.Auc gives its interval, resampling whole customers when each has several rows, and says how it compares with DeLong’s, which takes rows as independent:

// how sure is an AUC: a bootstrap over customers (their months together), 2,000 draws
var all = Synthetic.Retention(customers: 3000, months: 12, seed: 7);
Bootstrap.Auc(all, score: "days_since_last", outcome: "churned", cluster: "customer")

AUC of days_since_last: 0.707, 95% interval 0.693 to 0.720 (BCa, 2,000 resamples of 3,000 customer clusters, seed 1; bootstrap standard error 0.0068). Rows taken as independent (DeLong): 0.693 to 0.720; the clusters add little uncertainty here (standard error 1.01 times DeLong's).

Bootstrap.AucDifference and Bootstrap.DeLongDifference say whether one model beats another on the same rows.

Survival

How long until an event, a customer leaving or a machine failing, when some haven’t had it yet. Those still going are censored, not “stayed”: they lasted at least so long. One row per customer:

// one row per customer: how many months we saw them, whether they left (else still going: censored)
var people = Table.FromRecords(Synthetic.Retention(customers: 3000, months: 12, seed: 7).Rows
    .GroupBy(r => r.Text("customer")).Select(g => new
    {
        customer = g.Key, months = (double)g.Count(), left = g.Last().Num("churned"),
        tickets = g.First().Num("support_tickets"), days_since_last = g.First().Num("days_since_last"), plan = g.First().Text("plan"),
    }));
var km = Survival.KaplanMeier(people["months"], people["left"]);
$"{people.Count:N0} customers; still with us after 6 months: {Survival.At(km, 6):P1}, after 12: {Survival.At(km, 12):P1}"

3,000 customers; still with us after 6 months: 62.2 %, after 12: 43.8 %

Survival.KaplanMeier gives the curve; Survival.Cox fits Cox’s proportional hazards model, its hazard ratios per unit of each number and against a reference for each text value:

var cox = Survival.Cox(people, new[] { "tickets", "days_since_last", "plan" }, "months", "left");
cox

Cox's proportional hazards on 3,000 rows (1,686 events): tickets 1.116, days_since_last 1.021, plan: plus 0.806, plan: premium 0.769

Each support ticket raises the hazard by about 12 %; the plus and premium plans lower it against basic. cox.Predict(rows) gives each row’s risk and Metrics.Concordance how well it orders who leaves first. With entry, the model takes start-stop rows, a row per month with what held then; Chart.Survival draws curves by risk.

Statistics in OAX

OAX has statistics for steps and measures alike: regressions (LINESTX), correlations, distributions and their inverses, confidence intervals, clustering and decomposition. In a step:

// a regression in OAX: revenue against liters over the shops
EVALUATE
LINESTX (
    SUMMARIZECOLUMNS ( Sales[Shop], "Liters", [Liters sold], "Revenue", [Revenue] ),
    [Revenue], [Liters]
)

The same query works in a measure or on a report page, on the rows in its filters (see the OAX functions).

Every member is in the Statistics and survival reference.

Last updated 6 November 2026 · OpenBI Studio 0.3.78