Learning

LightGBM from C#, and what keeps a model honest: datasets, training with early stopping, time and group folds, tuning, metrics and the verdict, calibration, drift, walk-forward backtests, uplift and targeting policies, and forecasts.

LightGbm

public static class LightGbm

LightGBM (MIT licensed) from C#: the same library OpenBI's engine scores with, so a model learned in a notebook scores the same in the model, Get data and HTTP.

Properties

static IReadOnlyDictionary<string, string?> DatasetParameters { get; }

LightGBM's parameters of the rows, read when a Dataset is made (and their defaults): given to Train with another value they cannot apply, and some (linear_tree) end the process instead of failing.

static (string Path, string Version, string Sha256) Library { get; }

The library's file, version and fingerprint (kept with each model version).

Methods

static Dataset Dataset(Table rows, string label, IEnumerable<string>? features = null, IEnumerable<string>? categorical = null, Dataset? reference = null, Dictionary<string, object>? parameters = null)

Rows to learn from (or to check on, with reference: the training rows, whose coding they share).

• rows: The rows.
• label: The outcome column (true/false, 0/1 or a number).
• features: The input columns; all but the outcome when not given.
• categorical: Number columns to treat as categories (text columns always are).
• reference: The training rows, when these are validation rows.
• parameters: Dataset parameters (max_bin, min_data_in_bin, …).

static IReadOnlyList<(Table Train, Table Valid)> TimeFolds(Table rows, string date, int folds = 4, int? periods = null, int gap = 0)

Folds in time order, as scikit-learn's TimeSeriesSplit but over the values of date (a period's rows stay together): the latest periods are cut into folds parts of periods each (by default the dates split into folds + 1 equal parts); fold k learns from what came before its part, less gap periods, and is checked on it. No fold learns from what came after the rows it is checked on.

static CrossValidation CrossValidate(IEnumerable<(Table Train, Table Valid)> folds, string label, Dictionary<string, object> parameters, IEnumerable<string>? features = null, IEnumerable<string>? categorical = null, Dictionary<string, object>? datasetParameters = null, int rounds = 1000, int earlyStopping = 50)

Cross-validation: on each fold (LightGbm.TimeFolds for time), a model learns from its training rows and stops early on its checking rows; the folds' best values and iterations say how good the parameters are and how many trees to give the final model (Trees), which then learns from all the training rows without looking at the held-back ones.

static Booster Train(Dataset train, Dataset? valid, Dictionary<string, object> parameters, int rounds = 100, int earlyStopping = 0, bool trainingCurve = true)

Learns a model. Each iteration adds a tree; with validation rows and earlyStopping, it stops when the validation metric has not improved for that many iterations, and the model keeps the best iteration. Unless given, seed=42, deterministic=true and force_row_wise=true: the same rows and parameters give the same model, bit for bit.

The booster keeps the metric after every iteration on the validation rows (Booster.History) and on the training rows (Booster.TrainingHistory, as LightGBM's Python package records it with the training set in valid_sets): Chart.Loss draws both, so a training loss far below the validation loss shows the model learning its training rows by heart. Reading the training metric does not change the model; with very many rows and a metric that sorts them (auc, ndcg, map), trainingCurve: false saves its time.

static bool HigherIsBetter(string metric)

Whether a larger value of a LightGBM metric is better (auc, ndcg, map, average_precision); else smaller is (a loss).

static Booster TrainSurvival(Dataset train, Dataset? valid, string evt, Dictionary<string, object> parameters, int rounds = 1000, int earlyStopping = 0)

Learns the time until an event, like Cox's proportional hazards model but boosted. The datasets' label is each row's duration (until its event, or until now); evt names a column that is 1 (true) for a row whose stretch ended with the event and 0 for one still running (censored). LightGBM has no survival objective itself, so every tree fits the gradient of Cox's partial likelihood (Breslow's handling of ties, as XGBoost's survival:cox), and training stops once the partial likelihood on the validation rows no longer improves. The raw score is the log of a row's relative hazard; Booster.Survival on the booster keeps Breslow's baseline from the training rows, which gives each row its curve: S0(t) raised to exp(score - offset).

static string Monotone(Dataset data, params (string Feature, int Direction)[] constraints)

Monotone constraints by feature name, for Train's "monotone_constraints": +1 a feature can only raise the score as it grows, -1 only lower it, others free. LightGbm.Monotone(data, ("days_since_last", 1), ("discount", -1)).

static string Interactions(Dataset data, params string[][] groups)

Interaction constraints by feature name, for Train's "interaction_constraints": each group's features may be in one tree's branches together; a feature in no group is used alone. LightGbm.Interactions(data, ["price", "discount"], ["tenure", "plan"]).

static int KeyFold(object? key, int folds = 5)

Folds that keep each group's rows together (scikit-learn's GroupKFold, reproducible by seed): a customer's monthly snapshots are all learned from or all checked on, never both - otherwise a fold checks on rows it nearly saw.

static Tuning Tune(IEnumerable<(Table Train, Table Valid)> folds, string label, Dictionary<string, object> parameters, Dictionary<string, object[]> space, int trials = 20, int seed = 1, IEnumerable<string>? features = null, IEnumerable<string>? categorical = null, Dictionary<string, object>? datasetParameters = null, int rounds = 1000, int earlyStopping = 50)

A hyperparameter search (random, over the values given for each parameter) cross-validated on the folds: each trial's parameters are the base ones with one value of each searched parameter; the first trial is each one's first value. Folds from LightGbm.TimeFolds (time) or GroupFolds (customers): the held-back test rows stay out of it. Then Train(final data, null, tuning.Parameters, rounds: tuning.Trees) on all the training rows.

• space: For each parameter, the values to try: ["num_leaves"] = new object[] { 15, 31, 63 }.
• trials: How many different combinations to try (at most all of them).

static Booster TrainCustom(Dataset train, Dataset? valid, Dictionary<string, object> parameters, Func<double[], double[], (double[] Gradient, double[] Hessian)> objective, Func<double[], double[], double>? loss = null, int rounds = 100, int earlyStopping = 0)

Learns with an objective of your own: objective(scores, labels) gives each row's gradient and Hessian of the loss at its raw score (the second derivative: above 0); loss(scores, labels), when given, is the validation rows' loss (lower is better) for early stopping. Scores are raw (no link applied): Predict gives raw scores too. Such a model is the notebook's: a version cannot keep a function, so Models.Save refuses it (save its scores with Results.Save). LightGbm.TrainCustom(data, valid, new() { ["learning_rate"] = 0.05 }, (s, y) => (s.Select((v, i) => v - y[i]).ToArray(), s.Select(_ => 1.0).ToArray())).

Also: static IReadOnlyList<(Table Train, Table Valid)> GroupFolds(Table rows, string group, int folds = 5, int seed = 1).

Dataset

public sealed class Dataset : IDisposable

Rows LightGBM learns from or is checked on: a table's features and outcome, as LightGBM keeps them.

Properties

IReadOnlyDictionary<string, string> Parameters { get; }

The dataset parameters it was read with (a model version keeps them, to retrain it the same).

string? WeightColumn { get; }

The column whose values weigh each row (Weights), kept with a version so that it retrains the same.

string? GroupColumn { get; }

The column whose values group the rows into queries (Groups: lambdarank), when given.

Also: Table Rows { get; }, string Label { get; }, IReadOnlyList<Feature> Features { get; }, int Count { get; }.

Methods

void Dispose()

Frees LightGBM's copy of the rows (also done when the dataset is no longer used: notebooks write var, not using var).

Dataset Weights(string column)

Weighs each row by a column's value (LightGBM's weight): rarer classes up (instead of scale_pos_weight, which leaves the scores uncalibrated less often than is_unbalance does), recent rows more than old ones, a policy's exposure. Before Train; a version keeps the column with the training rows. Returns the dataset.

Dataset Weights(double[] weights)

Weighs each row (one weight per row, 0 or more).

Dataset Groups(string column)

The queries a ranking model (objective lambdarank or rank_xendcg) ranks within: a column whose rows of one value are next to each other (OrderBy that column first). Each query's rows are ranked against each other only; the label is each row's relevance (0, 1, 2 …). Returns the dataset.

Dataset InitScore(double[] scores)

A starting raw score per row, that the trees add to (LightGBM's init_score): an offset such as the log of each policy's exposure for a poisson model of claims, or another model's log-odds to boost on. Scoring new rows does not add it back: add it to PredictRaw yourself. Returns the dataset.

Also: string ToString().

Booster

public sealed class Booster : IDisposable

A learned LightGBM model: it scores rows, explains scores (SHAP contributions) and saves as LightGBM's own text format, which any LightGBM (Python, R, C) reads.

Properties

IReadOnlyDictionary<string, string> Parameters { get; }

The parameters it learned with (OpenBI's defaults included).

string? Metric { get; }

The validation metric (its first), or null without validation rows.

IReadOnlyList<Iteration> History { get; }

The validation rows' metric after each iteration (empty without validation rows).

string? TrainingMetric { get; }

The metric the training rows were checked with (the first, as Booster.Metric), or null when not recorded.

IReadOnlyList<Iteration> TrainingHistory { get; }

The training rows' metric after each iteration: beside Booster.History, how much better the model does on the rows it learned from than on new ones (a wide gap is overfitting). Empty for a model read from a file.

(double Training, double Validation) NullLoss { get; }

A survival model's loss (cox_nloglik) of a model that rates every row alike, on the training and the validation rows (NaN: not known). Cox's loss grows with how many rows are at risk, so two sets of different sizes compare from their own: Chart.Loss draws each less its own.

int EarlyStopping { get; }

How many iterations without improvement on the validation rows would stop it (0: none).

bool StoppedEarly { get; }

Whether it stopped early: Booster.EarlyStopping iterations after its best, before its last round.

int BestIteration { get; }

The iteration it keeps: the best on the validation rows (all of them without).

int Iterations { get; }

How many iterations it ran.

IReadOnlyList<string> Notes { get; }

Notes from reading a model file (categories it could not place).

SurvivalBaseline? Survival { get; }

A survival model's baseline (LightGbm.TrainSurvival's): each row's curve from its raw score; null for other models.

bool Custom { get; }

Learned with an objective of the notebook's own (LightGbm.TrainCustom): its scores are raw.

Also: IReadOnlyList<Feature> Features { get; }, double BestValue { get; }, string Objective { get; }.

Methods

double[] SurvivalAt(Table rows, double t)

A survival model's chance that each row is still without the event at t (its curve, from Breslow's baseline).

double[] Predict(Table rows)

A score per row: a probability for binary objectives, the predicted value for regression.

double[] PredictRaw(Table rows)

Raw scores (log-odds for binary).

double[,] Contributions(Table rows)

Each row's SHAP contributions: per feature, how much it moved the raw score, and last the base value.

IReadOnlyList<(string Feature, double Strength)> Drivers(Table rows)

The features by how much they moved the scores of these rows (mean absolute SHAP contribution).

string[][] Reasons(Table rows, int top = 3)

Per row, the features that pushed its score up the most, written out ("days_since_last = 212").

IReadOnlyList<(string Feature, double Gain)> Importance()

How much each feature was used: total gain of its splits.

string ToModelText()

The model in LightGBM's text format (the best iteration's trees), with the text categories on the last line as LightGBM's Python package writes them (pandas_categorical).

static Booster Load(string modelText, IReadOnlyList<Feature>? features = null)

A model from LightGBM's text format. Features (with their categories) from OpenBI's schema when given; else read from the text.

void Dispose()

Frees the model (also done when it is no longer used).

double[,] PredictClasses(Table rows)

A multiclass model's probabilities (objective multiclass, num_class = k): a row per row, a column per class, in the order of the label's values 0 … k-1. One-score models give one column.

int[,] Leaves(Table rows)

The leaf each row lands in, per tree (a row per row, a column per tree): rows that share leaves are alike to the model.

IReadOnlyList<(string Feature, double Value)> Importance(string type)

How much each feature was used: "gain" (the default: its splits' total gain) or "split" (how many splits).

Also: string ToString().

Iteration

public sealed record Iteration(int Number, double Value)

A validation result after an iteration.

Properties

Its properties: int Number { get; init; }, double Value { get; init; }.

Feature

public sealed record Feature(
    string Name,
    string Column,
    string Kind,
    bool Categorical,
    IReadOnlyList<string>? Categories = null)

An input of a model: its name in the model, the table column it reads, and how values become numbers. Text is categorical, coded by its position in Feature.Categories (an unseen category is missing), as LightGBM's Python package codes pandas categories.

Properties

Its properties: string Name { get; init; }, string Column { get; init; }, string Kind { get; init; }, bool Categorical { get; init; }, IReadOnlyList<string>? Categories { get; init; }.

Methods

Its methods: JsonObject ToJson(), static Feature FromJson(JsonNode n).

CrossValidation

public sealed record CrossValidation(
    IReadOnlyList<CvFold> Folds,
    string Metric,
    double Mean,
    double Worst,
    int Trees)

A cross-validation's folds, their average and worst value of the metric, and the trees to train the final model with.

Properties

int EarlyStopping { get; init; }

How many iterations without improvement stopped each fold (0: none).

Also: IReadOnlyList<CvFold> Folds { get; init; }, string Metric { get; init; }, double Mean { get; init; }, double Worst { get; init; }, int Trees { get; init; }.

Methods

Table ToTable()

The folds as a table: fold, rows, best iteration, value.

CvFold

public sealed record CvFold(
    int Fold,
    int TrainRows,
    int ValidRows,
    int BestIteration,
    double Value)

One fold of a cross-validation: its rows, its best iteration and its best validation value.

Properties

IReadOnlyList<Iteration> History { get; init; }

Its validation rows' metric after each iteration.

IReadOnlyList<Iteration> TrainingHistory { get; init; }

Its training rows' metric after each iteration.

bool StoppedEarly { get; init; }

Whether it stopped early: no better on its validation rows for the early stopping's iterations.

Also: int Fold { get; init; }, int TrainRows { get; init; }, int ValidRows { get; init; }, int BestIteration { get; init; }, double Value { get; init; }.

Tuning

public sealed record Tuning(
    IReadOnlyList<TuningTrial> Trials,
    TuningTrial Best,
    Dictionary<string, object> Parameters,
    int Trees,
    bool HigherIsBetter)

A hyperparameter search: every trial's parameters and cross-validated score, the best trial's parameters (with the base ones) and how many trees to train the final model with.

Properties

Its properties: IReadOnlyList<TuningTrial> Trials { get; init; }, TuningTrial Best { get; init; }, Dictionary<string, object> Parameters { get; init; }, int Trees { get; init; }, bool HigherIsBetter { get; init; }.

Methods

Table ToTable()

The trials as a table, the best first: each searched parameter, the mean and worst fold, the trees.

TuningTrial

public sealed record TuningTrial(
    int Trial,
    IReadOnlyDictionary<string, object> Parameters,
    CrossValidation Result)

One trial of a search: its parameters and how they cross-validated.

Properties

Its properties: int Trial { get; init; }, IReadOnlyDictionary<string, object> Parameters { get; init; }, CrossValidation Result { get; init; }.

Metrics

public static class Metrics

How well scores rank and predict an outcome. Rows where either is missing are left out.

Methods

static Metric Brier(double[] probability, double[] outcome)

The Brier score: the mean squared distance between probability and outcome (0 is perfect; always saying the rate scores rate × (1 - rate)).

static Metric CalibrationError(double[] probability, double[] outcome, int bins = 10)

The expected calibration error: over bins of equal width, how far each bin's mean probability is from how often its rows had the outcome, weighted by its rows (0 is calibrated; 0.02 is two points off on average).

static Metric Auc(double[] score, double[] outcome, string name = "AUC")

The area under the ROC curve: the chance a row with the outcome scores above one without (0.5: no better than chance).

static Metric Concordance(double[] risk, double[] time, double[] evt, string name = "concordance")

Harrell's concordance of a survival model: of two rows, how often it says right which has the event first (0.5: chance). risk: higher, sooner (a raw score, a linear predictor); time: to the event or to now; evt: 1 when it happened, 0 while still going.

static Metric LogLoss(double[] probability, double[] outcome)

The mean log loss of probabilities against a 0/1 outcome (lower is better).

static Metric Wape(double[] predicted, double[] actual)

How far off, relative to the total: the sum of absolute errors over the sum of actuals.

static Metric WapeOfTotals(double[] predicted, double[] actual, Column period, double[]? sameRowsAs = null)

How far off the total is: every series' forecast and actual added up per period, then the totals' absolute errors over the actual totals - what a plan of the total sees (per row, Wape, the series' errors are counted apart; in the total they partly cancel, or add up when they move together). Rows where either is missing are left out, and with sameRowsAs (another forecast of the same rows: the simple rule's) the rows where that one is missing too, so two totals compare on the same rows.

static Metric Rate(double[] outcome)

The share of rows with the outcome.

static Metric Coverage(double[] low, double[] high, double[] actual)

The share of actual values inside their range (low to high, either way round): for a range from the 10th to the 90th percentile, about 80% when the range is right.

static IReadOnlyList<Tenth> Tenths(double[] score, double[] outcome)

The rows in ten groups by score, highest first: how often each has the outcome.

Also: static Metric Rmse(double[] predicted, double[] actual), static Metric Mae(double[] predicted, double[] actual), static Metric R2(double[] predicted, double[] actual).

Metric

public sealed record Metric(
    string Name,
    double Value,
    string? Note = null)

A measured quality of a model: a name and a value (shown as "AUC 0.821").

Properties

Its properties: string Name { get; init; }, double Value { get; init; }, string? Note { get; init; }.

Operators

Its operators: static implicit operator double(Metric m).

Tenth

public sealed record Tenth(
    int Number,
    int Rows,
    double MeanScore,
    double Rate)

One tenth of the rows, ranked by a score (first: the highest scores).

Properties

Its properties: int Number { get; init; }, int Rows { get; init; }, double MeanScore { get; init; }, double Rate { get; init; }.

Verdict

public sealed record Verdict(string Level, string Text)

How good a model is, in words anyone reads: strong, useful, weak, no better (than the simple rule) or uncertain, a sentence that says why, and what to check (warnings). The engine's rules, so the same numbers read the same in a notebook, in a version's evidence and from the agent.

Properties

Its properties: string Level { get; init; }, string Text { get; init; }, IReadOnlyList<string> Warnings { get; init; }.

Methods

Verdict With(IEnumerable<string> warnings)

Verdict With(params string[] warnings)

The same verdict, with these warnings too.

static Verdict Ranking(double[] score, double[] outcome, double[]? rule = null, string? ruleName = null, (double High, double Medium)? bands = null)

A model that ranks how likely something is: how well it ranks the test rows (AUC), against the simple rule's ranking when there is one; what its highest and lowest tenths hold.

• score: The model's scores of the test rows.
• outcome: What happened (1 or 0, true or false).
• rule: The simple rule's scores of the same rows (higher: more likely), when there is one.
• ruleName: The rule in words ("the longest since the last activity").
• bands: The chances its bands are cut at (High from, Medium from), when they are: a rare outcome (under a tenth of the rows) puts nearly every row in Low at 50% and 20%, which the verdict says, with the cut-offs that band it.

static string? RareBands(double rate, double high, double medium, int highRows, int rows)

The advice when an outcome is rare for bands cut at chances: under a tenth of the rows had it, so at High's cut-off few or none are High. The cut-offs that band it: three and one and a half times the usual rate (null: not rare, or already set lower).

static Verdict Survival(double[] risk, double[] time, double[] evt, double horizon, double[]? rule = null, string? ruleName = null, string? unit = null)

A survival model (how long until an event): how well it orders the test rows by when they have it (Harrell's concordance), against Cox's proportional hazards (or another simple model) when given; how many of its riskiest and safest tenths had the event within the horizon (Kaplan-Meier's, so those still going count).

• risk: The model's risks of the test rows (higher: sooner).
• time: Their durations, to the event or to now.
• evt: 1 when it happened, 0 while still going.
• horizon: The time the tenths are read at, in the durations' units.
• rule: The simple model's risks of the same rows (Survival.Cox(...).Predict(test)).
• ruleName: It in words ("Cox's proportional hazards").
• unit: The durations' unit, in words ("months").

static Verdict Amount(double[] predicted, double[] actual, double[]? rule = null, string? ruleName = null)

A model of an amount: how far off on the test rows (the share of the actual total), against the simple rule's when there is one.

• predicted: The model's predictions of the test rows.
• actual: What happened.
• rule: The simple rule's predictions of the same rows, when there is one (missing counts as 0).
• ruleName: The rule in words ("the same as in the past period").

static Verdict Forecast(double off, double ruleOff, double inRange, int heldBack, ForecastHistory history, double totalOff = double.NaN, double totalRuleOff = double.NaN)

A forecast judged: how far off it was over the periods held back, compared with the simple rule (the value a season earlier, or else the latest), and how much of what happened fell inside its likely range. With several series it also judges the total (all series added up per period, Metrics.WapeOfTotals), which is what a plan for the total sees, beside the error per series (the level being the per-series one, as in the engine).

Checks

public static class Checks

What to check before trusting a model: the engine's checks, as warnings a verdict carries.

Methods

static IReadOnlyList<string> Leakage(Table rows, string label, IEnumerable<string> features)

Leakage: one input that predicts the outcome almost perfectly on its own (an AUC above 0.97 for an outcome that happens or not, a rank correlation above 0.97 for an amount) is usually recorded after the outcome, not known before it: a status, a closing date.

• rows: The rows learned from.
• label: The outcome column.
• features: The inputs.

Calibration

public static class Calibration

Calibration of a model's scores (LightGBM's binary scores, or any probability): learn a map on a calibration slice (Platt or isotonic, as scikit-learn's CalibratedClassifierCV), apply it to the test rows, and judge it with Metrics.Brier, Metrics.CalibrationError and Chart.Reliability. A version keeps it when saved with it (Models.Save(…, calibrator: c)): PREDICT then gives each row's calibrated probability, as c.Apply gives it, and bands that. Without it a version scores the model's own probabilities, uncalibrated (its bands say so). Learn the calibrator on the scores of the very booster saved: one learned on another model's scores maps them wrongly.

Methods

static Calibrator Platt(double[] score, double[] outcome, bool smooth = true)

Platt's scaling: a logistic regression of the outcome on the scores' log-odds (Newton's method on the log loss, with Platt's targets (positives + 1) / (positives + 2) and 1 / (negatives + 2) against overconfidence when smooth).

static Calibrator Isotonic(double[] score, double[] outcome)

Isotonic regression (pool adjacent violators, as scikit-learn's IsotonicRegression with out_of_bounds="clip"): steps that only rise with the score, each its rows' rate; between two steps a straight line. Needs more rows than Platt (a thousand or more) and follows any shape.

static IReadOnlyList<CalibrationBin> Curve(double[] score, double[] outcome, int bins = 10, bool quantile = false)

The reliability curve: the rows in bins of equal width by score (0-0.1, 0.1-0.2 …; quantile: bins of equal rows instead), each with its mean score and how often its rows had the outcome. Calibrated, the two are equal.

Calibrator

public sealed class Calibrator

A map from a model's scores to probabilities that hold up: of the rows it gives 0.3, about 30 % have the outcome. Learned on rows the model did not learn from (a calibration slice), checked on others (the test rows).

Properties

string Kind { get; }

Isotonic (a staircase that never goes down) or Platt (a logistic curve on the log-odds of the score: sigmoid(A × logit(score) + B)).

double A { get; }

Platt's slope on the score's log-odds (1 and B 0: already calibrated; below 1, the model is too sure of itself).

double B { get; }

Platt's intercept.

int Rows { get; }

How many rows it was learned from.

IReadOnlyList<(double Score, double Probability)> Steps { get; }

Isotonic: the scores where its steps are and the probability at each (in between, a straight line; outside, the end's).

Methods

double Apply(double score)

A score's probability.

double[] Apply(double[] scores)

Each score's probability.

JsonObject ToJson()

What it keeps, for a version's evidence: its kind, coefficients or steps.

Also: string ToString().

CalibrationBin

public sealed record CalibrationBin(
    int Bin,
    double From,
    double To,
    int Rows,
    double MeanScore,
    double Rate)

A tenth (or bin) of a reliability curve: its scores' range and mean, its rows and how often they had the outcome.

Properties

Its properties: int Bin { get; init; }, double From { get; init; }, double To { get; init; }, int Rows { get; init; }, double MeanScore { get; init; }, double Rate { get; init; }.

Drift

public static class Drift

Drift: how far the rows a model scores now have moved from the rows it learned from. PSI (the population stability index over the reference's tenths: below 0.1 stable, 0.1 to 0.25 worth watching, above 0.25 shifted) and the KS test, per column (Compare) or for the scores. The reference is a version's training rows (Models.ReadParquet(version.PathOf ("training.parquet"))) or an earlier month's; the current rows are the features at today's date.

Methods

static Metric Psi(double[] reference, double[] current, int bins = 10)

PSI of a number over the reference's quantile bins (missing values a bin of their own).

static Metric Psi(string?[] reference, string?[] current)

PSI of a category (text): each value its own bin, missing one more.

static string Words(double psi)

PSI in words: stable, worth watching, shifted.

static KsTest Ks(double[] reference, double[] current)

The two-sample Kolmogorov-Smirnov test (scipy.stats.ks_2samp's statistic; the p-value from Kolmogorov's limit).

static Table Compare(Table reference, Table current, IEnumerable<string>? columns = null, int bins = 10)

Every column both tables have (or those given), the most shifted first: its PSI and verdict, KS for numbers, the share missing and the mean before and now. A table to show, keep as evidence, or watch month by month.

Backtest

public sealed class Backtest

A walk-forward backtest (Backtest.WalkForward): per test period, out of sample, how well a model learned only from what came before ranked (AUC, lift) and how true its probabilities were (calibration), against a simple rule; and a summary over the periods. Its Backtest.Scores are every test row with its out-of-sample score, for Bootstrap.Auc and charts.

Properties

IReadOnlyList<BacktestFold> Folds { get; }

The test periods, earliest first.

Table Scores { get; }

Every test row of every fold, as it was (all its columns), with backtest_fold, backtest_period and backtest_score (the score of the model that fold learned): Bootstrap.Auc(bt.Scores, "backtest_score", outcome, cluster: "…") gives the out-of-sample AUC's interval, Chart.Reliability(bt.Scores["backtest_score"], bt.Scores[outcome]) its reliability.

string Date { get; }

The date column the rows are ordered by.

string Outcome { get; }

The outcome column.

string Every { get; }

What a test period is: "period" (each value of the date column), "month", "quarter" or "year".

int Gap { get; }

How many of the date column's values lie between the last rows learned from and the first tested.

int? GapMonths { get; }

The gap in calendar months instead (a date column): each fold learned from rows dated before its first test date less these months.

int? Window { get; }

How many of the date column's values each fold learns from (null: all before it, an expanding window).

string? Rule { get; }

The simple rule's column, when there is one.

string? RuleName { get; }

The rule in words.

double Top { get; }

The top share lift is measured on (0.1: the tenth with the highest scores).

int Bins { get; }

The calibration error's bins.

bool Probabilities { get; }

Whether the scores are probabilities (all between 0 and 1): only then are Brier, calibration error and slope measured.

Table Summary { get; }

The folds' metrics summed up: per metric (auc, rule_auc, lift, rule_lift, brier, calibration_error, calibration_slope, mean_score, rate) its mean, standard deviation (n - 1), lowest and highest over the folds, and its value on all the test rows together (all_folds: a different question when the periods' rates differ).

BacktestFold AllFolds { get; }

The metrics on every test row together (all folds' scores, each from its own fold's model).

Methods

static Backtest WalkForward(Table rows, string date, string outcome, Func<Table, Booster> learn, string every = "period", object? from = null, int folds = 5, int gap = 0, int? gapMonths = null, int? window = null, string? rule = null, string? ruleName = null, double top = 0.1, int bins = 10)

A walk-forward backtest of a model of a 0/1 outcome: the rows' periods are put in time order; for each test period, the model learns (learn) from the rows before it only, less a gap, and scores that period's rows once; then it rolls forward one period. Per period: AUC, lift of the top tenth, calibration (Brier, calibration error, slope; for probabilities), against a simple rule when given; a summary; and every test row's out-of-sample score. Rows whose date or outcome is missing are left out (an outcome not known yet).

LightGbm.TimeFolds is for tuning (folds that learn and check inside the training months); this is the report of how the chosen recipe would have done in each period, had it been run then.

• rows: The rows: one per entity and date (snapshots), with the outcome.
• date: The column that orders them in time (a date, or a number: a period).
• outcome: The outcome column: 1 or 0 (true or false).
• learn: Learns a model from a fold's rows: train => LightGbm.Train(LightGbm.Dataset(train, …), null, parameters, rounds: …).
• every: What a test period is: "period" (each value of the date column, the default), or for a date column "month", "quarter" or "year" (a calendar year: a per-year report).
• from: The first test period: a date (new DateTime(2016, 1, 1)) or number in it. Without it, the latest folds periods.
• folds: How many of the latest periods are tested when from is not given.
• gap: How many of the date column's own values (its periods: months for monthly snapshots) to leave out between the last rows learned from and the first tested, as LightGbm.TimeFolds' gap: at least the outcome's window (6 for an outcome 6 months ahead of monthly snapshots), or a fold learns from outcomes that reach into its test period. A value missing from the data widens it by one (gapMonths does not).
• gapMonths: The gap in calendar months instead, for a date column: a fold learns from the rows dated before its first test date less this many months (an outcome six months ahead: 6). The same as gap on monthly data without holes; a month missing from the data then moves nothing.
• window: Learn from only the latest this many values of the date column before the gap (a rolling window); null learns from all before it (an expanding window).
• rule: A column of the simple rule's scores (higher: more likely; "the furthest below the cut-off"), measured beside the model in every period.
• ruleName: The rule in words.
• top: The share of rows with the highest scores whose lift is measured (0.1: the top tenth, rounded up; ties at the cut in row order).
• bins: The calibration error's bins (of equal width).

static Backtest WalkForward(Table rows, string date, string outcome, Func<Table, Func<Table, double[]>> learn, string every = "period", object? from = null, int folds = 5, int gap = 0, int? gapMonths = null, int? window = null, string? rule = null, string? ruleName = null, double top = 0.1, int bins = 10)

A walk-forward backtest of any model: learn gives, from a fold's rows, a function that scores rows (train => test => scores; the model may be a rule, a regression, a model of the notebook's own). The rest as the LightGBM one.

• rows: The rows: one per entity and date, with the outcome.
• date: The column that orders them in time.
• outcome: The outcome column: 1 or 0.
• learn: From a fold's rows, a function that scores rows (one score per row, higher: more likely).
• every: "period", "month", "quarter" or "year".
• from: The first test period (a date or number in it); without it the latest folds.
• folds: How many of the latest periods are tested when from is not given.
• gap: How many of the date column's values between learning and testing (the outcome's window).
• gapMonths: The gap in calendar months instead (a date column).
• window: Learn from only the latest this many values (rolling); null: all before (expanding).
• rule: A column of the simple rule's scores.
• ruleName: The rule in words.
• top: The top share for lift (0.1).
• bins: The calibration error's bins.

Table ToTable()

The folds as a table: fold, period, what each learned from and was tested on, and its metrics.

ChartSpec Chart(string metric = "auc")

A metric per test period as a line ("auc", "lift", "brier", "calibration_error", "calibration_slope", "rate"), with the simple rule's beside AUC and lift.

Also: string ToString().

BacktestFold

public sealed class BacktestFold

One test period of a walk-forward backtest: what its model learned from, what it was tested on, and how it did there.

Constructors

Its constructors: BacktestFold().

Properties

int Fold { get; init; }

Its number, 1 for the earliest test period.

string Period { get; init; }

The test period: "2016" (every: "year"), "2016 Q1", "2016-01", or the date column's value (every: "period").

string LearnedFrom { get; init; }

The earliest value of the date column learned from.

string LearnedUntil { get; init; }

The latest value of the date column learned from: before the test period, less the gap.

string TestFrom { get; init; }

The test period's first value of the date column.

string TestUntil { get; init; }

The test period's last value of the date column.

int LearnedRows { get; init; }

How many rows its model learned from.

int TestRows { get; init; }

How many rows it was tested on (scored once, out of sample).

double Rate { get; init; }

The share of the test rows with the outcome.

double MeanScore { get; init; }

The mean score of the test rows: with Rate, calibration in the large (the two are equal when calibrated).

double Auc { get; init; }

The model's AUC on the test rows (a missing score ranks last; NaN when the period's rows all had, or all lacked, the outcome).

double RuleAuc { get; init; }

The simple rule's AUC on the same rows (NaN without a rule).

double Lift { get; init; }

The lift of the model's top share (top, a tenth by default): how many times the period's rate its riskiest rows had.

double RuleLift { get; init; }

The simple rule's lift of its top share.

double Brier { get; init; }

The Brier score of the model's probabilities (NaN when the scores are not probabilities).

double CalibrationError { get; init; }

The expected calibration error (Metrics.CalibrationError, bins of equal width; NaN when the scores are not probabilities).

double CalibrationSlope { get; init; }

The calibration slope: the logistic regression of the outcome on the probabilities' log-odds (R: glm(y ~ qlogis(p), binomial)); 1 is calibrated, below 1 too extreme (overfitted), above 1 too timid. NaN when the scores are not probabilities or the regression does not converge.

Methods

Its methods: string ToString().

Uplift

public static class Uplift

Causal effects, from an experiment's rows (a randomized treatment) or, with care, from observational ones: the average effect, who the treatment moves (uplift: Uplift.TwoModel, judged with Qini), and an effect adjusted for who was treated (inverse propensity weighting). The outcome's direction is as it is: for churn (1 = left), a good offer has a negative uplift; use 1 - churned (stayed) to have it positive.

Methods

static Effect Ate(double[] treatment, double[] outcome, double confidence = 0.95)

The average effect of the treatment (1/0 or true/false per row) on the outcome: treated against untreated, with its interval.

static UpliftModel TwoModel(Table train, string treatment, string outcome, IEnumerable<string> features, Dictionary<string, object> parameters, Table? valid = null, int rounds = 1000, int earlyStopping = 50, int bootstraps = 0, int seed = 1)

Learns an uplift model: LightGBM on the treated rows and on the control rows (parameters as Train's; with valid, each stops early on its arm of the validation rows). bootstraps > 0 learns that many more pairs on resampled training rows (with the main models' trees), for UpliftModel.Interval.

static QiniCurve Qini(double[] uplift, double[] treatment, double[] outcome, int points = 50)

The Qini curve of an uplift ranking on rows of a randomized experiment: targeting the top share, the treated rows' outcomes less the control rows' scaled to as many (Radcliffe). The coefficient is the area between it and treating at random, per row: above 0 the ranking finds those the treatment moves. Pass outcomes where more is better (stayed, bought).

static Effect Ipw(double[] treatment, double[] outcome, double[] propensity, double clip = 0.01, double confidence = 0.95)

The average effect from observational rows, weighting each by the inverse of its chance of being treated as it was (Hájek's normalized weights; propensities clipped to [clip, 1 - clip]). The propensity is a model's (LightGBM on the treatment) from what decided who was treated. Holds only if nothing unmeasured decided both who was treated and the outcome, and every kind of row could have been either; an experiment needs none of that.

UpliftModel

public sealed class UpliftModel

An uplift model (the two-model or T-learner): one LightGBM learns the outcome of the treated rows, another of the control rows; a row's uplift is the difference of their scores (for churn: how much the offer changes its chance of leaving). With bootstraps, each row also has an interval from models learned on resampled rows.

Properties

Booster Treated { get; }

The model of the treated rows' outcome.

Booster Control { get; }

The model of the control rows' outcome.

bool Binary { get; }

A 0/1 outcome: the uplift is a change of probability.

int Bootstraps { get; }

How many bootstrap pairs give the intervals.

Also: string Treatment { get; }, string Outcome { get; }.

Methods

double[] Predict(Table rows)

Each row's uplift: the outcome treated less the outcome not treated (for a 0/1 outcome, a change of probability).

(double[] Low, double[] High, double[] Deviation) Interval(Table rows, double confidence = 0.9)

Each row's uplift interval (confidence 0.9: the 5th to 95th percentile of the bootstrap pairs' uplift) and their standard deviation, for a robust plan (Decisions.SumAtWorst with the deviation).

Also: string ToString().

QiniCurve

public sealed record QiniCurve(
    IReadOnlyList<double> Share,
    IReadOnlyList<double> Gain,
    double Coefficient,
    double Total)

A Qini curve: by the share of rows targeted (highest uplift first), the extra outcomes the targeting brings against treating at random.

Properties

Its properties: IReadOnlyList<double> Share { get; init; }, IReadOnlyList<double> Gain { get; init; }, double Coefficient { get; init; }, double Total { get; init; }.

Policy

public static class Policy

A targeting policy judged before it is rolled out (off-policy evaluation): on rows where the treatment was random (an experiment), the rows that happened to get what the policy would give stand for everyone, weighted by the chance of that (inverse propensity, Hájek). Compare policies on the same held-out rows: treat no one, everyone, the top by uplift, an optimization plan's choices.

Methods

static PolicyValue Evaluate(double[] decision, double[] treatment, double[] outcome, double[]? propensity = null, string name = "policy")

The policy's value: decision 1/0 per row (treat or not), treatment what each row got, outcome what happened (more is better: stayed, margin), propensity each row's chance of being treated (the experiment's share when not given).

static PolicyValue Evaluate(IReadOnlyList<string> chosen, IReadOnlyList<string> taken, double[] reward, double[] propensity, IReadOnlyDictionary<string, double[]>? predicted = null, string name = "policy", string none = "control")

A rule that chooses among several actions (offers), judged on a randomized test of several arms: chosen the rule's action for each row, taken the arm each row got, reward what came of it (more is better: the value kept less the offer's cost), propensity each row's chance of the arm it got (its stratum's share of that arm). Without predictions, inverse propensity weighting (Hájek's, as Evaluate's); with predicted[action][row] - outcome models' expected reward of each action, learned on other rows (cross-fitted) - doubly robust: the model's value of the rule's action for every row, corrected on the rows that got it by their error over their chance; right when either the chances or the models are, and with good models a narrower interval. none names the action that is no treatment (ShareTreated counts the others).

static PolicyValue Difference(IReadOnlyList<string> chosen, IReadOnlyList<string> baseline, IReadOnlyList<string> taken, double[] reward, double[] propensity, IReadOnlyDictionary<string, double[]>? predicted = null, string name = "gain", string none = "control")

What one rule gains over another on the same rows (the rule less the baseline: offering nothing, the current rule), paired row by row - a far narrower interval than the two values' own. Each row's term is doubly robust with predictions (Evaluate's), else inverse propensity weighted (Horvitz-Thompson: the reward over its chance where the row got the rule's action).

static Table Compare(double[] treatment, double[] outcome, params (string Name, double[] Decision)[] policies)

Several policies on the same rows, as a table (name, value, interval, share treated), the best first.

PolicyValue

public sealed record PolicyValue(
    string Name,
    double Value,
    double StandardError,
    double Low,
    double High,
    int Rows,
    int Followed,
    double ShareTreated)

A targeting policy's estimated value: the mean outcome per row had it been followed, with its interval.

Properties

Its properties: string Name { get; init; }, double Value { get; init; }, double StandardError { get; init; }, double Low { get; init; }, double High { get; init; }, int Rows { get; init; }, int Followed { get; init; }, double ShareTreated { get; init; }.

Forecast

public static class Forecast

Forecasts made the way OpenBI's engine makes them (the recipe "openbi-series/1"): a measure per period, for the total or per series. A single model learns from all the series together, each one scaled to its own recent level; it learns from the most recent values, their trend and averages, the value a season earlier, the time of year and the number of periods ahead. A forecast's model version keeps the OAX that reads its periods, and the engine scores it with the same recipe, so the features in the notebook must be exactly these.

Fields

Its fields: const string Recipe = "openbi-series/1".

Properties

static IReadOnlyList<string> Features { get; }

The recipe's inputs, in the models' order.

Methods

static int Season(string grain)

Periods in a season: 12 months, 4 quarters, 52 weeks, 7 days; none for years.

static int MaxPeriods(string grain)

The history the recipe reads at most: 10 years of months, 5 of weeks, 3 of days.

static double TimeOfYear(string grain, DateTime d)

Where a period falls in the year: its month, its quarter, its week of the year, or its weekday.

static (IReadOnlyList<DateTime> Starts, DateTime Next) Periods(DateTime first, DateTime last, string grain, DateTime? today = null)

The periods that are complete, from the first date with data to the last, at most the latest MaxPeriods of them; the first period after the data is the first one to forecast (Next). A period counts as complete once the data reaches its last day, or, when each date is its period's own first day (as 'Date'[Month_Start]), once the period is over (today stands for this day). The engine counts them the same way.

static ForecastHistory History(Table rows, string? series, string period, string value, IReadOnlyList<DateTime> starts, DateTime next, string grain, IReadOnlyList<string>? known = null, string? calendar = null, IReadOnlyDictionary<string, Table>? regressors = null)

The history of every series: the values of rows of (series, period, value), one row per series and period. A series starts at its first value; after it, a period without one is a period of nothing (0). Series are numbered in the order they first appear (or as known lists them; others are left out).

• calendar: A country's calendar as inputs ("no": Norway's working days and holidays, ForecastCalendar).
• regressors: Outside series as inputs, by name: each a table of one value per period (its columns period and value, as a period query's runs give): its latest value at the origin and its value a season before the period forecast, both known when forecasting.

static Dictionary<string, object> Parameters(double alpha, int rows)

The engine's LightGBM parameters for a quantile model (alpha 0.1, 0.5 or 0.9) learned from this many rows.

Also: static DateTime Start(DateTime d, string grain), static DateTime Next(DateTime d, string grain).

ForecastHistory

public sealed class ForecastHistory

Every series' values per period (NaN before a series starts), and the recipe's features over them.

Properties

string? Calendar { get; }

The country whose calendar is one of the inputs ("no" for Norway), or null.

IReadOnlyList<string> Regressors { get; }

The outside series among the inputs, in order.

IReadOnlyList<string> Features { get; }

The inputs, in the models' order: the recipe's (Forecast.Features), the calendar's, then each outside series' latest and a season before.

DateTime Next { get; }

The first period not complete: the first to forecast.

int Last { get; }

The last complete period.

Also: string Grain { get; }, IReadOnlyList<string> Series { get; }, IReadOnlyList<DateTime> Starts { get; }, int Periods { get; }, int Season { get; }.

Methods

double Regressor(int r, int p)

An outside series' value at period p (NaN when it has none).

double Scale(int s, int t)

A series' level at an origin: the mean size of its last three periods (1 when there is none).

DateTime TargetStart(int t, int k)

The date of the period k ahead of origin t.

double[] FeaturesAt(int s, int t, int k)

The inputs for forecasting k periods ahead of origin t (the last known period): the recipe's 14, then the calendar's and the outside series'.

double Rule(int s, int t, int k)

The simple rule to beat: the value of the same period one season earlier, or else the latest value.

Table Rows(int horizon, int heldBack)

The rows to learn from: every series, origin and step ahead (1 to horizon) whose value is known, with the target scaled to the series' level. "training": its target is no later than origin Last − heldBack; "validation": its origin is that one (what the held-back periods are checked on).

Table Test(int heldBack)

The held-back periods: forecast from origin Last − heldBack, with what happened and the simple rule.

Table Ahead(int horizon)

The periods to forecast: 1 to horizon ahead of the last complete period, for every series.

Table Past()

Every past period forecast one ahead from the one before it, with what happened: what the forecast expected, so periods well outside it stand out (history.Score marks them as anomalies).

Table Score(Table rows, Booster p10, Booster p50, Booster p90)

Forecasts for rows (Ahead, Test or Past) from the three models, in the series' own units (the level multiplied back in): low, forecast (the median) and high, in that order (quantile models may cross). Where the outcome is known (Test, Past), "anomaly" flags a value far outside the likely range, more than half the range's width beyond it (by chance about one period in fifty), that also misses by over a tenth of the level; the engine flags the same.

string Label(DateTime d)

A period as a label: 2024-03 for a month, 2024 Q1, 2024, or the day.

Table Totals()

The totals of every series per period (for a chart).

Also: double Y(int s, int p), string ToString().

ForecastCalendar

public static class ForecastCalendar

A country's calendar as inputs of a forecast (Forecast.History's calendar: "no"): for each period forecast, its working days, their change on the same period a year before, its Fridays and Saturdays and whether Easter falls in it - whole numbers, as the engine counts them, so a version's forecasts are the notebook's. Norway ("no"): weekdays less its public holidays (New Year's Day, Maundy Thursday, Good Friday, Easter Monday, 1 May, 17 May, Ascension Day, Whit Monday, Christmas Day and Boxing Day), Easter's moving with the year.

Properties

static IReadOnlyList<string> Features { get; }

The inputs a calendar adds, in the models' order.

Also: static IReadOnlyList<string> Countries { get; }.

Methods

static DateTime EasterSunday(int y)

Easter Sunday of a year (the Gregorian computus, as the engine reckons it).

static bool IsHoliday(DateTime day, string country = "no")

A public holiday of the country (Norway's).

static int WorkingDays(DateTime from, DateTime until, string country = "no")

The working days in [from, until): Monday to Friday, less public holidays.

static int FridaysSaturdays(DateTime from, DateTime until)

The Fridays and Saturdays in [from, until).

static DateTime YearBefore(DateTime start, string grain)

The same period a year before: 12 months back, or 52 weeks for weeks and days (the same weekday).

static double[] At(string country, string grain, DateTime start)

The calendar's inputs for the period that starts at start: working days, their change on a year before, Fridays and Saturdays, Easter (1 or 0).

Also: static string Check(string country).

Last updated 6 November 2026 · OpenBI Studio 0.3.78