Learning

LightGBM from C#, with what keeps a model honest built around it: splits by time, a held-out test, a simple rule to beat, a verdict in words, and checks for calibration and drift. The examples learn on made-up customers whose truth is known.

Honest by design

A model that looks good in a notebook and fails in use usually saw the future while it learned. The library is built so it doesn’t:

• What was known at a date. An OAX step with an @asOf parameter gives each customer’s features as they stood on a date; run again at earlier dates, it gives the history to learn from (see Parameters).
• Three sets in time order. Learn on the earliest rows, stop early on the next, and measure once on the latest, rows neither learning nor stopping saw.
• A simple rule to beat. The verdict says whether the model is strong, useful, weak or no better than a rule anyone could apply, and says so plainly.

The data here is Synthetic.Retention, a subscription service’s customers month by month. In your own notebook, an OAX step of your model’s rows in the same shape takes its place.

// a subscription service's customers, month by month: what was known at each month's start, and whether they left in
// the 30 days after. Made up, so that what a method finds can be checked against the truth
var rows   = Synthetic.Retention(customers: 3000, months: 12, seed: 7);
var inputs = new[] { "tenure_months", "monthly_spend", "days_since_last", "support_tickets", "plan", "region" };
// by time: learn from the earlier months, stop early on the next two, test on the last three, once
var train = rows.Where(r => r.Date("as_of") <  new DateTime(2024, 8, 1));
var valid = rows.Where(r => r.Date("as_of") >= new DateTime(2024, 8, 1) && r.Date("as_of") < new DateTime(2024, 10, 1));
var test  = rows.Where(r => r.Date("as_of") >= new DateTime(2024, 10, 1));
$"learn {train.Count:N0}, check {valid.Count:N0}, test {test.Count:N0} rows"

learn 16,539, check 3,386, test 4,405 rows

Train

A Dataset is the rows LightGBM learns from: a table, its label and its features (text columns become categories). LightGbm.Train learns until the checking rows stop improving:

var data  = LightGbm.Dataset(train, "churned", inputs);
var check = LightGbm.Dataset(valid, "churned", inputs, reference: data);
var booster = LightGbm.Train(data, check,
    new() { ["objective"] = "binary", ["metric"] = "auc", ["learning_rate"] = 0.05, ["num_leaves"] = 15,
            ["num_threads"] = 1, ["deterministic"] = true, ["seed"] = 1 },
    rounds: 1000, earlyStopping: 50);
booster

LightGBM binary: 67 trees, 6 features, auc 0.7163 on the validation rows

The parameters are LightGBM’s own, by their names. num_threads, deterministic and seed make the same data give the same model on every run, as these pages need; leave them out for speed.

Chart.Loss(booster)

Judge it

The test months are scored once, at the end, against a simple rule:

// the test months, scored once, against a simple rule: whoever has gone longest without using it leaves first
var p = booster.Predict(test);
Display(Metrics.Auc(p, test["churned"]),
        Verdict.Ranking(p, test["churned"], rule: test["days_since_last"], ruleName: "most days since the last use"));

AUC 0.702

No better than the simple rule (most days since the last use): AUC 0.70 against 0.69. Use the rule, or add measures.

Here the honest answer is that the model barely beats the rule. That is what the verdict is for: use the rule, or bring features that tell more.

Explain it

Chart.Shap(booster, test, top: 6)

booster.Reasons(rows) gives each row the features that pushed its score up most, in words; Chart.ShapDependence and Chart.ShapWaterfall draw one feature, or one row.

Calibrate

A probability of 0.3 should come true about 3 times in 10. Calibration.Isotonic and Calibration.Platt learn a map from scores to probabilities on rows the model didn’t learn from:

// probabilities that hold up: learned on the checking months, judged on the test months
var iso = Calibration.Isotonic(booster.Predict(valid), valid["churned"]);
Display(iso, Metrics.CalibrationError(p, test["churned"]),
        Metrics.CalibrationError(iso.Apply(p), test["churned"]) with { Name = "calibration error, calibrated" });

isotonic calibration from 3,386 rows: 38 steps, from 0 to 0.3529

calibration error 0.004 · mean distance between the probability given and how often it happened

calibration error, calibrated 0.008 · mean distance between the probability given and how often it happened

This model’s probabilities were close already, and isotonic, learned on fewer than 3,400 rows, makes them a little worse on the test months. Calibrate when the check says the scores are off, and judge the result on rows it didn’t see; Chart.Reliability draws both.

Watch for drift

A model scores rows like the ones it learned from. Drift.Compare says how far each input has moved, by PSI and a two-sample test:

// how far the test months' rows moved from the rows it learned from
Drift.Compare(train, test, inputs)

Tenure grows by a month every month, so it always drifts between an old snapshot and a new one: judge such a column by what the model does with it.

More

Folds and tuning

LightGbm.TimeFolds (in time order, with a gap) and LightGbm.GroupFolds (each customer’s rows on one side), LightGbm.CrossValidate and LightGbm.Tune for a search over parameters.

Constraints and objectives

LightGbm.Monotone and LightGbm.Interactions, weights with Dataset.Weights, your own loss with LightGbm.TrainCustom, ranking and several classes through LightGBM’s objectives.

Backtests

Backtest.WalkForward: a model per test period, each learned only from the periods before it.

Uplift and targeting

Uplift.TwoModel for who an offer changes, Chart.Qini, and Policy to judge a targeting rule before it is rolled out.

Forecasts

Forecast learns from every series at once, as the engine’s own forecasts do, with a likely range; Verdict.Forecast judges it.

Survival

LightGbm.TrainSurvival, a boosted Cox model (see Statistics and survival).

Each has a recipe that runs in the analytics catalog (see The analytics catalog) and every member is in the Learning reference.

Save the model

A prediction notebook ends with Models.Save: the booster, the OAX query of features it scores with, the key of who is scored, and the evidence (metrics, verdict, charts). The version is a draft until a person approves it; once it is in use, PREDICT("name") scores with it in OAX, at every refresh (see Prediction and Versions and approval). A LightGBM model made elsewhere comes in with Import a model…, read and never run.

Last updated 6 November 2026 · OpenBI Studio 0.3.78