SURVIVAL.COX.PENALIZED OpenBI
A penalized Cox model (the lasso and elastic net, as glmnet).
Group: Survival analysis. OpenBI’s own: DAX has no function of this name.
Syntax
SURVIVAL.COX.PENALIZED ( <table>, <duration>, <event> [, <options>] [, <entry>], <name>, <covariate> [, <name>, <covariate>] … )| Parameter | Type | Description |
|---|---|---|
table | Table | The rows to follow, one a subject such as a customer, a policy or a machine: a table of the model or a table expression, as the filters leave it. |
duration | Number evaluated for each row | How long each row was followed: until its event, or until we stopped seeing it. In any unit (days, months, years), the same for every row. |
event | Number evaluated for each row | 1 or TRUE when the row's duration ended in the event (the customer left); 0 or FALSE when it did not, so all we know is that it lasted at least that long (censored). |
options | Text optional | Settings in one text, name=value with commas between: alpha (1, the lasso; 0 ridge; between, the elastic net), folds (10), seed (1), lambda (1se, min or a number), nlambda (100) and ratio. Left out, the defaults. |
entry | Number optional, evaluated for each row | When each row came under observation: it is at risk from just after its entry to the end of its duration. An expression, not a text, given right before the first name. |
name | Text repeatable | A covariate's name, in quotes: the result calls it so. |
covariate | Expression repeatable, evaluated for each row | The covariate's value on each row: a number, or a text for a factor whose levels are each compared with the first. |
Returns
Table. A row a term. The columns: Variable and Reference; Coefficient and Hazard ratio; Selected; Lambda, Lambda min and Lambda 1se; Alpha; CV deviance and CV std error; Folds; Observations and Events; and a Note.
Remarks
- The fit is the elastic net for Cox's model, as R's glmnet computes it (Efron's handling of ties): alpha=1 is the lasso, which can drop terms altogether; alpha=0 is ridge, which only shrinks them.
- The terms are put on a common scale before the penalty, as glmnet does, and the coefficients come back on the covariates' own scale.
- The path runs over nlambda values of lambda (100 by default, from 5 to 500), from the smallest that keeps every term at 0 down to ratio times it (0.0001, or 0.01 when there are fewer rows than terms).
- Lambda is chosen by cross-validation: the rows are split into folds (10 by default, from 3 to 50; at least twice as many rows as folds), shuffled with the seed (1 by default; seed=0 keeps the rows' order), and each fold is scored by its deviance. Another seed can choose another lambda.
- lambda=1se (the default) takes the largest lambda whose deviance is within a standard error of the best: the simplest model as good as the best. lambda=min takes the best; a number takes that lambda, and lambda=0 is the plain Cox fit.
- Selected is TRUE for a term whose coefficient is not 0. Note sums up how many terms were kept, and at which lambda.
- A covariate of text is a factor: it gets a term for each level but the first in sorted order, the reference, and is named like
Plan: Plus. - A row is fitted when its duration, its event and every covariate (and the entry, when given) have a value; rows with a blank are left out.
Compared with DAX
An OpenBI function: DAX has none like it, so a measure or query that uses it works in OpenBI only, not in Power BI or Analysis Services.
Examples
The lasso picks its terms
With the defaults (ten folds, the one-standard-error lambda), all three terms stay, shrunk.
EVALUATE
VAR Customers =
DATATABLE (
"Plan", STRING, "Months", INTEGER, "Left", INTEGER, "Spend", INTEGER, "Tickets", INTEGER,
{
{ "Basic", 3, 1, 20, 4 }, { "Basic", 5, 1, 25, 3 }, { "Basic", 6, 0, 40, 1 }, { "Basic", 8, 1, 30, 5 },
{ "Plus", 9, 1, 35, 2 }, { "Basic", 10, 0, 45, 0 }, { "Basic", 12, 1, 28, 3 }, { "Plus", 14, 0, 60, 1 },
{ "Plus", 15, 1, 38, 4 }, { "Basic", 16, 0, 50, 1 }, { "Plus", 18, 1, 42, 2 }, { "Plus", 20, 0, 55, 0 },
{ "Basic", 22, 1, 33, 2 }, { "Plus", 24, 0, 65, 1 }, { "Plus", 25, 1, 48, 3 }, { "Basic", 27, 0, 36, 0 },
{ "Plus", 30, 1, 52, 1 }, { "Plus", 32, 0, 70, 0 }, { "Basic", 34, 0, 44, 2 }, { "Plus", 36, 0, 58, 1 },
{ "Basic", 4, 1, 22, 2 }, { "Plus", 7, 1, 30, 3 }, { "Basic", 13, 1, 26, 1 }, { "Plus", 28, 0, 62, 2 }
}
)
RETURN
SELECTCOLUMNS (
SURVIVAL.COX.PENALIZED ( Customers, [Months], [Left], "Plan", [Plan], "Spend", [Spend], "Tickets", [Tickets] ),
"Variable", [Variable],
"Coefficient", [Coefficient],
"Hazard ratio", [Hazard ratio],
"Selected", [Selected],
"Lambda", [Lambda]
)| Variable | Coefficient | Hazard ratio | Selected | Lambda |
|---|---|---|---|---|
| Plan: Plus | 1.18988076291311 | 3.28668928876166 | TRUE | 0.0691503218998539 |
| Spend | -0.195018609145882 | 0.822819345948303 | TRUE | 0.0691503218998539 |
| Tickets | 0.392895090572833 | 1.4812629827266 | TRUE | 0.0691503218998539 |
Four folds, the best lambda
EVALUATE
VAR Customers =
DATATABLE (
"Plan", STRING, "Months", INTEGER, "Left", INTEGER, "Spend", INTEGER, "Tickets", INTEGER,
{
{ "Basic", 3, 1, 20, 4 }, { "Basic", 5, 1, 25, 3 }, { "Basic", 6, 0, 40, 1 }, { "Basic", 8, 1, 30, 5 },
{ "Plus", 9, 1, 35, 2 }, { "Basic", 10, 0, 45, 0 }, { "Basic", 12, 1, 28, 3 }, { "Plus", 14, 0, 60, 1 },
{ "Plus", 15, 1, 38, 4 }, { "Basic", 16, 0, 50, 1 }, { "Plus", 18, 1, 42, 2 }, { "Plus", 20, 0, 55, 0 },
{ "Basic", 22, 1, 33, 2 }, { "Plus", 24, 0, 65, 1 }, { "Plus", 25, 1, 48, 3 }, { "Basic", 27, 0, 36, 0 },
{ "Plus", 30, 1, 52, 1 }, { "Plus", 32, 0, 70, 0 }, { "Basic", 34, 0, 44, 2 }, { "Plus", 36, 0, 58, 1 },
{ "Basic", 4, 1, 22, 2 }, { "Plus", 7, 1, 30, 3 }, { "Basic", 13, 1, 26, 1 }, { "Plus", 28, 0, 62, 2 }
}
)
RETURN
SELECTCOLUMNS (
SURVIVAL.COX.PENALIZED ( Customers, [Months], [Left], "folds=4, lambda=min", "Plan", [Plan], "Spend", [Spend], "Tickets", [Tickets] ),
"Variable", [Variable],
"Selected", [Selected],
"Lambda", [Lambda],
"Lambda min", [Lambda min],
"Lambda 1se", [Lambda 1se],
"CV deviance", [CV deviance]
)| Variable | Selected | Lambda | Lambda min | Lambda 1se | CV deviance |
|---|---|---|---|---|---|
| Plan: Plus | TRUE | 0.0156073640147004 | 0.0156073640147004 | 0.0434284174827736 | 1.80193649665295 |
| Spend | TRUE | 0.0156073640147004 | 0.0156073640147004 | 0.0434284174827736 | 1.80193649665295 |
| Tickets | TRUE | 0.0156073640147004 | 0.0156073640147004 | 0.0434284174827736 | 1.80193649665295 |
No penalty
At lambda=0 the coefficients are SURVIVAL.COX's: the plan's runs far out, which the lasso pulled in.
EVALUATE
VAR Customers =
DATATABLE (
"Plan", STRING, "Months", INTEGER, "Left", INTEGER, "Spend", INTEGER, "Tickets", INTEGER,
{
{ "Basic", 3, 1, 20, 4 }, { "Basic", 5, 1, 25, 3 }, { "Basic", 6, 0, 40, 1 }, { "Basic", 8, 1, 30, 5 },
{ "Plus", 9, 1, 35, 2 }, { "Basic", 10, 0, 45, 0 }, { "Basic", 12, 1, 28, 3 }, { "Plus", 14, 0, 60, 1 },
{ "Plus", 15, 1, 38, 4 }, { "Basic", 16, 0, 50, 1 }, { "Plus", 18, 1, 42, 2 }, { "Plus", 20, 0, 55, 0 },
{ "Basic", 22, 1, 33, 2 }, { "Plus", 24, 0, 65, 1 }, { "Plus", 25, 1, 48, 3 }, { "Basic", 27, 0, 36, 0 },
{ "Plus", 30, 1, 52, 1 }, { "Plus", 32, 0, 70, 0 }, { "Basic", 34, 0, 44, 2 }, { "Plus", 36, 0, 58, 1 },
{ "Basic", 4, 1, 22, 2 }, { "Plus", 7, 1, 30, 3 }, { "Basic", 13, 1, 26, 1 }, { "Plus", 28, 0, 62, 2 }
}
)
RETURN
SELECTCOLUMNS (
SURVIVAL.COX.PENALIZED ( Customers, [Months], [Left], "lambda=0", "Plan", [Plan], "Spend", [Spend], "Tickets", [Tickets] ),
"Variable", [Variable],
"Coefficient", [Coefficient],
"Hazard ratio", [Hazard ratio]
)| Variable | Coefficient | Hazard ratio |
|---|---|---|
| Plan: Plus | 4.77261238824053 | 118.227695526513 |
| Spend | -0.612029431999423 | 0.542249293598225 |
| Tickets | 0.866047595553555 | 2.37749543548052 |