typestar

Probabilities, thresholds and AUC in Python

predict_proba gives scores; the threshold is a decision you make, not the model.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)

scores = model.predict_proba(X_test)[:, 1]
print(f"auc {roc_auc_score(y_test, scores):.3f}")

fpr, tpr, thresholds = roc_curve(y_test, scores)
print(len(thresholds), fpr[:3].round(3), tpr[:3].round(3))

How it works

  1. predict_proba returns one column per class.
  2. roc_auc_score grades the ranking, independent of any threshold.
  3. roc_curve gives the tradeoff to plot or pick from.

Keywords and builtins used here

The run, in numbers

Lines
19
Characters to type
750
Tokens
168
Three-star pace
110 tpm

At the three-star pace of 110 tokens a minute, this run takes about 92 seconds.

Type this snippet

Step 3 of 4 in Scoring, step 18 of 25 in Machine learning with scikit-learn.

← Previous Next →