{
"cells": [
{
"cell_type": "markdown",
"id": "intro",
"metadata": {},
"source": [
"# 05 - Uncertainty quantification\n",
"\n",
"Three drop-in estimators in `maldideepkit.uncertainty` wrap a fitted classifier and report per-sample uncertainty alongside the usual predictions:\n",
"\n",
"1. **Monte Carlo Dropout** - stochastic forward passes with dropout layers active. Decomposes total uncertainty into *epistemic* (model disagreement) and *aleatoric* (data noise) components (Gal & Ghahramani, 2016; Kendall & Gal, 2017).\n",
"2. **Laplace approximation** - last-layer or full-network Gaussian posterior over the weights, via [`laplace-torch`](https://github.com/aleximmer/Laplace). Requires the optional `uncertainty` extra: `pip install \"maldideepkit[uncertainty]\"`.\n",
"3. **Split conformal prediction** - distribution-free prediction *sets* with a coverage guarantee. Pure NumPy, no extra deps.\n",
"\n",
"All three return a common `UncertaintyResult` dataclass, so call sites stay the same when you swap methods.\n",
"\n",
"Uses the **MALDI-Kleb-AI** dataset cached by `notebooks/_demo.py` (see notebook 01)."
]
},
{
"cell_type": "markdown",
"id": "load-md",
"metadata": {},
"source": [
"## 1. Load the demo dataset"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "load",
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-12T08:46:28.436736Z",
"iopub.status.busy": "2026-05-12T08:46:28.436628Z",
"iopub.status.idle": "2026-05-12T08:46:46.132104Z",
"shell.execute_reply": "2026-05-12T08:46:46.131340Z"
}
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "349dbac404fd4f17bad367f673f70488",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Processing spectra: 0%| | 0/743 [00:00"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"\n",
"x_max = float(np.maximum(mc_result.epistemic.max(), mc_result.aleatoric.max()))\n",
"bins = np.linspace(0.0, x_max, 31)\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(10, 3.5), sharex=True, sharey=True)\n",
"axes[0].hist(mc_result.epistemic, bins=bins, color='C0', alpha=0.85)\n",
"axes[0].set_title('Epistemic (model disagreement)')\n",
"axes[0].set_xlabel('uncertainty')\n",
"axes[0].set_ylabel('test spectra')\n",
"axes[1].hist(mc_result.aleatoric, bins=bins, color='C1', alpha=0.85)\n",
"axes[1].set_title('Aleatoric (data noise)')\n",
"axes[1].set_xlabel('uncertainty')\n",
"axes[0].set_xlim(0.0, x_max)\n",
"fig.suptitle('MC Dropout decomposition on the test split')\n",
"fig.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "conformal-md",
"metadata": {},
"source": [
"## 5. Split conformal prediction\n",
"\n",
"`ConformalPredictor` learns a single quantile on the calibration split, then returns a *prediction set* per test spectrum that contains the true class with marginal probability $\\geq 1 - \\alpha$. With `alpha=0.1` we target 90 % coverage.\n",
"\n",
"Per-sample uncertainty is the prediction set size normalised by `n_classes`: `1 / n_classes` for a confident singleton, `1.0` when every class is included."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "conformal",
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-12T08:46:48.871345Z",
"iopub.status.busy": "2026-05-12T08:46:48.871107Z",
"iopub.status.idle": "2026-05-12T08:46:48.879635Z",
"shell.execute_reply": "2026-05-12T08:46:48.879050Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"target coverage: 0.90\n",
"empirical coverage: 0.960\n",
"mean set size: 1.62 (singleton fraction: 0.38)\n",
"calibration n: 148\n"
]
}
],
"source": [
"from maldideepkit.uncertainty import ConformalPredictor\n",
"\n",
"cp = ConformalPredictor(clf, alpha=0.10).calibrate(X_cal, y_cal)\n",
"cp_result = cp.predict_with_uncertainty(X_te)\n",
"\n",
"sets = cp_result.metadata['prediction_sets'] # (n_test, n_classes) bool\n",
"set_sizes = cp_result.metadata['set_sizes'] # (n_test,) int\n",
"covered = sets[np.arange(len(y_te)), np.asarray(y_te)]\n",
"\n",
"print(f'target coverage: {1 - cp.alpha:.2f}')\n",
"print(f'empirical coverage: {covered.mean():.3f}')\n",
"print(f'mean set size: {set_sizes.mean():.2f} '\n",
" f'(singleton fraction: {(set_sizes == 1).mean():.2f})')\n",
"print(f'calibration n: {cp_result.metadata[\"n_calibration\"]}')"
]
},
{
"cell_type": "markdown",
"id": "conformal-md2",
"metadata": {},
"source": [
"Conformal prediction *sets* are most useful when the model is uncertain: a singleton means the model is confident, a full set means \"could be either\". Inspect a few examples on the test split."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "conformal-show",
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-12T08:46:48.880866Z",
"iopub.status.busy": "2026-05-12T08:46:48.880764Z",
"iopub.status.idle": "2026-05-12T08:46:48.889020Z",
"shell.execute_reply": "2026-05-12T08:46:48.888580Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" true_y | \n",
" argmax_y | \n",
" p(argmax) | \n",
" set | \n",
" set_size | \n",
" covered | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 1 | \n",
" 1 | \n",
" 0.768 | \n",
" [0, 1] | \n",
" 2 | \n",
" True | \n",
"
\n",
" \n",
" | 1 | \n",
" 1 | \n",
" 1 | \n",
" 0.997 | \n",
" [1] | \n",
" 1 | \n",
" True | \n",
"
\n",
" \n",
" | 2 | \n",
" 1 | \n",
" 0 | \n",
" 0.532 | \n",
" [0, 1] | \n",
" 2 | \n",
" True | \n",
"
\n",
" \n",
" | 3 | \n",
" 1 | \n",
" 1 | \n",
" 1.000 | \n",
" [1] | \n",
" 1 | \n",
" True | \n",
"
\n",
" \n",
" | 4 | \n",
" 0 | \n",
" 0 | \n",
" 0.931 | \n",
" [0, 1] | \n",
" 2 | \n",
" True | \n",
"
\n",
" \n",
" | 5 | \n",
" 0 | \n",
" 0 | \n",
" 0.977 | \n",
" [0] | \n",
" 1 | \n",
" True | \n",
"
\n",
" \n",
" | 6 | \n",
" 1 | \n",
" 1 | \n",
" 0.731 | \n",
" [0, 1] | \n",
" 2 | \n",
" True | \n",
"
\n",
" \n",
" | 7 | \n",
" 0 | \n",
" 0 | \n",
" 0.759 | \n",
" [0, 1] | \n",
" 2 | \n",
" True | \n",
"
\n",
" \n",
"
\n",
"