--- name: decision-model-calibration description: | Turning a model's class probabilities into numbers you can set a threshold on: building the labelled corpus, measuring accuracy and calibration honestly (ECE with a noise floor, Brier, risk–coverage), fitting temperature/Platt, removing option-order bias, conformal prediction sets, and choosing per-action thresholds. Applies to Jev and any typed classifier with probabilities, and says which LLM providers expose logprobs at all. USE WHEN: user asks whether a confidence/probability can be trusted, how to pick a threshold, about "calibration", "ECE", "reliability diagram", "temperature scaling", "Platt", "isotonic", "CalibratedClassifierCV", "conformal prediction", "abstain", "selective prediction", "risk coverage", "position bias", "option order", "logprobs", or wants to evaluate Jev or an LLM classifier against labels. DO NOT USE FOR: training classifiers from scratch, or generic LLM evals of free-text output (no class probabilities to calibrate). allowed-tools: Read, Grep, Glob, Write, Edit, Bash --- # Calibrating decision models > **Deep Knowledge**: Use `mcp__documentation__fetch_docs` with technology: > `decision-calibration` and one of these topics: > - `calibration`: ECE variants, debiased estimators, isotonic and vector scaling, bootstrap CIs, contextual calibration. > - `order-bias`: position and label bias, flip-rate measurement, PriDe. > - `conformal`: LAC, APS and RAPS, plus class-conditional coverage. > - `provider-logprobs`: request examples for each provider. > - `eval-harness`: a complete runnable harness that compares Jev with an LLM on your own corpus and writes the decision report. A probability is useful only if "0.9" means right about nine times in ten **on your data**. No vendor can promise that for you: calibration depends on your inputs, your language and your label set, and it shifts with every model version. This skill is the procedure; `typed-decision-models` has the evidence about Jev in particular. The code below was run on simulated data with known over-confidence before it was written here (Python 3.12, numpy 2.5, scipy 1.18, scikit-learn 1.9.1). ## 1. The corpus comes first It outlives every model you try against it, and it is the only thing that lets you compare them. - **Draw from production traffic**, not from what is easy to label. Same language, same state shape, same class balance — and include the ambiguous cases, which are the ones thresholds exist for. - **Size.** About 50 labels per question can fit a one-parameter correction; **fewer than ~30 can make calibration worse** (observed on Jev). Isotonic regression needs on the order of 1,000 (scikit-learn warns against it when the number of calibration samples is "too low (≪1000)"). Thresholds for rare, expensive classes need enough examples *of that class*. - **Split** into a calibration half and a test half; never report a metric on the data a correction was fitted on. - **Label what a person on the team would do**, not what the model said. A corpus labelled by another model measures agreement, not correctness — the exact flaw in the Jev launch benchmark. - **Record the model version** alongside every stored probability, so a re-tune can be triggered when it changes. ## 2. Measure ```python import numpy as np def ece(confidence, correct, n_bins=10): """Expected calibration error with equal-mass bins (less biased than equal-width).""" confidence = np.asarray(confidence, float) correct = np.asarray(correct, float) order = np.argsort(confidence) total = len(confidence) return sum( len(b) / total * abs(correct[b].mean() - confidence[b].mean()) for b in np.array_split(order, n_bins) if len(b) ) def ece_noise_floor(confidence, n_bins=10, n_sims=2000, seed=0): """ECE a *perfectly calibrated* model would show on this many items with these confidences. A measured ECE inside this band is indistinguishable from noise.""" rng = np.random.default_rng(seed) confidence = np.asarray(confidence, float) sims = [ece(confidence, rng.random(confidence.size) < confidence, n_bins) for _ in range(n_sims)] return np.percentile(sims, [50, 95]) def brier(probs, labels): """Multiclass Brier score; probs is (n, k), labels are column indices.""" probs = np.asarray(probs, float) onehot = np.eye(probs.shape[1])[labels] return np.mean(np.sum((probs - onehot) ** 2, axis=1)) ``` Report, per question: - **accuracy with a confidence interval**. n = 77 gives roughly ±10 points; - **ECE next to its noise floor**. The floor depends on n *and* on how spread the confidences are: at n = 60 a perfectly calibrated model scores a median ≈ 0.06 when confidences cluster near 1, but ≈ 0.13 when they spread over 0.4–1 (simulated with `ece_noise_floor`). A bare ECE is uninterpretable; - **Brier score**, which punishes both miscalibration and poor discrimination; - **direction**: is the error over-confident (most binned accuracy below binned confidence) or compressed toward the middle? What "confidence" means: - **Choice:** use `p_max`, or the probability of the chosen option. - **Noul:** use `max(p, 1 − p)`, with correctness judged at your threshold. **Do not calibrate Jev's `confidence` field.** It is a dispersion statistic derived from the distribution (`(p_max − 1/n)/(1 − 1/n)` for Choice), not a probability. Calibrate `probabilities` or `noul`. ## 3. Fit ### Choice / Score: one temperature ```python from scipy.optimize import minimize_scalar from scipy.special import softmax # Jev rounds probabilities to 0.01 and returns exact zeros. log(0) is -inf, and a # tiny floor (1e-6) turns every zero into a huge negative logit that inflates the # fitted temperature. Half the rounding quantum is the honest floor. FLOOR = 0.005 def to_matrix(answers, options): """[{'billing': 0.88, ...}, ...] -> (n, k) array in a fixed option order.""" return np.array([[a[o] for o in options] for a in answers], float) def fit_temperature(probs, labels, floor=FLOOR): logits = np.log(np.clip(probs, floor, 1.0)) labels = np.asarray(labels) def nll(t): p = softmax(logits / t, axis=1) return -np.mean(np.log(p[np.arange(len(labels)), labels])) return minimize_scalar(nll, bounds=(0.05, 20.0), method="bounded").x def apply_temperature(probs, t, floor=FLOOR): return softmax(np.log(np.clip(probs, floor, 1.0)) / t, axis=1) ``` T > 1 means the model was over-confident; T < 1 means it was under-confident. The floor is not a detail: - **Simulation:** true over-confidence T = 2.0, outputs rounded to 0.01 like Jev's. A 0.005 floor recovered T = 1.84; a 1e-6 floor reported **3.02**. - **Published study:** the same artefact made an independent Jev audit publish temperatures of 3.29 and 3.40. Its author corrected them on 2026-09-22 to 1.30 and 1.92. ### Noul: Platt (two parameters) ```python from scipy.special import logit from sklearn.linear_model import LogisticRegression def fit_platt(p_yes, y, floor=FLOOR): x = logit(np.clip(np.asarray(p_yes, float), floor, 1 - floor)).reshape(-1, 1) return LogisticRegression(C=1e6).fit(x, y) # effectively unregularised: two parameters def apply_platt(model, p_yes, floor=FLOOR): x = logit(np.clip(np.asarray(p_yes, float), floor, 1 - floor)).reshape(-1, 1) return model.predict_proba(x)[:, 1] ``` Platt's slope also corrects **compression toward 0.5**, which independent work reports for Jev and which a pure shift cannot fix. ### Your own scikit-learn classifier scikit-learn ≥ 1.8 has `method="temperature"` alongside `"sigmoid"` and `"isotonic"`. `cv="prefit"` was deprecated in 1.6 and 1.9.1 rejects it with `InvalidParameterError`; wrap an already-fitted model in `FrozenEstimator`: ```python from sklearn.calibration import CalibratedClassifierCV from sklearn.frozen import FrozenEstimator calibrated = CalibratedClassifierCV(FrozenEstimator(clf), method="temperature").fit(X_cal, y_cal) ``` ### What one fit buys - **Classic result:** temperature scaling alone fixes most of the miscalibration of modern networks (Guo et al., ICML 2017). - **On Jev:** refitting a Platt intercept on 50 labels cut held-out ECE by 62% in-domain; the slope did not transfer to new domains. - **This skill's simulation:** ECE went 0.145 → 0.035 on held-out data. Fit **one correction per question**. Never share one across Noul, Choice and Score, or across fields. ## 4. Remove option-order bias before trusting a Choice Multiple-choice selectors favour positions: - Zheng et al., *Large Language Models Are Not Robust Multiple Choice Selectors* (ICLR 2024). - Pezeshkpour & Hruschka (2023) measured gaps of 13–75% from reordering alone. - TypeSafe documents that `jev-1.13` "leans toward the option that comes first". **Cyclic permutation** averages over n orders instead of n!. With Jev, all the orders go into **one request**: questions are independent and share one read of the state, so the extra orders cost tokens, not a round trip. ```python def rotations(criteria, limit=None): """Cyclic rotations of a Choice's options (Zheng et al., ICLR 2024): n orders, not n!.""" items = list(criteria.items()) n = len(items) count = n if limit is None else min(limit, n) shifts = sorted({round(j * n / count) % n for j in range(count)}) # evenly spread, first order kept return [dict(items[i:] + items[:i]) for i in shifts] def order_checked_choice(client, state, instructions, criteria, limit=None): """Ask the same Choice under several option orders in ONE request and average. Questions in a request are independent and share one read of the state, so the extra orders cost only their question tokens, not another round trip. """ from typesafe_sdk import Choice orders = rotations(criteria, limit) questions = {f"order_{i}": Choice(instructions=instructions, criteria=c) for i, c in enumerate(orders)} result = client.system_one(state, questions) answers = [result.choices[key] for key in questions] averaged = {o: sum(a.probabilities[o] for a in answers) / len(answers) for o in criteria} winner = max(averaged, key=averaged.get) stable = all(a.choice == winner for a in answers) return winner, averaged, stable ``` - **Cost:** n orders cost ~n× the question tokens. All of them must fit in the 64k request budget. - **Many options:** set `limit` (e.g. 3–4 evenly spaced rotations). - **How to use it:** - Run it **offline over the corpus** first, to measure how often the answer flips. - Add it in production only for decisions whose `stable=False` rate justifies the tokens. - Calibrate the *averaged* probabilities, not the single-order ones. - **Option names matter too.** Renaming labels moved AUC from 0.81 to 0.58 on hosted Jev (arXiv 2609.26758). Once a wording is tuned, freeze option names like an API. - **Contextual calibration** (Zhao et al., *Calibrate Before Use*, ICML 2021) is the complementary fix for a label prior. Score a content-free input such as "N/A", then divide out the bias it reveals. ## 5. Choose thresholds from cost, then check coverage ```python def risk_coverage(confidence, correct): """For each candidate bar: the share of items kept and the error rate among them.""" confidence = np.asarray(confidence, float) correct = np.asarray(correct, bool) rows = [] for bar in np.unique(confidence): kept = confidence >= bar rows.append((bar, kept.mean(), 1 - correct[kept].mean())) return rows def lowest_bar_for(target_error, confidence, correct, min_kept=30): """Lowest bar whose kept items err at most target_error, with at least min_kept items.""" for bar, coverage, error in risk_coverage(confidence, correct): if error <= target_error and coverage * len(confidence) >= min_kept: return bar, coverage, error return None ``` - **Ask for each direction's cost separately.** For a Noul, the false-positive and false-negative costs set where the bar sits. TypeSafe's own advice: raise it when acting on a false yes is expensive, lower it when missing a true yes is. - **Read coverage alongside error.** A bar that hands 80% of traffic to an LLM costs more than no decision model at all; the hand-off rate is the number that shows it. - **Bars are ranges, not points.** Jev's probabilities move by a few hundredths between identical calls, so leave margin. - **For a guarantee rather than an estimate**, use selective classification with a risk bound (Geifman & El-Yaniv, NeurIPS 2017). It picks the bar so that the target error holds with high probability. - **Re-tune whenever any of these change:** model version, option wording, option set, or traffic mix. ## 6. Conformal sets: "one of these, with 90% coverage" When acting needs a guarantee and a single label is too strong, return a **set**. Use split conformal with the LAC score (Kumar et al. 2023), on calibrated probabilities: ```python def conformal_qhat(p_true_label, alpha): """p_true_label: calibrated probability each calibration item gave its *true* label.""" scores = np.sort(1.0 - np.asarray(p_true_label, float)) n = len(scores) k = int(np.ceil((n + 1) * (1 - alpha))) return 1.0 if k > n else scores[k - 1] # too few items: the set must be everything def prediction_set(probabilities, qhat): return [option for option, p in probabilities.items() if 1.0 - p <= qhat] ``` - **What it guarantees:** the true label is in the set at least 1 − α of the time, provided traffic is exchangeable with the calibration data. In simulation, α = 0.1 gave 91.7% coverage. - **How to act on it:** - singleton → act automatically; - two or more options → ask the user or a human; - empty → treat as out of distribution. - **Variants:** APS (Romano et al. 2020) and RAPS (Angelopoulos et al., ICLR 2021) give adaptive or smaller sets. - **When the guarantee breaks:** it does not survive a shift in traffic. Re-calibrate when the input mix changes. ## 7. Which providers give you probabilities at all (2026-10-03) | Provider | Class probabilities | Notes | |---|---|---| | **TypeSafe Jev** | yes, native | `probabilities` / `noul`, rounded to 0.01 | | **Anthropic (Claude)** | **no** | Messages API has no logprobs; OpenAI-compat `logprobs`/`top_logprobs` are "Ignored", response field "Always empty". Structured outputs give a typed enum, not a distribution | | **OpenAI** | yes, restricted | `logprobs` + `top_logprobs` 0–20. On GPT-6, with reasoning effort other than `none` these must be removed — no logprobs from a reasoning model that is reasoning | | **Google Gemini** | documented, **not on 3.x** | `responseLogprobs: true`, `logprobs` 0–20; `text/x.enum` for enums. A Google staff member stated on the developer forum (2026-08-05) that "logprobs are no longer returned for 3.X models"; Vertex users report 2.5 losing them too. Test the exact model | | **vLLM** | yes | `logprobs` (cap `max_logprobs`, default 20); structured `choice` output. Default `logprobs_mode` is `raw_logprobs` — **before** logit processors, so a constraint is not reflected | | **llama.cpp server** | yes | `n_probs`; `post_sampling_probs` for after the sampling chain; `grammar` / `json_schema` constraints | With logprobs, constrain the label to a **single token**. For a single-token enum, a grammar's renormalisation is exactly the conditional class posterior. For multi-token labels it is not. **Without logprobs (Claude), each option costs more than one call:** - **Sample k times** and use label frequency. - **Verbalized confidence.** It is better calibrated than token probabilities for RLHF models (Tian et al., EMNLP 2023), but over-confident (Xiong et al., ICLR 2024). - **A panel of prompts that vote.** Post-training makes this worse: GPT-4's MMLU ECE went from 0.007 pre-trained to 0.074 after RLHF (GPT-4 Technical Report, Fig. 8). Calibrate whatever you end up with. ## Checklist before a threshold goes to production - [ ] Labelled corpus from production traffic, in the production language, split cal/test - [ ] Accuracy with CI, ECE with noise floor, Brier — per question, on the test half - [ ] Option-order flip rate measured; averaging adopted where it matters - [ ] One temperature (Choice/Score) or Platt (Noul) fit per question, floor = half the rounding quantum - [ ] Threshold chosen from asymmetric costs; coverage / hand-off rate recorded - [ ] Model version pinned and logged; re-tune trigger defined - [ ] Destructive actions still gated by approval, not by a probability alone