bakeoff

Race every reasonable approach to a text-classification problem and rank them on accuracy, cost, latency and calibration. Then show where each one quietly fails.

"Should we fine-tune something or just prompt an LLM?" usually gets answered by vibes, or by an accuracy number with no bill attached. bakeoff puts the trade on one screen: the prompted arm may cost two orders of magnitude more and take three orders of magnitude longer for a result the tf-idf baseline already had.

The leaderboard is the shop window. The substance is the two things underneath it: calibration, because the confident model is often the wrong one, and slice discovery, because the average hides the subgroup where a model falls over.

The leaderboard

Ten arms on the bundled corpus: 2,252 training and 748 held-out synthetic support tickets over 14 intents, one seeded stratified split, seed 20260831. Produced by just bake on 2026-08-31, on a Linux x86_64 box with an Intel i7-9700K and OMP_NUM_THREADS=1, Python 3.12.10, scikit-learn 1.9.0.

arm	macro-F1	accuracy	ECE	p50 ms	p95 ms	cost/1k	fit s	note
finetuned-encoder	0.653	0.684	0.083	7.1	21.9	free	2887.80	cached, trained offline
tfidf-linearsvc-raw	0.624	0.668	0.408	0.7	0.9	free	0.12
tfidf-logreg	0.619	0.654	0.059	0.8	0.9	free	0.33
tfidf-linearsvc	0.614	0.662	0.081	5.1	6.0	free	0.27
llm-fewshot	0.611	0.658	0.251	1511.7	2726.6	$0.0630	0.03	cached, no training
char-nb	0.602	0.635	0.351	0.6	0.9	free	0.23
embed-logreg	0.584	0.610	0.140	0.3	0.4	free	0.22
embed-knn	0.584	0.635	0.088	6.5	10.9	free	0.01
llm-zeroshot	0.527	0.560	0.289	1612.6	5731.4	$0.0313	0.00	cached, no training
keyword-rules	0.331	0.316	0.177	0.0	0.0	free	0.00

Every number in this file came out of that run. just bake reproduces it, just check tests it, and CI runs both on every push.

What the columns are hiding

The fine-tuned encoder wins on macro-F1, and its fit s column says what that cost: 2,888 seconds of CPU training against 0.33 for the tf-idf logistic regression, for three points of macro-F1. That number is deliberately not hidden. Training happened offline in scripts/finetune.py, so a naive "how long did fit take in this run" would have read 0.00 and made the comparison a lie.

The two prompted arms are the only ones carrying a bill, and neither of them wins. llm-fewshot comes fifth at 0.611, below a tf-idf LinearSVC that scores 0.614, costs nothing and answers in 5 milliseconds. The prompted arm takes 1.5 seconds per ticket — 296x the latency — and $0.063 per thousand tickets, for three thousandths of a point less macro-F1. llm-zeroshot costs half as much again and comes ninth.

keyword-rules is the ordered keyword list this kind of system usually starts as: first match wins, everything else to the busiest queue. It scores 0.331. Everything above it is what a model buys, and the fine-tuned encoder nearly doubles it.

Cost is never guessed. LLM arms price counted tokens against src/bakeoff/data/pricing.yaml (read off the OpenRouter model list on 2026-08-31). An arm whose model is not in that table reports unknown, never 0, because a zero in a cost column reads as free.

Calibration, and the decision it pays for

A model that says 0.95 and is right 95% of the time has nothing to answer for. A model that says 0.95 and is right two thirds of the time is the dangerous one, because every threshold downstream is now wrong. Expected calibration error is the mean gap between confidence and accuracy over ten equal-width confidence bins, weighted by how many records land in each.

That matters because of what you do with it. A triage model that routes everything is judged on accuracy. A triage model that can hand the hard mail to a person is judged on two numbers at once: how accurate it is on what it kept, and how much it handed over. bakeoff escalate asks the question that way round.

arm                  ECE    at conf >= 0.95            to hold 90%              escalates
-------------------  -----  ---------------  --------  -------------  --------  ---------
finetuned-encoder    0.083  never that sure  0% kept   conf >= 0.639  61% kept  39%
tfidf-linearsvc-raw  0.408  never that sure  0% kept   conf >= 0.278  43% kept  57%
tfidf-logreg         0.059  0.966 acc        4% kept   conf >= 0.713  41% kept  59%
tfidf-linearsvc      0.081  never that sure  0% kept   conf >= 0.701  46% kept  54%
llm-fewshot          0.251  0.913 acc        43% kept  conf >= 0.950  43% kept  57%
char-nb              0.351  0.648 acc        94% kept  unreachable    -         -
embed-logreg         0.140  never that sure  0% kept   conf >= 0.542  36% kept  64%
embed-knn            0.088  0.880 acc        24% kept  unreachable    -         -
llm-zeroshot         0.289  0.888 acc        12% kept  unreachable    -         -
keyword-rules        0.177  0.628 acc        45% kept  unreachable    -         -

Read the char-nb row. It reports at least 0.95 confidence on 94% of the held-out set and is right 65% of the time on those. Threshold on its stated confidence and you have bought nothing. Its ECE of 0.351 said so before you tried.

Then read finetuned-encoder. It never claims 0.95 on this problem at all, which on a 14-way task at 68% accuracy is the correct thing for a model to do. Ask it for 90% instead of a round number and it answers: above 0.639 it is right 90% of the time, which keeps 61% of the queue and hands 39% to a person. That is a number you can staff against.

keyword-rules is the honest extreme. A rule that fires is reported at confidence 1.0 because a rule has no notion of doubt; it fires on 45% of tickets and is right on 63% of those, and there is no threshold anywhere that gets it to 90%.

Read the two tfidf-linearsvc rows together. They are the same model. One has Platt scaling on top and the other has a softmax over the raw decision function, which is what people reach for when they need a probability out of an SVM. Their macro-F1 differs by a rounding error and their ECE differs by a factor of five, and the escalation columns are where that stops being cosmetic.

Slice discovery

A slice is a subgroup you can already name: a metadata column, a quantile bin of a numeric column, the true class, or the presence of a common n-gram. For each one bakeoff reports the accuracy on that subgroup, its size, and a 95% Wilson interval, and ranks the worst by the gap from the arm's own average scaled by the square root of the slice size, so a small unlucky subgroup cannot top the list.

Two filters keep noise out. A slice needs a minimum number of records, and the top of its confidence interval has to sit below the arm's own average, so a subgroup that merely got unlucky never becomes a finding.

From the same run, bakeoff explain out/report.json --arm char-nb:

worst slices (min support, 95% Wilson interval, own average 0.635)
  ngram = also                 n=137   acc 0.409 [0.330, 0.492]  +0.226 vs average  (discovered)
  ngram = you                  n=263   acc 0.510 [0.449, 0.569]  +0.126 vs average  (discovered)
  length_bucket = long         n=81    acc 0.457 [0.353, 0.565]  +0.178 vs average  (declared)
  ngram = is                   n=230   acc 0.530 [0.466, 0.594]  +0.105 vs average  (discovered)
  ngram = not                  n=155   acc 0.510 [0.432, 0.587]  +0.125 vs average  (discovered)

The top finding is not a metadata column anybody thought to declare. It is the word "also", on 137 tickets, where accuracy falls from 63.5% to 40.9% with an interval that does not come near the average. Those are the tickets carrying a second intent — "Also, while I have you..." — and the model is filing them under whichever intent it noticed second. length_bucket = long is the same population seen through a declared column.

And --arm keyword-rules:

worst slices (min support, 95% Wilson interval, own average 0.316)
  class = bug-report           n=77    acc 0.130 [0.072, 0.223]  +0.186 vs average  (declared)
  channel = phone              n=152   acc 0.184 [0.131, 0.253]  +0.131 vs average  (declared)
  ngram = my                   n=147   acc 0.184 [0.129, 0.254]  +0.132 vs average  (discovered)
  ngram = while                n=164   acc 0.213 [0.158, 0.282]  +0.102 vs average  (discovered)
  class = performance-issue    n=54    acc 0.148 [0.077, 0.266]  +0.167 vs average  (declared)

The rules do worst on phone tickets, which are transcripts: no punctuation, filler words and about 45% of the words missing, so the keyword often is not there to match. A model that reads the whole ticket does not care as much. That is the argument for replacing the rules, made in one line of a report rather than in a meeting.

Nothing here infers a protected attribute and nothing here is causal. A finding says "on these N records the model was right X% of the time against Y% overall". What that means is the reader's call.

The fine-tuned arm

finetuned-encoder is a real fine-tune, not a wrapper. scripts/finetune.py trains an encoder end to end on CPU with the Hugging Face Trainer: the same seeded stratified split the bake-off uses, the training portion split again into train and validation, early stopping with patience 2 on validation macro-F1, seed 20260831. Run it with just finetune.

Model choice. microsoft/deberta-v3-xsmall was tried first and rejected. Measured on this machine at batch 16 and sequence length 128, one training step took 120.75 s against 1.70 s for distilbert-base-uncased and 0.56 s for sentence-transformers/all-MiniLM-L6-v2. DeBERTa-v3's disentangled attention has no fast CPU path in this build, and 846 steps at two minutes each is fourteen hours for a demo that has to be reproducible. distilbert-base-uncased was taken instead: a general-purpose encoder, six layers, 67M parameters, and 48 minutes of real training.

What the run produced. 1,916 training records, 336 validation records, 6 epochs requested, 2,887.8 s of CPU, validation macro-F1 0.7353 and validation accuracy 0.7440. The card is committed at src/bakeoff/data/model_card.json.

What ships. The model is exported to ONNX and dynamically quantised to int8 by scripts/export_onnx.py: 268 MB fp32 down to 67 MB int8. The int8 file is what the arm serves and therefore what the leaderboard measures, so the 0.653 above is the artifact's number and not the torch model's. Weights are not committed to git; they are attached to the model-v1 release.

How the arm gets them. In order: $BAKEOFF_MODEL_DIR, a local models/release from your own just finetune, ~/.cache/bakeoff/model-v1, then the release itself (which needs BAKEOFF_ALLOW_NETWORK=1). Running inference needs the ONNX runtime, which is an extra:

uv sync --extra onnx
BAKEOFF_ALLOW_NETWORK=1 bakeoff run configs/tickets.yaml

The demo and the test suite need none of that. Predictions for the held-out set are cached in src/bakeoff/data/finetuned_cache.jsonl, keyed by text, one record per batch so the recorded latency is a genuine single-record serving latency comparable with every other arm's. Training never happens in the demo, and a cache miss with no model available raises rather than guessing.

Add your own arm

A class with fit and predict_proba, one line in the registry, one entry in the YAML. Nothing in the runner, the report, the gate or the demo needs to know it exists.

# src/bakeoff/arms/centroid.py
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

from ..cost import TokenUsage
from ..dataset import Dataset
from .base import ARMS, CostModel


class NearestCentroid:
    """One tf-idf centroid per label; a ticket goes to the nearest one."""

    needs_fit = True
    cost_model = CostModel("local")

    def __init__(self, temperature: float = 8.0) -> None:
        self.temperature = temperature
        self.usage = TokenUsage()
        self.vectorizer = TfidfVectorizer(sublinear_tf=True, min_df=2)

    def fit(self, train: Dataset) -> None:
        self.labels = train.labels
        matrix = np.asarray(self.vectorizer.fit_transform(train.texts).todense())
        by_label = np.array([r.label for r in train.records])
        self.centroids = np.vstack(
            [matrix[by_label == label].mean(axis=0) for label in self.labels]
        )

    def predict_proba(self, texts: list[str]) -> np.ndarray:
        vectors = np.asarray(self.vectorizer.transform(texts).todense())
        scores = vectors @ self.centroids.T * self.temperature
        exp = np.exp(scores - scores.max(axis=1, keepdims=True))
        return exp / exp.sum(axis=1, keepdims=True)


ARMS["nearest-centroid"] = NearestCentroid

Then in configs/tickets.yaml:

  - arm: nearest-centroid
    params: { temperature: 12.0 }

tests/test_extensibility.py runs exactly that: a suite defined entirely in YAML, against an arm registered at test time, including two instances of the same class with different parameters.

The gate

bakeoff gate out/report.json --baseline baselines/main.json

Exit 1 if, for any arm in the baseline, macro-F1 dropped by more than --max-f1-drop, ECE rose by more than --max-ece-rise, or any declared slice fell further behind that arm's own average by more than --max-slice-drop.

The third condition is the one worth having. An average can improve while a subgroup gets much worse, and only the slice check sees that. tests/test_gate.py has that case explicitly: macro-F1 up, ECE down, and the gate fails anyway because phone tickets lost thirteen points.

CI runs the gate against this repo's own committed baseline on every push.

The CLI

bakeoff run configs/tickets.yaml --report out/report.json --html out/report.html
bakeoff arms                                            # the registry
bakeoff explain out/report.json --arm tfidf-linearsvc --target-precision 0.9
bakeoff escalate out/report.json --target-precision 0.9
bakeoff gate out/report.json --baseline baselines/main.json
bakeoff warm configs/tickets.yaml                       # fill the prediction caches

The demo

just demo   # http://localhost:8080

Tick arms on and off and re-run. The local arms genuinely retrain in the browser round trip and the page says which ones it just fitted and how long that took; the prompted and fine-tuned arms replay from their shipped caches and are badged cached. Click an arm for its reliability diagram, its coverage-against-accuracy curve, per-class F1, the confusion matrix and its five worst slices with sizes and intervals. Set a latency ceiling and a cost ceiling and it names the best arm that fits inside both.

The data

src/bakeoff/data/tickets.jsonl is 3,000 synthetic support tickets over 14 intents, with channel, length_bucket, word_count and has_attachment as metadata. It is generated by scripts/generate_dataset.py from the vocabulary in scripts/intents.yaml, seeded, so just data reproduces it byte for byte. Everything in it is invented: no real names, companies, addresses, order numbers or ticket text.

Three things about it are deliberate:

• Fourteen intents, unevenly sized, from 338 records down to 58. Macro-F1 over four balanced classes is easy and says nothing.
• The problem has a ceiling. About half the tickets state their intent plainly. The rest come from pools shared by a pair of intents a triage desk genuinely confuses, from a vague pool with no signal at all, or carry a second intent in the same ticket. Without that every arm scores 1.000 and the leaderboard says nothing.
• Phone tickets are transcripts — lowercased, unpunctuated, padded with filler and missing about 45% of their words. The labels are untouched; only the surface form changes.

What this does not do

• It does not fine-tune on a GPU or serve a transformer from the demo. Training happens offline on CPU and the demo replays cached predictions, because a 512 MB shared machine cannot host the model and a demo nobody can run proves nothing.
• The embedding arms read precomputed vectors from a committed .npz. They do not run an encoder either.
• The prompted arms replay a committed cache of real OpenRouter answers. Re-running them on new text costs money and needs a key.

Those are trades, and they are the reason the demo starts in a second rather than a minute.

Development

uv sync --extra onnx   # once: the extra is what the fine-tuned arm imports
just check      # ruff format + ruff check + pyright + pytest
just bake       # the leaderboard above
just escalate   # the escalation table above
just finetune   # retrain the encoder and re-export it (needs the finetune group)
just data       # regenerate the corpus
just embed      # recompute the sentence embeddings
just demo

Python 3.12, uv for dependencies, just for commands. No test in the suite touches the network: a session fixture blocks socket connections, so a live call fails loudly. The prompted and fine-tuned arms are served entirely from their caches, and a cache miss raises rather than guessing.

quorum is the other half of this pipeline: it labels a corpus that has no labels, and bakeoff picks the model for the set that comes out. Neither imports the other.

Licence

MIT. See LICENSE.