Chris Robertson / Models trained here
A model this system trained on its own data.
Every note captured in PersonalOS is triaged by an LLM into a goal, a task or a note. This student model learned those verdicts from the text alone. It runs inside the web process in under a millisecond for nothing, answers when it is confident, and leaves the rest for the LLM. Inbox triage student v1, trained 2026-10-12.
Try it
Type a note the way you would capture it. The model shows its probabilities and whether it is confident enough to answer.
Choosing where the model stops
Agreement and coverage by confidence threshold
â– agreement on answered inputs â– share of inputs answered
The cascade
The threshold is the lowest confidence at which the student agreed with the LLM at least 90% of the time, out of fold. At 0.75, it answers 28% of inputs with 95% agreement; every other input goes to the LLM, as before.
Temperature scaling (T = 0.26) rescales confidence without changing which class wins, so the threshold means what it says.
The student agrees with its teacher about 61% of the time overall, and an embedding-based variant does no better. That plateau points at how consistently the teacher separates goals from tasks, which is why corrections from the Tasks page outrank its verdicts.
Where it is right and wrong
Confusion matrix (out of fold)
| true ↓ / predicted → | goal | task | note |
|---|---|---|---|
| goal | 57 | 8 | 3 |
| task | 18 | 20 | 3 |
| note | 9 | 11 | 4 |
Per class
| class | precision | recall | F1 | examples |
|---|---|---|---|---|
| goal | 0.679 | 0.838 | 0.75 | 68 |
| task | 0.513 | 0.488 | 0.5 | 41 |
| note | 0.4 | 0.167 | 0.235 | 24 |
Macro-F1 0.495 against 0.226 for the majority-class baseline.
What was tried and not shipped
Embedding features
The same classifier on text-embedding-3-small (1536-d): accuracy 0.609, macro-F1 0.546, calibration error 0.0448. Slightly better on the rare class and better calibrated, but it needs a paid network call for every prediction. The hashed model is served.
Duplicate detection
Nearest-neighbour similarity to earlier inputs, tuned on 17 duplicates: best F1 0.339 (precision 0.238, recall 0.588). Below the 0.5 bar, so it is not served. The LLM's "duplicate" usually means "already a task or goal elsewhere", which similarity between inputs cannot see.
Data and limits
Training data
- 133 labelled inputs: 68 goal, 41 task, 24 note.
- Label sources: 133 from the LLM, 0 human corrections.
- Excluded: 24 the LLM marked unsure, 17 duplicates, 104 not yet triaged.
- Features: hashed word 1–2-grams + char 3–5-grams, 16384 buckets. Hyperparameters chosen on one cross-validation split and reported on another.
- No training text is stored in the artifact or this card; features are hashed.
Limitations
- Labels are LLM verdicts, not human ones: the student can only be as right as its teacher.
- Small dataset; cross-validated metrics carry wide uncertainty.
- Temperature and threshold are fitted on the same out-of-fold predictions they are reported on.
- English short notes only; trained on one person's writing.
Versions: v1 (current). Retrain with python3 scripts/ml/train_triage.py.