# Read evaluation metrics correctly

> What Word-F1, the baselines, the fixup estimate, degenerate flags, and the regression flag actually tell you.
>
> Verified against the current RoughCut app on 28 August 2026
> https://www.roughcuteditor.com/docs/evaluation-metrics

Word-F1 is the primary metric — a duration-weighted measure of agreement on which words get removed, where higher is better. Interval F1 measures the same agreement in time rather than words. Never read either in isolation: compare them against the remove-all, keep-all, and silence-only baselines, and against a previous run of the same kind. The delta over remove-all shows whether the model beats simply cutting everything. Fixup per 30 minutes estimates the correction work left over from false-positive cuts, missed cuts, and boundary nudges. Degenerate clips are flagged and excluded from aggregates. A drop of more than three points in mean Word-F1 against the previous comparable run raises a regression flag, which flags and never blocks. Cloud runs are only comparable with other cloud runs.

> ! **Advanced.** These numbers are a tool for comparing RoughCut against itself under controlled changes. They are not a product quality rating, and a score from your golden set does not transfer to anyone else's footage. Read the whole page before drawing a conclusion from a single figure; the fastest way back to solid ground is always the same — re-read the baselines next to the score.

## Where the numbers appear

After a run, the dashboard shows a `Results` card with aggregate statistics, then a per-clip table.

| Element | What it is |
|---|---|
| `Word-F1 mean` / `Word-F1 median` | The primary metric, averaged over non-degenerate clips |
| `Interval mean` | The time-interval version of the same agreement |
| `Fixup mean` | Estimated correction minutes per 30 minutes of source |
| `Remove-all base` | The mean score of the trivial "cut everything" baseline, for comparison |
| `Gold fingerprint` | A short hash identifying the exact reference set this run scored against |
| `Transcription` | Present on cloud runs, with the reminder to compare only against other cloud runs |
| Per-clip `Clip`, `Word-F1`, `Δ remove-all`, `Fixup/30m`, `Flags` | One row per scored clip |
| `degenerate` and `cached` flags | Why a clip was left out of the aggregates, and whether it replayed from cache |

Skipped clips and per-clip failures are listed separately, each with its reason.

## Word-F1 — the primary metric

Word-F1 measures agreement about **which words get removed**, weighted by how long each word lasts, and reported as a percentage where higher is better.

Two details matter for interpreting it:

- It is **duration-weighted**, so disagreeing about a long word costs more than disagreeing about a short one. This is deliberate: what a viewer notices is seconds, not tokens.
- A word counts as removed when **less than 50% of its duration overlaps the kept ranges**. Boundaries that land in roughly the right pause agree; boundaries in the wrong pause do not.

Word-F1 is an agreement score against your reference edit, and nothing more. It is not a measure of whether the resulting cut is good.

## Interval F1 — the same question, in time

Interval F1 compares the **removed time ranges** rather than the words: precision, recall and F1 in remove-space, plus a keep-side overlap measure.

It answers a slightly different question — did we cut the same parts of the timeline — and is the metric that carries the run when word labels are not usable. Where the two metrics disagree sharply, the usual cause is many short disagreements clustered in a few places.

## Always read a baseline beside the score

A score with no baseline is not a result. Every run computes three:

| Baseline | What it does | What it tells you |
|---|---|---|
| `remove_all` | Keeps nothing | The score a system gets by cutting the entire clip. If your gold removes most of the clip, this baseline is high |
| `keep_all` | Removes nothing | The score for doing nothing at all |
| `silence_only` | Removes exactly the detected silences | The score with no AI editing pass involved |

**Delta over remove-all** is the per-clip column `Δ remove-all`: the run's Word-F1 minus the remove-all baseline's F1. A model that cannot beat "cut everything" is not editing, it is deleting. That delta, not the raw score, is the honest headline for a single clip.

The `silence_only` baseline is the one to watch when you are evaluating whether the AI editing pass earns its cost on your material. If a full pipeline run barely beats silence-only on your clips, [Bad-take removal](/docs/bad-take-removal) is not doing much for that kind of footage.

## Fixup per 30 minutes

`Fixup/30m` estimates how much correction work a run leaves behind, in minutes per 30 minutes of source. It is built from three kinds of error, each with a different cost:

| Error | What it is | Relative cost |
|---|---|---|
| False-positive cut | A cut with no counterpart in your reference edit | Moderate — you notice it and restore it |
| Missed cut | Something your reference edit removed that survived | Highest — you have to find it yourself |
| Boundary nudge | A matched cut whose start or end is off by more than about a tenth of a second | Lowest — a drag on the timeline |

Missed cuts are weighted most heavily because finding a problem is more expensive than fixing one you can see. Lower is better, and the number is an estimate of effort, not a measurement of it.

## Degenerate clips

Some clips produce a score that is arithmetically valid and editorially meaningless. Those are flagged `degenerate` and **excluded from the aggregates**, while still being fully scored and shown in the table.

A clip is degenerate when any of these is true, checked in this order:

1. The predicted edit **kept nothing** — everything was removed.
2. The predicted edit **removed nothing**.
3. Your **gold removed nothing**, so there is no removal ratio to compare against.
4. The ratio of removed time, prediction against gold, falls outside the range **0.5 to 2.0** — one side removed less than half, or more than twice, what the other did.

A cluster of degenerate flags is almost always a golden-set problem, not a model problem. Cases 2 and 3 usually mean a reference edit that keeps the whole clip; case 4 usually means a timebase mistake. Check [Create gold_v2.json reference edits](/docs/gold-reference-edit) first.

## The regression flag

When a run's mean Word-F1 falls **more than 3 points** below the previous run of the same kind, the run is flagged:

`Regression flagged: word-F1 mean dropped more than 3 points vs the previous run.`

Two things to understand about it:

- **It flags, it never blocks.** The summary is written either way; nothing is discarded and nothing is prevented.
- **It only compares like with like.** Runs of different modes are never diffed against each other, because the clip populations differ and the delta would be a lie.

Treat the flag as a prompt to investigate — what changed between the runs, and was it the setting you meant to change — not as a verdict.

## Cloud runs are a separate universe

If you ran with the Pro `Cloud transcription (whisper-1)` toggle on:

- The transcript came from a different model, which tokenises differently, so the word labels are projected from intervals rather than matched word for word.
- The results are written to **separate summary files** with a cloud suffix, and use a separate transcript cache, so your on-device history stays intact.
- The dashboard prints `Transcription: <source> — compare only against other cloud runs`.

**Compare cloud to cloud and on-device to on-device.** A cloud-versus-on-device difference in Word-F1 is dominated by label projection, not by transcription quality. See [Transcription options](/docs/transcription-options).

## Arabic and other scripts

Absolute word matching reads pessimistically on Arabic material. Dialect speech transcribed towards Modern Standard Arabic orthography produces word forms that do not match token for token, even when the editorial decision was identical — and the reference edits for such sets tend to under-keep by construction.

The response is not to distrust the system but to change what you read: **trust controlled deltas and the baselines**, not the absolute figure. Compare an Arabic run to the previous Arabic run with one setting changed, and look at `Δ remove-all`. See [Set the transcription language](/docs/language).

## The gold fingerprint

Each summary records a short hash computed over the reference files of every clip that actually scored, in manifest order. It identifies the exact reference set the run was measured against.

Its purpose is to stop dishonest comparisons. If you edit a `gold_v2.json` between two runs, the fingerprint changes, and the two runs are no longer measuring the same thing — a difference between them may be your gold moving rather than the app's behaviour. Comparisons refuse to trust deltas across different gold, and warn loudly when one side has no fingerprint at all.

Before comparing two runs, check that the fingerprints match. If they do not, the honest answer is that you have no comparison, only two separate measurements.

## What not to conclude

- **Do not read a mean as a grade.** Two golden sets built by two editors will produce different numbers for identical app behaviour, because the reference edits differ.
- **Do not compare your numbers to figures printed in the dashboard header.** Those anchors were measured on a different golden set with different clips and different reference edits; the only anchor that means anything for you is your own previous run.
- **Do not quote a score as a product claim.** It is agreement with one person's editing style on one set of clips.
- **Do not chase a tenth of a point.** The unchanged band in run-to-run comparison exists because small differences are noise.

## Common mistakes

- **Reading the aggregate without the clip table.** One catastrophic clip can move a mean of fifteen.
- **Ignoring the degenerate count.** If half the set was excluded, the mean describes the other half.
- **Comparing across modes.** Smoke and full contain different clips.
- **Changing two things at once.** One variable per run, or the delta tells you nothing.
- **Comparing runs with different gold fingerprints.** That is not an A/B.

## Next steps

- [Connect Langfuse tracing](/docs/langfuse) to keep run history and scores in a dashboard over time.
- [Personalise editing with exemplars](/docs/exemplars) — one of the few changes that can move these numbers.
- [Build a golden set](/docs/golden-set) if the numbers point at your reference data rather than the app.