# Create eval_manifest.json

> The file at the root of a golden set that decides which clips belong to which evaluation run.
>
> Verified against the current RoughCut app on 28 August 2026
> https://www.roughcuteditor.com/docs/eval-manifest

eval_manifest.json sits at the root of a golden set and maps run names to lists of clips. Its top-level keys are run names and every value must be an array. smoke and full are the two standard modes behind the dashboard buttons; any other valid array name — arabic, for example — appears in the Subsets menu and writes its own summary file. An entry is either a bare clip folder name or an object with clip_id plus options such as language, which accepts en, ar, or auto. Run names must be non-empty, at most 40 characters, and made only of ASCII letters, digits, underscores and hyphens. A missing list, a malformed list, or invalid JSON fails the run loudly before any clip is scored, while a named folder that does not exist is skipped as clip_dir_missing.

> ! **Advanced.** This file only matters if you are running evaluations. If you have no golden set, you need no manifest — the safe default is not to have one. The manifest costs nothing to get wrong in a destructive sense: a bad manifest fails or skips loudly, it never damages your clips. What it can cost is provider spend, if you accidentally list far more clips than you meant to and run a full evaluation.

## Where it goes

`eval_manifest.json` sits at the **root** of the golden-set folder — the same folder you connected with `Choose golden-set folder…`, alongside the per-clip subfolders. See [Build a golden set](/docs/golden-set).

## The shape of the file

The file is a JSON object. Every top-level key is a **run name**, and every value is an **array of clip entries**.

| Run name | Status | Where it appears |
|---|---|---|
| `smoke` | Standard mode | `Run smoke eval (5)` |
| `full` | Standard mode | `Run full evaluation` |
| Any other valid name | User-defined subset | The `Subsets` menu, as `Run <name> subset` |

A subset behaves exactly like a standard run and writes its own summary file, so an `arabic` list and a `full` list never overwrite each other's results. The `Subsets` menu only appears when your manifest actually defines an extra list.

Extra top-level keys that are not run lists are tolerated — the runner reads only the key for the mode you asked for.

## A complete example

```json
{
  "smoke": [
    "my_clip_01",
    "my_clip_02",
    "my_clip_03",
    "my_clip_04",
    "my_clip_05"
  ],
  "full": [
    {"clip_id": "my_clip_01"},
    {"clip_id": "my_clip_02", "language": "en"},
    {"clip_id": "my_clip_03"},
    {"clip_id": "my_clip_04"},
    {"clip_id": "my_clip_05"},
    {"clip_id": "my_clip_ar_01", "language": "ar"}
  ],
  "arabic": [
    {"clip_id": "my_clip_ar_01", "language": "ar"},
    {"clip_id": "my_clip_ar_02", "language": "ar"}
  ]
}
```

That file defines five smoke clips, six full clips, and one subset that shows up as `Run arabic subset` in the dashboard.

## Entries

An entry is one of two things.

**A bare string** — the clip's folder name, exactly as it appears on disk:

```json
"my_clip_01"
```

**An object** with `clip_id` and optional fields:

```json
{"clip_id": "my_clip_ar_01", "language": "ar"}
```

| Field | Type | Meaning |
|---|---|---|
| `clip_id` | string | The clip's folder name. Required in practice — an object without it has no folder to find |
| `language` | string | The transcription language for this clip: `en`, `ar`, or `auto` |
| `duration_s` | number | Optional bookkeeping value carried in the manifest |
| `coverage` | number | Optional bookkeeping value carried in the manifest |
| `gold_source` | string | Optional bookkeeping value carried in the manifest |

The clip ID is the folder name and nothing else — there is no path resolution, no globbing, and no fuzzy matching.

## How the language is chosen

For each clip, RoughCut takes the first of these that exists:

1. The `language` on the manifest entry.
2. A `language` value in that clip folder's own `manifest.json`, if you keep one.
3. `auto`.

Setting the language explicitly is worth doing for a mixed-language set: it removes one source of run-to-run variation from your measurement. See [Set the transcription language](/docs/language).

## Run-name rules

A run name must be:

- **non-empty**,
- **at most 40 characters**,
- made only of **ASCII letters, digits, underscore, and hyphen**.

Anything else is rejected with `mode must be 'smoke', 'full', or a subset list defined in eval_manifest.json (letters, digits, _ and - only)`. A name with a space, an accent, or Arabic characters will not run, and will not be offered in the `Subsets` menu either.

## What goes wrong, and how it tells you

| Problem | What happens |
|---|---|
| The file is not valid JSON | Decoding fails and the run stops before any clip is scored. Nothing is billed |
| The file is missing | The run fails with a message naming the expected path and pointing at the format documentation |
| The requested run name has no list | The run fails with `Eval manifest has no '<mode>' list` |
| A run name's value is not an array | The list is malformed, and the run fails loudly rather than guessing. It also never appears in `Subsets` |
| An entry names a folder that does not exist | That clip is skipped with reason `clip_dir_missing`; the rest of the run continues |
| An object entry has no `clip_id` | There is no folder to resolve, so it is skipped as `clip_dir_missing` |
| The clip folder exists but has no media | Skipped with reason `missing_media` |
| The clip folder has no `gold_v2.json` | Skipped with reason `missing_gold` |
| An unrecognised run name is requested | Rejected by the run-name rules above |
| macOS denies access to the folder | The run fails with an explicit permissions message, not a silent empty result |

Skips are always reported per clip with their reason, so an imperfect manifest still gives you a working run over the clips that are correct.

## Duplicates and ordering

Clips are processed in the order they appear in the list, and a full run applies its cap by taking the first entries — so the order of the `full` list decides which clips survive a cap of 15.

Do not list the same clip twice in one run. It gives you no new information and weights that clip twice in the run's aggregates.

## Keeping the file readable

- Keep every code sample and every clip ID in plain ASCII, left to right, even when the clips themselves are Arabic. JSON keys and folder names are technical identifiers, not display text.
- Comments are not part of JSON. Do not add them — the file will stop parsing.
- Trailing commas are not valid JSON either. This is the single most common reason a hand-edited manifest stops working.

## Common mistakes

- **Editing the manifest with an editor that adds smart quotes.** JSON needs straight quotes.
- **Listing a path instead of a folder name.** `clips/my_clip_01` is not a clip ID.
- **Assuming `full` means everything.** The full run is capped by `Full eval max clips`, default 15. Set that setting to 0 to run the whole list. See [Advanced settings reference](/docs/advanced-settings-reference).
- **Expecting a subset to appear without defining it.** The `Subsets` menu is built from your manifest, and only from lists whose names pass the run-name rules.
- **Renaming a clip folder without updating the manifest.** That is an instant `clip_dir_missing`.

## Next steps

- [Create gold_v2.json reference edits](/docs/gold-reference-edit)
- [Read evaluation metrics correctly](/docs/evaluation-metrics)
- [Build a golden set](/docs/golden-set)