Create eval_manifest.json

Advanced. This file only matters if you are running evaluations. If you have no golden set, you need no manifest — the safe default is not to have one. The manifest costs nothing to get wrong in a destructive sense: a bad manifest fails or skips loudly, it never damages your clips. What it can cost is provider spend, if you accidentally list far more clips than you meant to and run a full evaluation.

Where it goes

eval_manifest.json sits at the root of the golden-set folder — the same folder you connected with Choose golden-set folder…, alongside the per-clip subfolders. See Build a golden set.

The shape of the file

The file is a JSON object. Every top-level key is a run name, and every value is an array of clip entries.

A subset behaves exactly like a standard run and writes its own summary file, so an arabic list and a full list never overwrite each other's results. The Subsets menu only appears when your manifest actually defines an extra list.

Extra top-level keys that are not run lists are tolerated — the runner reads only the key for the mode you asked for.

A complete example

{
  "smoke": [
    "my_clip_01",
    "my_clip_02",
    "my_clip_03",
    "my_clip_04",
    "my_clip_05"
  ],
  "full": [
    {"clip_id": "my_clip_01"},
    {"clip_id": "my_clip_02", "language": "en"},
    {"clip_id": "my_clip_03"},
    {"clip_id": "my_clip_04"},
    {"clip_id": "my_clip_05"},
    {"clip_id": "my_clip_ar_01", "language": "ar"}
  ],
  "arabic": [
    {"clip_id": "my_clip_ar_01", "language": "ar"},
    {"clip_id": "my_clip_ar_02", "language": "ar"}
  ]
}

That file defines five smoke clips, six full clips, and one subset that shows up as Run arabic subset in the dashboard.

Entries

An entry is one of two things.

A bare string — the clip's folder name, exactly as it appears on disk:

"my_clip_01"

An object with clip_id and optional fields:

{"clip_id": "my_clip_ar_01", "language": "ar"}

The clip ID is the folder name and nothing else — there is no path resolution, no globbing, and no fuzzy matching.

How the language is chosen

For each clip, RoughCut takes the first of these that exists:

1. The language on the manifest entry.
2. A language value in that clip folder's own manifest.json, if you keep one.
3. auto.

Setting the language explicitly is worth doing for a mixed-language set: it removes one source of run-to-run variation from your measurement. See Set the transcription language.

Run-name rules

A run name must be:

• non-empty,
• at most 40 characters,
• made only of ASCII letters, digits, underscore, and hyphen.

Anything else is rejected with mode must be 'smoke', 'full', or a subset list defined in eval_manifest.json (letters, digits, _ and - only). A name with a space, an accent, or Arabic characters will not run, and will not be offered in the Subsets menu either.

What goes wrong, and how it tells you

Skips are always reported per clip with their reason, so an imperfect manifest still gives you a working run over the clips that are correct.

Duplicates and ordering

Clips are processed in the order they appear in the list, and a full run applies its cap by taking the first entries — so the order of the full list decides which clips survive a cap of 15.

Do not list the same clip twice in one run. It gives you no new information and weights that clip twice in the run's aggregates.

Keeping the file readable

• Keep every code sample and every clip ID in plain ASCII, left to right, even when the clips themselves are Arabic. JSON keys and folder names are technical identifiers, not display text.
• Comments are not part of JSON. Do not add them — the file will stop parsing.
• Trailing commas are not valid JSON either. This is the single most common reason a hand-edited manifest stops working.

Common mistakes

• Editing the manifest with an editor that adds smart quotes. JSON needs straight quotes.
• Listing a path instead of a folder name. clips/my_clip_01 is not a clip ID.
• Assuming full means everything. The full run is capped by Full eval max clips, default 15. Set that setting to 0 to run the whole list. See Advanced settings reference.
• Expecting a subset to appear without defining it. The Subsets menu is built from your manifest, and only from lists whose names pass the run-name rules.
• Renaming a clip folder without updating the manifest. That is an instant clip_dir_missing.

Next steps

• Create gold_v2.json reference edits
• Read evaluation metrics correctly
• Build a golden set