Build a golden set

Advanced, optional, and only for evaluation. A golden set is editorial ground truth you build by hand. Ordinary rough-cut jobs never read it, so nothing here changes your everyday workflow, and the safe default is to have no golden set at all. The only real risk is wasted provider spend from running evaluations against a set that is wrong — which is why the first run should always be a smoke run.

Prerequisites

• Clips you have already edited manually, where you can still read the kept ranges off your editor timeline.
• Space on a local disk. Cloud-synced folders can be slow, and macOS may block reads outright.
• The evaluation concepts in Evaluate RoughCut against your own edits.

What the folder has to look like

One root folder. One eval_manifest.json at its root. One subfolder per clip, named with that clip's ID.

golden_set/
├── eval_manifest.json          ← which clips are in which run
├── my_clip_01/
│   ├── raw_audio.wav           ← the clip's audio
│   └── gold_v2.json            ← your reference edit
├── my_clip_02/
│   ├── source_media.mov        ← or video, if that is what you have
│   └── gold_v2.json
└── my_clip_03/
    ├── raw_audio.wav
    └── gold_v2.json

Inside a clip folder RoughCut looks for media in a fixed order: raw_audio.wav if it is there, otherwise the first source_media.* file by name. The reference edit must be gold_v2.json.

Steps

1. Create the root folder on a local disk — an external SSD or a folder in your home directory is fine, a synced cloud folder is not recommended.
2. Create one subfolder per clip, named with a short, stable clip ID: lowercase, no spaces, no private client or customer names. interview_2026_03 is a good ID; Jane Doe final v3 FINAL is not.
3. Put that clip's media in its folder as raw_audio.wav, or as source_media.mov (or another source_media.* file).
4. Write the reference edit for that clip as gold_v2.json in the same folder, listing the time ranges you kept.
5. Write eval_manifest.json at the root, listing which clips belong to the smoke run and which belong to the full run.
6. In RoughCut, open Settings → Models & Data, go to Golden set (evaluation data), and click Choose golden-set folder…. Select the folder that directly contains eval_manifest.json.
7. Confirm the label reads Access granted.
8. Open Evaluation in the sidebar and click Run smoke eval (5).
9. Read the skip list and repair whatever it names, then run smoke again.

Expected result

The Golden set (evaluation data) section shows Access granted, the Evaluation dashboard's run buttons become active, and a smoke run processes your clips and reports a per-clip table plus aggregates.

The two files you write by hand each have their own page: Create eval_manifest.json for the run lists, and Create gold_v2.json reference edits for the per-clip reference edits.

Why you grant the folder instead of typing a path

RoughCut is a sandboxed Mac App Store app. Picking the folder in the panel is what creates the permission that lets it read your clips later; the app remembers that grant across launches. Choosing the folder also fills in Eval dataset root for you, so there is no reason to type a path into that field by hand — a path without the grant is not access.

If the folder is moved or renamed, the grant no longer resolves and the run fails with Golden-set access lost — pick the folder again in Settings. Pick it again and you are back.

What RoughCut writes into your folder

On the first evaluation of a clip, RoughCut stores the clip's transcript and the AI editor's responses beside the clip, and writes a per-clip report there when the clip scores. Run summaries are written at the root of the set.

This is deliberate and useful: repeat runs replay those artifacts, so a clip you have already scored costs nothing at your provider and finishes almost immediately. Cloud-transcription runs keep their own separate transcript cache, so the two arms never replay each other's words.

The consequence is that a golden set grows over time and is not purely your own files any more. Keep that in mind when you copy or share it.

Choosing the clips

• Between five and a few dozen clips is a workable range: enough for the mean to mean something, small enough to re-run.
• Use recordings that look like your real work. A golden set of pristine studio takes tells you nothing about how the app handles your kitchen-table vlog.
• Cover the cases you care about: a clip with many false starts, one with long pauses, one with a tangent you cut, one that was nearly clean.
• Keep languages separate if you work in more than one. A dedicated subset list is better than mixing — see Create eval_manifest.json and Set the transcription language.

Be consistent about pauses and takes

The reference edit is a claim about what the right edit is. If you trim breathing pauses in one clip and leave them in the next, the scores are measuring your inconsistency as much as the app's behaviour.

Decide once: do you cut pauses, or not? Do you keep a stumbled first sentence when the retake is worse? Then apply that decision across every clip in the set.

Back it up separately

A golden set is valuable editorial ground truth. The reference edits represent hours of your judgement, and they are not reproducible from anything else — not from the source media, and certainly not from RoughCut's output. Back the folder up like the work product it is, independently of the app.

Treat it as sensitive, too. Clip folders can end up holding transcripts of the recordings, so a golden set built from client work carries client speech.

Common mistakes

• Keeping the set in a cloud-synced folder. macOS can deny the app access, and the run fails with an explicit permissions error instead of scoring anything.
• Pointing the picker at the parent of the golden set. Select the folder that directly contains eval_manifest.json.
• Naming clip folders and manifest entries differently. The manifest entry is the folder name. A mismatch is a clip_dir_missing skip.
• Putting the media in the root instead of the clip folder. Media is resolved inside each clip folder only.
• Naming media something else. audio.wav is not raw_audio.wav and is not source_media.*; it will not be found.
• Using client names as clip IDs. IDs appear in run summaries and traces. Keep them neutral.

Troubleshooting

Skips are never silent — every one is recorded with its reason so you can repair the set incrementally. Per-clip failures are listed separately with the error text.

If the manifest itself cannot be read, see Create eval_manifest.json. For anything else, Troubleshooting by symptom.