# Build a golden set

> How to lay out the folder of clips and reference edits that RoughCut's evaluation system measures against.
>
> Verified against the current RoughCut app on 28 August 2026
> https://www.roughcuteditor.com/docs/golden-set

A golden set is one root folder holding an eval_manifest.json and one subfolder per clip. Each clip folder needs source media — raw_audio.wav, or the first source_media file — plus a gold_v2.json reference edit. Keep the set on a local disk, because cloud-synced folders can be slow or blocked by macOS permissions, and connect it through Settings → Models & Data → Choose golden-set folder… rather than typing a path, because that grant is what lets the sandboxed app read the folder at all. Use short, stable clip IDs with no private client names, and make your reference edits consistent about pauses and takes. On the first evaluation RoughCut writes transcript and AI cache artifacts beside each clip so repeat runs are free and instant. Anything missing is skipped with a named reason, never silently.

> ! **Advanced, optional, and only for evaluation.** A golden set is editorial ground truth you build by hand. Ordinary rough-cut jobs never read it, so nothing here changes your everyday workflow, and the safe default is to have no golden set at all. The only real risk is wasted provider spend from running evaluations against a set that is wrong — which is why the first run should always be a smoke run.

## Prerequisites

- Clips you have already edited manually, where you can still read the kept ranges off your editor timeline.
- Space on a **local disk**. Cloud-synced folders can be slow, and macOS may block reads outright.
- The evaluation concepts in [Evaluate RoughCut against your own edits](/docs/evaluation-overview).

## What the folder has to look like

One root folder. One `eval_manifest.json` at its root. One subfolder per clip, named with that clip's ID.

```
golden_set/
├── eval_manifest.json          ← which clips are in which run
├── my_clip_01/
│   ├── raw_audio.wav           ← the clip's audio
│   └── gold_v2.json            ← your reference edit
├── my_clip_02/
│   ├── source_media.mov        ← or video, if that is what you have
│   └── gold_v2.json
└── my_clip_03/
    ├── raw_audio.wav
    └── gold_v2.json
```

Inside a clip folder RoughCut looks for media in a fixed order: `raw_audio.wav` if it is there, otherwise the first `source_media.*` file by name. The reference edit must be `gold_v2.json`.

## Steps

1. Create the root folder on a local disk — an external SSD or a folder in your home directory is fine, a synced cloud folder is not recommended.
2. Create one subfolder per clip, named with a short, stable clip ID: lowercase, no spaces, no private client or customer names. `interview_2026_03` is a good ID; `Jane Doe final v3 FINAL` is not.
3. Put that clip's media in its folder as `raw_audio.wav`, or as `source_media.mov` (or another `source_media.*` file).
4. Write the reference edit for that clip as `gold_v2.json` in the same folder, listing the time ranges you kept.
5. Write `eval_manifest.json` at the root, listing which clips belong to the `smoke` run and which belong to the `full` run.
6. In RoughCut, open `Settings → Models & Data`, go to `Golden set (evaluation data)`, and click `Choose golden-set folder…`. Select the folder that directly contains `eval_manifest.json`.
7. Confirm the label reads `Access granted`.
8. Open `Evaluation` in the sidebar and click `Run smoke eval (5)`.
9. Read the skip list and repair whatever it names, then run smoke again.

## Expected result

The `Golden set (evaluation data)` section shows `Access granted`, the Evaluation dashboard's run buttons become active, and a smoke run processes your clips and reports a per-clip table plus aggregates.

The two files you write by hand each have their own page: [Create eval_manifest.json](/docs/eval-manifest) for the run lists, and [Create gold_v2.json reference edits](/docs/gold-reference-edit) for the per-clip reference edits.

## Why you grant the folder instead of typing a path

RoughCut is a sandboxed Mac App Store app. Picking the folder in the panel is what creates the permission that lets it read your clips later; the app remembers that grant across launches. Choosing the folder also fills in `Eval dataset root` for you, so there is no reason to type a path into that field by hand — a path without the grant is not access.

If the folder is moved or renamed, the grant no longer resolves and the run fails with `Golden-set access lost — pick the folder again in Settings.` Pick it again and you are back.

## What RoughCut writes into your folder

On the first evaluation of a clip, RoughCut stores the clip's transcript and the AI editor's responses beside the clip, and writes a per-clip report there when the clip scores. Run summaries are written at the root of the set.

This is deliberate and useful: **repeat runs replay those artifacts, so a clip you have already scored costs nothing at your provider and finishes almost immediately.** Cloud-transcription runs keep their own separate transcript cache, so the two arms never replay each other's words.

The consequence is that a golden set grows over time and is not purely your own files any more. Keep that in mind when you copy or share it.

## Choosing the clips

- **Between five and a few dozen clips** is a workable range: enough for the mean to mean something, small enough to re-run.
- **Use recordings that look like your real work.** A golden set of pristine studio takes tells you nothing about how the app handles your kitchen-table vlog.
- **Cover the cases you care about**: a clip with many false starts, one with long pauses, one with a tangent you cut, one that was nearly clean.
- **Keep languages separate** if you work in more than one. A dedicated subset list is better than mixing — see [Create eval_manifest.json](/docs/eval-manifest) and [Set the transcription language](/docs/language).

## Be consistent about pauses and takes

The reference edit is a claim about what the right edit is. If you trim breathing pauses in one clip and leave them in the next, the scores are measuring your inconsistency as much as the app's behaviour.

Decide once: do you cut pauses, or not? Do you keep a stumbled first sentence when the retake is worse? Then apply that decision across every clip in the set.

## Back it up separately

A golden set is valuable editorial ground truth. The reference edits represent hours of your judgement, and they are not reproducible from anything else — not from the source media, and certainly not from RoughCut's output. Back the folder up like the work product it is, independently of the app.

Treat it as sensitive, too. Clip folders can end up holding transcripts of the recordings, so a golden set built from client work carries client speech.

## Common mistakes

- **Keeping the set in a cloud-synced folder.** macOS can deny the app access, and the run fails with an explicit permissions error instead of scoring anything.
- **Pointing the picker at the parent of the golden set.** Select the folder that directly contains `eval_manifest.json`.
- **Naming clip folders and manifest entries differently.** The manifest entry is the folder name. A mismatch is a `clip_dir_missing` skip.
- **Putting the media in the root instead of the clip folder.** Media is resolved inside each clip folder only.
- **Naming media something else.** `audio.wav` is not `raw_audio.wav` and is not `source_media.*`; it will not be found.
- **Using client names as clip IDs.** IDs appear in run summaries and traces. Keep them neutral.

## Troubleshooting

| Skip reason | Meaning | Fix |
|---|---|---|
| `clip_dir_missing` | The manifest names a folder that is not in the set, or the entry has no usable ID | Correct the entry, or create the folder |
| `missing_media` | No `raw_audio.wav` and no `source_media.*` in the clip folder | Add the media with one of those names |
| `missing_gold` | No `gold_v2.json` in the clip folder | Write the reference edit |

Skips are never silent — every one is recorded with its reason so you can repair the set incrementally. Per-clip failures are listed separately with the error text.

If the manifest itself cannot be read, see [Create eval_manifest.json](/docs/eval-manifest). For anything else, [Troubleshooting by symptom](/docs/troubleshooting).