I built SherpaEdit, a multi-agent system that takes raw documentary footage and builds a rough cut: it develops the narrative arc, picks clips, adds B-roll, and reviews the result. A lot of my design decisions behind the system were made on intuition, and I wanted to know whether they were good.

So I decided to build two evals in Python (here’s the source code). The first asked a simple question: given a gap in an edit project, how good is a model at deciding which clip belongs there? It’s a discrimination task, and if a model is good at it, it's useful as a quality control agent to confirm that an editbot chose the right shots.

The second eval asked a more interesting question: if a model knows the entire story, does it do a better job? Recognizing that a clip fits and inventing the arc of a film are not the same skill.

The reference film

I had two useful things lying around: all the raw footage from an adventure documentary I made and the finished, edited film. I diarized all of the raw footage and the final film, segmenting them into "moments" (a contiguous run of speech from one speaker, split on speaker changes or silence gaps), and then matching every raw moment against the final film's moments using rapidfuzz. That made it possible to determine which moments were used in the final film and which were cut. Editors tend to use fragments of longer takes, so fuzzy matching made the system more accurate.

The finished dataset:

Eval #1: Which clip fits here?

Eval #1’s design was simple: remove a moment from the finished film, then ask a model to pick which of six clip candidates fits there. From there, I varied how much context the model was allowed to see:

• candidates_only: the six options and nothing else
• position_only: the candidates, plus "this gap falls at 62% of the runtime"
• preceding_only: the candidates, plus the 90 seconds of transcript before the gap
• full: the candidates, plus 90 seconds before and after

I also tested two decoy tiers. In Tier A, all six candidates were real footage that made the film, which isolated the question, “is this the right position for this clip?” In Tier B, five of the decoys are real footage that got cut, forcing the model to judge both quality and placement at the same time. Candidate order was shuffled per question, which matters later.

Results

Tier A (every candidate is known-good footage). Chance is 16.7%, Wilson 95% confidence intervals in brackets, n=99 per cell:

Tier B (five decoys are footage the editor actually rejected), n=84 per cell:

The curve is steep and it replicates across all four models, which is about as good as an eval based on a single film can provide. With no context at all, every model sits at or below chance on Tier A. That makes sense: when all six options are genuinely good footage, there is no way to guess which one belongs in a given slot without knowing where the slot is in the film. But if you give the models surrounding context, accuracy climbs by 40 to 70 points.

Tier B is the reverse. Both Gemini models hit 43% with zero context whatsoever, so when five of six options are footage the editor threw out, raw quality alone is a strong enough signal to get clear of chance without knowing anything about the film. But the number that matters is the last one: 84 to 87 percent accuracy at picking the right clip for a slot, when the model can see all surrounding context, provides a very useful second opinion on a draft edit.

What I found especially interesting: having preceding_only context gets you most of the way there, landing within a few points of it for three of four models in Tier B. An agent that only knows what it has previously placed does nearly as well.

Note: The Haiku model picked choice B three times as often as choice A, and Gemini Pro had a slight lean toward A and D. Because I shuffled the orders of the choices, none of the bias unfairly penalized a model, but it did mean that some fraction of a skewed model's answers are coming from a choice preference rather than from reading the question!

The five questions no model got right

Five of the 183 questions I posed were never answered correctly by any model under any condition, and when I sat down to read them the pattern was immediately recognizable to anyone who has edited documentaries: they were all quick soundups.

Usually the context was an interview bite, then a cut to a quick sound-up, and then back to the interview bite. The soundup was there because of something happening visually on screen at that moment, not because it related to the ongoing interview narration.

For an eval based completely on transcribed video, this wasn’t a surprise, since the models were not shown any visuals.

Eval #2: does building an arc change everything?

In many ways, Eval #1 was a naive test: when a real editor sits down to make a radio cut, they’re starting with only the raw footage, with no existing structure or context to help them put raw moments into slots. SherpaEdit has a quality control agent, and what I learned from Eval #1 proved that, designed correctly, it can work surprisingly well.

But SherpaEdit composes its initial rough cut using a completely different method: an agent is given all of the raw footage transcripts (all context) and asked to construct a story arc. Then, another agent is tasked with building each story beat in the arc by selecting the right moments.

For Eval #2, I took all 106 moments in the finished film, shuffled them up in a random order, and asked models to recreate a sensible rough cut by proposing an order. Then, I scored the result as +1 is identical, 0 is random, and -1 is exactly backwards (tau), with an empirical p-value from 2,000 random-shuffle trials so I knew what better than random meant.

I ran each model five times on an identical prompt, which turned out to matter more than I expected.

Every valid run scored a p-value<0.0005 against the random-shuffle baseline, so the models recover meaningful ordering information from the footage, but remain far from reproducing the reference cut’s structure. A mean displacement of 20 to 24 positions out of 106 means the typical moment lands roughly a fifth of the film away from the reference.

The range column is particularly interesting. The spread within a single model across runs (up to 0.081) is wider than the gap between the model means (0.048). I had originally run this once per model, and that single run told a clean story: Flash won at 0.455, Opus came second at 0.402, Gemini Pro trailed at 0.316. But doing five runs dissolved most of that. Flash and Opus land at 0.413 and 0.415, a tie. Flash’s single run happened to be its best of five, and Pro’s happened to be its worst. If you’re going to make an architecture decision on an eval, run it more than once first.

Gemini Pro comes in last, trailing Flash by 0.047 and Opus by 0.048. With only three to five valid runs per model, neither gap is solid enough to call a real difference on its own, but Pro is at the bottom consistently. Since Pro was the best model in Eval #1 by a comfortable margin, hitting 84% and 87%, that's at least a hint that recognition and composition are distinct capabilities: being excellent at "does this moment fit here" doesn't mean a model is equipped to answer “what's the story of this film.”

When did they get it wrong?

The film I was testing against included some archival footage not tightly bound to the film’s story spine, positioned at the end of the film for dramatic effect. Gemini Pro moved the closing material to the beginning. Opus put it near the end in two of its three valid runs, and Flash was a coin flip. Flash’s own error ran in the other direction: reflective archival material the editor placed early got pushed to the very end.

So given only a one-paragraph premise, models often fall back on generic structure instead of deriving it from the footage. If you are looking for novelty or stylistic flair, you need to give specific instructions about the story and edit to the model. The distinction between the two failure types matters for that, though: a consistent bias like Pro’s is something an explicit instruction can fix, while it’s harder to prompt your way out of a coin flip.

Again, soundups are confusing

The moments no model could place in Eval #2 were — you guessed it — soundups again. The models saw them as throwaway lines, assumed they were outtakes, and buried them. The human editor put them early because of what was on screen.

Haiku didn’t finish the assignment, and neither did Opus twice

Haiku’s response was correct JSON but it used four clips twice while quietly omitting two others. That happened on all five runs, so it isn’t a fluke: at 106 items, the small fast model could not satisfy “use each item exactly once.” Opus failed less often and more strangely, inventing an ID that was never in the candidate list, on two of its five runs. That’s the more unsettling failure, because a made-up identifier in a real pipeline points at a clip that doesn’t exist.

What this means for building a film editing bot

1. Don't build an auto-editor as a sequential fill-forward loop. Placement works as a quality-control layer, not as a first-editor. ~87% accuracy at "does this clip belong in this slot, given its neighbors" is strong enough to check a draft. But the two conditions that make it work, bidirectional context and a short candidate list, both exist after a first draft and neither exists before one.
2. Make the story arc specific, because the model won't. Eval #2 shows models defaulting to generic structural conventions whenever the guidance is thin, and getting specific editorial choices confidently wrong. If the arc-development stage doesn't explicitly resolve questions like whether the film builds toward a thematic payoff or opens with the stakes, the generation step will answer them for you, conventionally.
3. Best at placement is not best at structure. Gemini Pro topped eval 1 but couldn’t beat Gemini Flash in eval 2. If you're picking models for a multi-agent pipeline, evaluate each agent against its actual job, because a single leaderboard won't tell you which model belongs where.

What's next

First, I want to expand these evals beyond a single reference film. The current results measure how closely models reproduce one professional editor’s completed cut, not objective editorial quality. Testing across many films, genres, and editing styles would make the results more generalizable, while human evaluation of model-generated edits could help distinguish between a model making a genuinely bad choice and simply making a different, defensible one.

Second, I want to combine these experiments into a single cohesive eval that scores the core editing decisions separately: which moments a model selects, how it orders them, and where it chooses to start and end each one. That would give me a much more useful way to diagnose where an editing system succeeds or fails.

Those measurements could then drive the architecture of a better automatic editor. One agent could identify the strongest moments, another could turn them into an explicit story arc, a third could assemble the material against that arc, and a final agent could review the cut for pacing, coherence, repetition, and other quality problems. Instead of asking one model to “edit the film,” the system could treat editing as a set of distinct problems and use the evals to improve each one independently.