Stats
100%
processing on device with no cloud uploads.
4
specialized story agents working together to create a story arc
human
collaboration with AI story agents is encouraged
100s
of hours of footage distilled into a rough cut
What It Does
Everything an assistant editor does — before you sit down.
SherpaEdit handles the two most tedious phases of the edit: archaeology and the rough cut.
Skip the footage archaeology
Hand over terabytes of raw footage and get back an organized, searchable picture of everything you shot — no more scrubbing through hours to find the moment you remember.
Your media stays private
The heavy analysis happens on your own machine, so your rushes never leave the building and you skip slow, costly cloud uploads.
A rough cut with a real arc
Get back an assembly built around a genuine story — not a shallow summary — with the strongest narration and dialog already chosen and matching B-roll laid in.
Ready to open in your editor
Everything arrives as an editable Premiere or Final Cut Pro sequence, so you pick up right where the tedious work ends and spend your time on the craft.
Under the Hood
I built SherpaEdit AI to tackle the manual drudgery of documentary and video editing — with the mission of preserving the human element of storytelling — while eliminating the tedious work. The pipeline breaks the job into four steps.
1. Ingests & analyzes, entirely on-device
Sending terabytes of video to a frontier LLM is far too expensive, so the heavy lifting happens locally. For every clip, analyze_clips.py builds a structured JSON manifest using local models: WhisperX for transcripts, pyannote for diarization, and Moondream to describe the visuals.
The result is a single manifest an LLM can read to pick A-roll lines and matching B-roll shots — without ever needing access to the actual video files.
2. A multi-agent system proven with thorough AI evals
In testing, a single general-purpose prompt produced shallow story arcs. So instead, I built a multi-agent system that works with any frontier LLM (Claude, Gemini). A story agent proposes an arc, which the user can refine in a conversation with the agent. Then an A-roll agent selects precise narration and dialog, a B-roll agent chooses visuals while avoiding repetition, and a quality-check agent reviews pacing and catches amateur editing mistakes. I first devised this approach based on intuition, then tested it using the evals I built in “Can frontier AI models make documentary film editorial decisions?” My experiment showed that a single agent can be very good at judging whether a clip fits when it has the surrounding context, but that shaping a good story arc from scratch requires a more thoughtful multi-agent engineering design.
3. Assembles the rough cut
With the arc decided, the agents lay down chosen A-roll and a layer with matching B-roll, automating the rough cut that an assistant editor or story producer would normally assemble by hand.
4. Hand off to your editor
The assembly is delivered as a real, editable timeline in Final Cut Pro or Premiere — the first two steps of the edit done, so you can spend your time on the craft that actually needs a human.
Built With
- Python
- WhisperX
- pyannote
- Moondream
- Gemini
- Claude
- Final Cut Pro
- Premiere
