Stats

human

collaboration with AI story agents is encouraged

4

specialized story agents working together to create a story arc

100%

processing on device with no cloud uploads.

100s

of hours of footage distilled into a rough cut

What It Does

Your AI assistant editor. You direct the story. SherpaEdit does the heavy lifting.

SherpaEdit handles the two most tedious phases of the edit: archaeology and the rough cut. It’s built to work with you, taking your creative input and doing the manual labor so you can focus on the craft.

Skip the footage archaeology

Hand over terabytes of raw footage and get back an organized, searchable picture of everything you shot. No more scrubbing through hours to find the moment you remember.

Your media stays private

The heavy analysis happens on your own machine, so your rushes never leave the building and you skip slow, costly cloud uploads.

A rough cut shaped by your vision

SherpaEdit doesn’t make story decisions in a black box. You collaborate with the AI to refine a genuine story arc, and it lays down the strongest dialogue and matching B-roll based on your direction.

Ready to open in your edit software

Everything arrives as an editable Premiere, Final Cut Pro, or Avid sequence. The groundwork is done, leaving you in total creative control to refine and polish the edit.

Under the Hood

I built SherpaEdit AI to tackle the manual drudgery of documentary and video editing — with the mission of preserving the human element of storytelling. It acts as an intelligent assistant editor that executes your creative vision. The pipeline breaks the job into four steps:

1. Ingests & analyzes, entirely on-device

Sending terabytes of video to a frontier LLM is far too expensive, so the heavy lifting happens locally. For every clip, analyze_clips.py builds a structured JSON manifest using local models: WhisperX for transcripts, pyannote for diarization, and Moondream to describe the visuals.

The result is a single manifest an LLM can read to pick A-roll lines and matching B-roll shots — without ever needing access to the actual video files.

2. A multi-agent system, guided by you, proven with thorough AI evals

A single general-purpose prompt produces shallow story arcs. Instead, I built a multi-agent system that works with any frontier LLM (Claude, Gemini) that acts like a team of specialized assistant editors working under your direction:

• A Story Agent proposes an arc, which you refine through a direct conversation to match your vision.
• The A-Roll Agent selects precise narration and dialog based on that agreed narrative.
• The B-Roll Agent adds matching visuals while avoiding repetition.
• The Quality-Check Agent reviews pacing and flags amateur editing mistakes.

I first devised this approach based on intuition, then tested it using the evals I built in “Can frontier AI models make documentary film editorial decisions?” My experiment showed that a single agent can be very good at judging whether a clip fits when it has the surrounding context, but that shaping a good story arc from scratch requires a more thoughtful multi-agent engineering design.

3. Assembles the rough cut

With the arc decided, the agents lay down chosen A-roll and a layer with matching B-roll, automating the initial assembly that an assistant editor or story producer would normally have to do by hand.

4. Hand off to the editor

The assembly is delivered as a real, editable timeline. SherpaEdit completes the first two manual steps with you, so you can spend your energy on the actual craft of editing.

Built With

• Python
• WhisperX
• pyannote
• Moondream
• Gemini
• Claude
• Final Cut Pro
• Premiere