I built SherpaEdit AI to tackle the massive manual drudgery involved in documentary and film editing, with the ultimate mission of preserving the human element of storytelling while eliminating the tedious work. The traditional editing process starts with "Archaeology"—watching hundreds of hours of raw footage, interviews, and vérité to find the best moments and figure out the story arc. This is followed by creating a "Radio Cut" and a "Rough Cut," which involves assembling an audio-only story and then adding supporting visuals. My goal was specifically to automate these first two steps: Archaeology and the Rough Cut assembly, delivering an output directly to a Final Cut Pro timeline.

To solve the "Archaeology" problem, where terabytes of video are too expensive to send to a cloud-based LLM, I developed a local processing solution. For every clip, analyze_video.py creates a structured JSON manifest using local models: whisperX for transcripts with word-level timestamps, pyannote for speaker diarization, and Moondream to describe visuals every few seconds. This creates a single full_analysis.json text document that an LLM can read to pick A-roll lines and matching B-roll shots without needing access to the actual video files. Next, to solve the "Rough Cut" problem—where a single general prompt gives shallow results—I created a multi-agent AI system. This system includes a Story Agent to propose a story arc, an A-Roll Agent to pick precise narration and dialog, a B-Roll Agent to select visuals while avoiding repetition, and a Quality Check agent to review pacing and repetition.