Building the Video Editor Workflows P1

🎥 Video lesson — video not yet migrated (source Skool video id: fc70a6d9b8ce406d9769efc7689f0dc0).

Building the Video Editor Workflows — Audio, Transcription, Visual Analysis & Clipping

This video covers building four video editing workflows simultaneously, running parallel Claude Code conversations across a wide monitor to speed up the process. We build the core infrastructure needed to automatically analyse and clip video content.

The four workflows built in this video:

  1. Audio Analysis — extracts the audio waveform data to understand volume levels, detect silence, background noise, and speaking sections
  2. Transcription — uses Deepgram to generate word-level timestamped transcriptions from your video audio
  3. Visual Analysis — uses Gemini Pro to analyse video frames and understand what's happening on screen (tools being used, typing, screen changes, talking head vs screen share)
  4. Video Clipping — combines all three data sources to build a clip plan that automatically removes dead air, silence, breathing, and repetitive sections

How these workflows connect: Audio analysis, transcription, and visual analysis all feed into the video clipping workflow. The clip plan uses the combined data to make intelligent editing decisions — deterministic cuts for silence/noise removal, plus AI-assisted identification of repetitive or irrelevant sections.

Key architecture decisions:

  • Proxy files for large videos — the workflows use smaller proxy video files when available instead of the full-size originals to reduce processing time and cost
  • Configurable frame sampling — users choose how often Gemini analyses frames. Every 1 second for short intros, every 15-30 seconds for multi-hour recordings
  • Chunking for long videos — Exa MCP research revealed Gemini's file upload limits, so longer videos get broken into manageable sections processed by sub-agents
  • Two clipping modes — deterministic script-based cutting (silence, noise, breathing) plus AI-assisted analysis that flags repetitive sections or content that doesn't align with the original script/ICP
  • Intro vs main content — the workflow asks upfront which type of content you're editing, since intros need tighter cuts (150ms buffer) versus main content (250ms buffer)

APIs and tools set up:

  • Deepgram — audio-to-text transcription with word-level timestamps. 200 free API credits on sign-up. Add DEEPGRAM_API_KEY to your .env file
  • Gemini Pro — visual frame analysis. Uses the API key from Google AI Studio already in your .env
  • FFmpeg — audio extraction and processing. Exa MCP research found this was the most lightweight approach, reducing complexity compared to alternatives

Where Exa MCP made a real difference:

  • Found a simpler audio extraction method using FFmpeg that reduced build complexity
  • Updated the Deepgram model to the latest version (Claude's built-in knowledge had an older one)
  • Verified best practices for video frame analysis with Gemini
  • Each of these catches would have been missed without web research — this is why Exa MCP should be part of every third-party integration

Running four conversations at once:

  • Usually two parallel conversations is recommended, four is pushing it
  • Only safe to do when the workflows are completely independent and don't edit any shared files
  • Use a wide monitor or multiple screens if you're running this many simultaneously
  • Some conversations hit context window limits before completing — just resume and finish the remaining steps

Tips from the video:

  • These workflows are mostly deterministic — unlike creative content generation, audio analysis and transcription are mathematical processes that should give consistent results every time. Less AI thinking means less variable output
  • Don't over-engineer the clip plan review — for a two-hour video, the clip plan will be massive. There's no realistic way to manually review every cut. Build it to give consistent results, and if it fails, delete and regenerate rather than trying to edit
  • Always verify the agent-to-workflow mapping — after building the workflows, check that the Video Editor agent's command menu correctly points to the workflow files in the right subfolders. Some ended up in the wrong location and needed to be moved
  • Create-only workflows — transcriptions and audio analysis never need editing. If they fail, just re-run them. Don't waste time building edit functionality for outputs that are too large to manually review anyway

What's next: Film a 10-minute test video based on the content ideation we've already done, then come back and test all four workflows against real footage. Motion graphics workflows still need to be built later, but these four are enough to start testing the full video editing pipeline.

← Previous
Next →