Video Editing Plugin Overview

🎥 Video lesson — video not yet migrated (source Skool video id: aa8ac9b848ae47818186f789c437956a).

The full 10-stage video editing pipeline: watch the video, then read the lesson.

This module is built around a 25-minute YouTube video that walks through the complete APG video editing pipeline, stage by stage: video ingest, audio analysis, transcription, visual analysis, clipping, storyboarding, B-roll, Hera motion graphics, Remotion editing, and the 41-component template library. The system has produced every video on the channel for two months. Under $3 per video. No Descript, no manual editing, no human editor.

The written lesson reinforces what you watched. It structures the data flow between stages, covers the architectural decisions in detail, and gives you a reference you can scan during Module 2 when you start customising the pipeline to your own editing style.

What you'll learn → What the pipeline produces: before/after, cost, and 2 months of production proof → Video Ingest [VI]: ffprobe metadata, proxy vs raw architecture, registry YAMLs, hardlinks → Audio Analysis [AA]: 4-layer analysis including Silero VAD at 20ms resolution → Transcription [TR]: DeepGram Nova-3 with word-level timestamps to 10ms → Visual Analysis [VA]: Gemini 2.5 Pro frame classification, runs in parallel → Video Clipping [VC]: script alignment, retake detection, VAD boundary refinement → Storyboard [SB]: 7 steps, 41 template components, MG trigger rules → B-Roll [BE] and Hera Motion Graphics [HM]: parallel asset generation → Remotion Edit [RE]: 8-step workflow, the 510ms drift lesson, per-clip Audio → Template Library: 41 React components across 7 categories

Next up: Video Editing Plugin, Module 2 — hands-on customisation

PART 2 — LESSON BODY (full markdown, paste into Skool lesson page)

The Video Editing Pipeline

Watch the YouTube video first. This lesson is the written companion: it structures the data flow, explains the architectural decisions, and gives you a reference to scan during Module 2.

The video: https://www.youtube.com/watch?v=bDCLT4Je6WA (~25 minutes)

Once you have watched it, come back here and read through the stages below. By the end you should be able to look at the diagram and explain every stage and every data handoff.

What this pipeline produces

A raw recording goes in. A published YouTube video comes out.

Before: Bad takes, dead air, no graphics, 42 minutes of unedited footage. After: Clean edits, motion graphics, captions, B-roll cutaways, published to YouTube.

Cost: under $3 per video across all API services (DeepGram, Gemini, Hera combined).

Production proof: 2 months, every video on the channel, 15,000 views per week. This is not a demo. It is the system that produced the video you just watched.

The architecture: why 10 stages

Two decisions drive the whole design.

1. Proxy vs raw

Every analysis task runs on a 720p proxy, not the 4K source. DeepGram receives 200MB instead of 3GB. Gemini frame extraction is fast. The entire analysis pipeline runs in minutes. The raw 4K file is only touched for final delivery in the Remotion render. You get full quality output without paying analysis costs at full resolution.

2. Three independent data streams

After ingest, the pipeline forks into three parallel branches that run independently:

  • Audio Analysis [AA] classifies every millisecond of audio
  • Transcription [TR] produces word-level timestamps
  • Visual Analysis [VA] classifies every frame

All three converge at Video Clipping [VC]. Neither depends on the others to start. The pipeline does not wait.

Why modular stages?

Each stage has one job and one output. When something breaks, you know exactly which stage to debug. When a tool changes (DeepGram releases a better model, Hera updates its API), you swap one stage without touching the rest. When two stages have no dependency, they run in parallel.

Stage 1: Video Ingest [VI]

The pipeline starts with ffprobe. It reads the raw file and extracts metadata as JSON: resolution, codec, frame rate, file size, duration. This metadata drives every downstream decision, starting with proxy generation.

Proxy generation: FFmpeg transcodes the raw file to 720p. Every API call from this point forward hits the proxy. The raw file sits untouched until the final render.

Registry YAMLs: Every clip gets a YAML file tracking its ID, role (proxy or raw), content type (body, intro, outro), the path to its paired file, and its processing status. This is how the pipeline knows what it has and what it still needs to do.

Hardlinks, not symlinks: Remotion's webpack dev server cannot serve symlinked files using HTTP range headers. You get a media playback error. Hardlinks fix it. This is the kind of thing you only learn after debugging for two hours at midnight.

Stage 2: Audio Analysis [AA]

Most AI video editing tools skip this stage. That is why their edits have bad cuts: they are cutting on volume, not on speech. This pipeline runs four layers.

Layer 1: FFmpeg silencedetect

A simple volume threshold: anything below -35 dBFS for longer than 150ms is silence. This catches obvious dead air. It does not distinguish a breath from silence, or background hiss from actual quiet.

Layer 2: dB waveform at 20ms windows

FFmpeg extracts raw PCM audio. RMS dB is calculated per 320-sample window, which is 20ms at 16kHz. This produces a loudness map of the entire video at 20ms resolution. You can see exactly where every silence, every breath, and every burst of speech falls on the timeline.

Layer 3: Silero VAD

A neural network that outputs speech probability at 20ms resolution. It runs locally via Python and PyTorch. No API call, no cost.

FFmpeg says "below -35 dBFS, must be silence." Silero says "speech probability 0.87 despite the background noise." That difference is what makes the cuts clean.

Layer 4: RNNoise denoising

A highpass filter at 80Hz removes HVAC rumble and traffic. The arnndn neural network filter removes remaining background noise. This denoised audio is only used for analysis. The original audio is untouched. You never want to denoise what the viewer hears, only what the pipeline hears.

Output: Every millisecond of audio classified as SPEECH, SILENCE, BREATH, or NOISE, with region boundaries, average dB, and average speech probability.

Stage 3: Transcription [TR]

DeepGram Nova-3 with word-level timestamps to approximately 10ms. Each word in the transcript has a start time, end time, confidence score, and speaker label.

Keyterm boosting: Domain-specific vocabulary (Remotion, ffprobe, Supabase) can be boosted so the model does not guess wrong on technical terms.

Why 10ms precision matters: The storyboard and caption timing depend on knowing exactly when every word starts and ends. VAD-calibrated caption timing pins each caption to the VAD speech onset, not the transcript timestamp. If the transcript is imprecise, every caption is off by the same amount.

Cost: Under $3 per video across all services. DeepGram is the largest single cost. Gemini and Hera fill the rest.

Stage 4: Visual Analysis [VA]

FFmpeg extracts frames from the proxy video. Body content: one frame every five seconds (0.2 FPS). Intro: one frame per second (1.0 FPS). The intro gets higher density because it needs more visual variety and tighter B-roll placement.

Each frame is sent to Gemini 2.5 Pro with structured output. Gemini classifies:

  • Scene type: talking-head, screen-share, mixed, B-roll, transition
  • Tools or software visible on screen
  • Activity description
  • B-roll extraction opportunities

Output: visual-analysis.json, a timeline of scene types with timestamps.

This stage runs in parallel with Audio Analysis and Transcription. While FFmpeg detects silence and Silero classifies speech, Gemini is looking at every frame and deciding what is on screen.

Stage 5: Video Clipping [VC]

All three data streams converge here. This is where the pipeline starts editing.

Phase 1: Transcript alignment

The transcript words array aligns against the original script. Retakes are detected where the speaker said the same section twice. The pipeline identifies the better take and marks the other for removal. Filler-heavy passages and long mid-sentence pauses are flagged.

Phase 2: Audio boundary refinement

The transcript says a word starts at 14.230 seconds. The VAD speech onset is detected at 14.215 seconds. That 15ms difference is what makes a cut clean versus clipping the beginning of the word.

Every cut point snaps to the VAD onset, not the transcript timestamp. Content-type buffers apply: 300ms for body content, 150ms for the intro where pacing is tighter.

Phase 3: FFmpeg clipping

Both proxy and raw get clipped using the same cut list. Word timestamps remap from the original timeline to the clipped timeline using pure math. No re-transcription needed.

Stage 6: Storyboard [SB]

Seven steps from an approved clip list to an approved production timeline.

StepWhat it does1Scope review: intro, body, or full video?2Production brief: identify all available visual assets3Speaker/B-roll map: build the visual timeline4Text placement: assign template components from the 41-component library5Timeline assembly: zero frame gaps, every millisecond accounted for6Pacing validation: check visual density against target rates7User review and approval

Three visual asset sources only. The pipeline can only use assets from these three places:

  • video-extract: B-roll pulled from the recording itself (screen shares, tool demos)
  • motion-graphic: generated via Hera API or selected from the showcase template library
  • branded-template: permanent reusable components like UpworkProfile or AgencyBrand

MG trigger rules: What the speaker says determines which template type the pipeline assigns.

Speaker saysTemplate typeA specific numberCounter animation (NumberCountUp, ROICalculator)A tool name for the first timeLogo reveal (ToolLogoGrid, AIPulseIcon)References a UI or interfaceHera interface zoomWalks through a listSequential reveal (ChecklistReveal)

VAD-calibrated captions: Two-pass system. Pass 1: transcript lookup defines a search window around the phrase. Pass 2: VAD speech onset pins the exact frame start. If the transcript and VAD disagree by more than three frames, VAD wins.

Stages 7-8: B-Roll + Hera Motion Graphics (parallel)

These two stages run at the same time. Neither depends on the other.

B-Roll Extraction [BE]

FFmpeg with the -an flag. Always. No exceptions. B-roll is silent in Remotion — if audio leaks into a B-roll clip, you get an echo when Remotion composites it over the main audio track.

Full resolution from the raw source, not the proxy. B-roll is a visual element and it should look as good as the main recording.

Hera Motion Graphics [HM]

Hera takes a reference image and a detailed text prompt and returns a motion graphic clip.

Reference image waterfall: The pipeline tries four methods in order to get a reference image for each MG:

  1. Extract a frame from the source video where the tool UI is visible
  2. Take a web screenshot of the tool's live interface
  3. Pull from branded assets
  4. Composite a logo and a UI screenshot together

If all four fail, the MG gets demoted to a showcase template instead.

5-part prompt structure: Subject (what is in the frame), Motion (what moves and how), Style (visual treatment), Colour (exact hex codes from the tool's brand), Duration (5-8 seconds for interface zooms). The prompt must be at least 100 words. Short prompts produce generic results.

Caps: 3 Hera MGs maximum in the intro. 1 per body section. Everything else uses the template library. A simple number counter does not need a Hera API call. The template library handles that for free.

Stage 9: Remotion Edit [RE]

Eight steps from an approved storyboard to a rendered MP4.

Step 1: Preflight Validate the storyboard is marked APPROVED. Check every B-roll clip and MG clip exists on disk. If anything is missing, chain back to the relevant stage before starting.

Step 2: Scaffold Create the project structure: package.json, tsconfig, src/ with all component files. Copy the 41 showcase templates from the library into components/showcase/. Hardlink all video assets into public/.

Step 3: Theme generation Run ffprobe on the source video to get exact resolution and FPS. Never hardcode 1920x1080 or 30fps. The composition must match the source. Build the SEGMENTS array: every timeline row from the storyboard becomes a segment object with startFrame, durationInFrames, visual type, template name, and props.

Step 4: VideoRenderer dispatch One component reads the SEGMENTS array and dispatches to the correct template automatically. Claude Code does not need to write 40 individual segment components. One dispatcher handles the whole timeline.

Step 5: Composition Per-clip Audio elements. Never concatenated.

We learned this the hard way. Concatenating intro and body audio into one file caused a 510ms drift. The body portion of the concatenated file included dead air that the clipping stage had removed. The clipped video had no dead air. The audio did. The result was audio arriving 510ms late and getting worse with every second.

The fix: extract audio via stream copy from each clipped video file separately.

ffmpeg -i intro-clipped-raw.mp4 -vn -acodec copy public/intro-audio.m4a
ffmpeg -i body-clipped-raw.mp4 -vn -acodec copy public/body-audio.m4a

One Audio element per clip. audioOffsetMs exposed as a prop, tunable in Remotion Studio.

Step 6: QA An 18-point checklist. Key checks: OffthreadVideo has the muted prop, pauseWhenBuffering on all media, staticFile() wrapping every file path, zero frame gaps in the timeline, single Audio element per clip, no raw <img> tags. The checklist auto-fixes what it can and re-runs until clean. You do not render until every check passes.

Step 7: Preview Remotion Studio at localhost:3000. Tune audioOffsetMs. Usually one frame at 60fps is enough: 16ms accounts for the AAC encoder priming plus recording offset.

Step 8: Render

npx remotion render src/index.ts CompositionId out/video.mp4 \
  --codec h264 --crf 15 --gl=angle --hardware-acceleration if-possible

CRF 15. Near-lossless. Never go above 18. Never add --scale, --width, or --height: the composition already matches source resolution from the ffprobe step in Stage 3.

The Template Library

41 React components across 7 categories. Every one is parameterized from the storyboard by Claude Code: it reads the storyboard, picks the right template, fills in the props, and moves on.

CategoryComponentsExamplesNumber5NumberCountUp, ROICalculator, MetricCard, StatSplitCard, ProgressRingText3BoldStatement, CinematicReveal, TextRevealMaskList3ChecklistReveal, SequentialPillBuild, StackedPillsRevealConcept3FlowchartAnimation, TransformationArrow, SVGLineTimelineLogo3ToolLogoGrid, AIPulseIcon, GlowingIconPopComparison2ComparisonTable, BeforeAfterSplitProof2ProofQuote, SocialProofStack

This is why the output looks like production and not a demo. Claude Code does not write animation code per video. It reads the storyboard, assigns a component from this library, sets the props, and renders. The library is the reason the whole system works at scale.

What to do next

  • Watch the video again with the diagram open. Pause at each stage transition and find it on the diagram.
  • Identify which stages would need to change to match your editing style. Where do you make different decisions?
  • Those decisions are exactly what Module 2 covers.
← Previous
Next →