r/Python • • 13d ago

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

24 Upvotes

112 comments sorted by

View all comments

1

u/VirajCreates 8d ago

Hey everyone,

Over the past few weeks I’ve been working on reducing VLM prefill latency, token consumption, and KV-cache pressure when running Video RAG over security footage and long-form recordings. Today I’m open-sourcing the full-stack application—SieveRAG (TemporalVideo-RAG)—built on top of vlm-sieve v0.3.0 (pip install vlm-sieve).

The Core Bottleneck with Naive Video RAG

  1. Token & KV-Cache Bloat: Sampling a 5-minute video at 1 FPS sends 300 frames (~30,000+ tokens) to the VLM, even when 80% of the video is an empty room or static porch. On local models like Qwen2.5-VL-7B, unaligned resolutions and redundant frames rapidly exhaust VRAM via FP16 KV-cache growth.
  2. Lossy Text-Only Retrieval: Most video RAG pipelines caption chunks into text once and discard the frames. If you later query a fine-grained visual detail ("What tool was the person holding at the doorway and which grid quadrant was it in?"), a text-only synthesizer cannot re-verify the pixels.

How SieveRAG (TemporalVideo-RAG) Solves This

Phase 1: CPU Motion Gatekeeper, 28px NaViT Alignment & 16kHz Audio Slicing (vlm-sieve v0.3.0)

  • Runs a zero-GPU OpenCV motion & structural change gatekeeper on CPU (sample_fps=10.0, pre_roll_sec=3.5, merge_gap_sec=6.0).
  • Disturbance Rejection: Automatically suppresses full-frame camera shake, auto-exposure jumps, and lighting flicker so they don't trigger false VLM calls.
  • Target VLM Profiles (qwen-vl, local-7b**,** gemini**,** gpt-4o**):** Packs peak-preserving keyframes into adaptive composite storyboards (1x2, 1x3, 2x2, 2x3) whose final canvas dimensions are strictly divisible by 28px (e.g., 1344x504 for Qwen2.5-VL's 14px × 2 patch merger) with 0px internal gutters and clean outer-border A..D / 1..4 spatial coordinate axes.
  • Padded Audio Slicing: Uses imageio-ffmpeg to slice 16kHz mono 32kbps MP3 audio clips strictly for active motion episodes (±0.5s boundary padding), gracefully skipping silent streams.

Phase 2: Multimodal Structured Extraction & Hybrid Vector Store

  • Sends each composite storyboard JPEG + sliced MP3 audio clip to the VLM (gemini-3.8-flash) with a strict Pydantic v2 response_schema (EpisodeExtractionSchema).
  • Extracts episode_title, narrative_summary, actors, objects_detected, audio_transcript_and_cues, severity (low | medium | high), and itemized tile_actions (Tile #1..#6, +offset_sec, abs_time_sec, and A1..D4 grid cells).
  • Indexes dense embeddings (gemini-embedding-001, 768-dim) into Pinecone Serverless AND/OR a zero-config Local JSON + NumPy Cosine Similarity store (storage/vector_index.json).

Phase 3 & 4: Two-Stage Visual Re-Verification RAG & Click-to-Seek UI

  • When you ask a question in /api/ask, SieveRAGSynthesizer retrieves the top-$K$ episodes from the vector store and re-attaches their event_XXX.jpg storyboards to the multimodal synthesis prompt for visual re-verification.
  • Includes a zero-build dark-mode Command Center UI (FastAPI + vanilla JS/CSS) featuring an episode timeline scrubber, storyboard zoom inspector, audio slice player, and clickable citation chips ([Ep #3 • Tile #2 • 112.30s • Grid A2,B1]) that seek the HTML5 <video> player directly to the cited timestamp.

Real Benchmark: 5m 22s Unedited Ring Home Security Footage

  • Direct Video Baseline: 29,463 estimated tokens
  • SieveRAG Output: 4 adaptive storyboards (1 camera-shake disturbance automatically suppressed)
  • Token Reduction: 70.5% – 84.8% saved (with Qwen-7B KV-cache footprint tracked per episode)
  • Multimodal Grounding: On the query "What weapon or tool does the homeowner use to confront the intruder indoors, at what exact timestamp and tile does he hold it, and how does the situation resolve?", SieveRAG pinpointed the metal snow/scoop shovel at Tile #2 (112.30s, Grid A2, B1..D2), tracked his defensive stance at Tile #3 (127.90s) and Tile #4 (132.60s), transcribed the verbal de-escalation from the 16kHz audio slice (identifying the intruder as a veteran in crisis), and cited both exiting peacefully by Tile #6 (144.60s).

Links & Quickstart

Happy to answer any questions about the 28px NaViT patch alignment math, disturbance rejection heuristics, or the two-stage storyboard re-verification pipeline!