- What: Google released Gemini Embedding 2, enabling native video embedding for sub-second semantic search.
- Cost: Approximately $2.84 per hour of footage indexed at default settings ($0.00079 per frame).
- Technical Edge: Raw video pixels are projected directly into a 768-dimensional vector space, bypassing the need for transcription or frame captioning.
- Implementation: SentrySearch, a new open-source CLI tool, uses this technology to index and search dashcam footage via ChromaDB.
Google has launched Gemini Embedding 2, a multimodal model that allows developers to search video files using natural language without the need for manual transcription or frame captioning. This advancement has been showcased through a new open-source project called SentrySearch, which enables sub-second semantic search over hours of dashcam footage. By projecting raw video pixels directly into the same vector space as text queries, the system can identify specific events—such as a "red truck running a stop sign"—in a matter of milliseconds.
The End of the "Text Middleman"
The core innovation behind Gemini Embedding 2 is its ability to treat video as a first-class citizen in the embedding space. Traditionally, searching video required a "text middleman": a separate AI model would have to transcribe audio or generate captions for individual frames, which were then indexed for text search. According to the SentrySearch documentation and Google's release data, Gemini Embedding 2 eliminates this step entirely.
Raw video pixels are projected into a 768-dimensional vector space that is shared with text queries. This means a text string like "a white car turning left" is directly comparable to a 30-second video clip at the vector level. This native multimodality allows for a level of speed and accuracy previously unattainable for local or small-scale developer projects.
How SentrySearch Operationalizes Video Search
SentrySearch, developed by a member of the developer community and shared via Hacker News and GitHub, provides a command-line interface (CLI) that bridges the gap between Google’s API and a functional user experience. The tool is designed specifically for dashcam footage, though its architecture is applicable to broader security and monitoring use cases.
The workflow involves four distinct stages:
- Chunking: The tool uses
ffmpegto split long video files into overlapping segments (defaulting to 30-second chunks with a 5-second overlap). - Embedding: Each chunk is sent to the Gemini Embedding API. The API natively extracts and tokenizes exactly one frame per second from the uploaded video.
- Storage: The resulting 768-dimensional vectors are stored in a local ChromaDB database.
- Retrieval: When a user types a query, the text is embedded into the same vector space. The tool calculates similarity scores, identifies the top match, and automatically trims the original video file into a shareable clip.
Pricing and Technical Optimizations
Indexing high-resolution video can be computationally expensive, but the SentrySearch implementation highlights several optimizations to keep costs manageable. At default settings, indexing one hour of footage costs approximately $2.84. This is calculated based on 3,600 frames (one per second) at a rate of $0.00079 per frame processed by the Gemini API.
To improve performance and reduce overhead, the tool includes two primary optimizations:
- Local Preprocessing: Chunks are downscaled to 480p at 5fps before being uploaded. While the Gemini API still bills based on its internal 1fps extraction rate, this local downscaling significantly reduces payload size and prevents API request timeouts.
- Still-Frame Skipping: This heuristic-based feature compares JPEG file sizes across sampled frames to detect "idle" footage, such as a parked car or an empty driveway. If no meaningful visual change is detected, the chunk is skipped entirely, saving the cost of the API call.
Impact on Developers and Industry
This release marks a significant shift in how developers will build video-reliant applications. The ability to query video as easily as one queries a text document opens the door for advanced home security systems, automated sports highlight generation, and more efficient digital asset management.
"There's no transcription, no frame captioning, no text middleman," the SentrySearch documentation notes. This direct vector comparison changes the fundamental architecture of search engines. For the first time ever, sub-second semantic search over massive local video archives is practical for independent developers and small businesses without requiring a massive GPU cluster.
For the security industry, this means the end of scrubbing through hours of "Sentry Mode" or dashcam recordings. A user can simply ask the system to "find the person wearing a red hat" and receive a trimmed clip instantly.
What’s Next for Multimodal Search
While Gemini Embedding 2 is a major leap forward, there are still limitations to navigate. The model currently supports audio track extraction from video natively, but the maximum video length for such tasks drops from 128 seconds to 80 seconds when audio is processed. Furthermore, the still-frame detection in current implementations like SentrySearch is heuristic-based; future updates may integrate more sophisticated motion detection to ensure subtle but important movements aren't skipped during the indexing phase.
As Google continues to refine the Gemini architecture, the integration of text, images, video, audio, and documents into a single embedding space suggests a future where all data types are searchable through a single, unified interface.