# What APIs exist for video indexing and search? > There are four kinds, and they are not substitutes for each other. Speech APIs turn audio into timed text. Canonical URL: https://vivu.ai/guide/what-apis-exist-for-video-indexing-and-search There are four kinds, and they are not substitutes for each other. Speech APIs turn audio into timed text. Frame-analysis APIs from the big cloud providers label objects, shots, and on-screen text. Multimodal video understanding APIs take a sentence in plain language and return ranked timecodes. And a smaller group of finished search products expose an API on top of an index they run for you. Before you compare endpoints, work out which of those four you are actually shopping for, because the wrong pick costs you a quarter of engineering time rather than a bad month. ## Speech and transcript APIs This is the largest and cheapest group. Whisper runs on your own hardware with open weights, and hosted transcription is available from Deepgram, AssemblyAI, AWS Transcribe, and Google's speech API. What you get back is word-level timestamps, usually speaker labels, and sometimes a rough topical segmentation. What you do not get is a search index. A transcript is a document, and you still have to store it somewhere queryable, join it back to the source file, and decide what a hit means when the phrase is said four times in one interview. For footage where the meaning lives in the audio, panels, interviews, lectures, earnings calls, this covers most of the retrieval need, and the mechanics of turning those transcripts into usable selects are worked through in [going from transcript to timecode](https://vivu.ai/guide/how-do-i-use-a-transcript-to-find-good-video). ## Frame analysis from cloud providers Google Cloud Video Intelligence, Azure AI Video Indexer, and AWS Rekognition all read the picture rather than the audio. They return shot boundaries, generic object and scene labels, text detected on screen, and face detection if you set up a face collection. The constraint is the vocabulary. These services classify against a fixed label set, so "person", "car", and "office" come back reliably, while "the moment the CEO steps away from the whiteboard" is not a label and never will be. They are good ingest-time enrichers and poor answer engines. ## Multimodal video understanding APIs This is the group people usually mean when they say video search API. Twelve Labs is the best-known vendor, and the open-weight route is to embed frames with a CLIP-family model, embed the query with the same model, and rank by similarity in a vector database. Either way you get natural-language retrieval over segments instead of keyword matching over labels. You are buying the retrieval, though, and not the system. Ingest orchestration, storage, permissions, dedupe, a review interface, and reindexing when the model version changes are all still yours. Budget for that or you will ship a demo and stall. ## Products that happen to expose an API The last group inverts the deal. The index is run for you, the unit you buy is a searchable library, and the API is an integration surface rather than the product. A team that mainly wants to ask its library questions can skip building anything. In [Vivu](https://vivu.ai/mcp), footage goes into a project by upload and is indexed in the cloud, and a producer who has added Vivu in Claude under Settings → Connectors asks for footage in plain language and gets back a list of time ranges, each with a preview. If a team wants search results rather than a build project, that difference usually settles whether "what API should we use" was the right question in the first place. ## What the pricing pages leave out Nearly every API in the first three groups charges per minute of video processed at ingest. The line item nobody plans for is the second pass. Model versions move, your label taxonomy changes, someone decides speaker diarization matters after all, and you pay to process the archive again. Ask each vendor what reindexing costs and whether it is charged at the same rate, because that number, not the first-pass rate, is what the API costs over three years. The other omission is who keeps new footage indexed. An API processes what you send it. Something has to send it, forever, including the files a producer drops into a folder on a Friday. Job listings for content operations roles keep describing exactly this work, a person hired to keep a media library and a set of stakeholders in sync, which is a fair signal that the pipeline maintenance is real work rather than a footnote. ## When you do not need an API at all If the archive is a few hundred files, one team owns naming, and the request rate is a clip or two a month, folder discipline beats any of this. The same is true if all your footage is one show in one format with an existing episode index. APIs earn their keep when the volume is past what one person can hold in memory and the requests come from people who were not in the room when the footage was shot. Decide by asking what you are building. If search is a feature inside a product you ship, you want a model API and you should plan for the surrounding system. If search is something your team needs on Tuesday, you want a product, and the API question was a detour. The middle case, where you want both, is the expensive one, and it is worth naming out loud before anyone writes a proof of concept. If you are still working out how any of this finds anything, [what happens between the query and the timecode](https://vivu.ai/guide/how-does-ai-search-inside-video-footage) is the shorter version, and the label-generation side is covered in [the tools that tag files automatically](https://vivu.ai/guide/what-software-can-automatically-tag-video-files). ## FAQ ### Can I use a speech-to-text API for video search? Yes, for footage where what matters was said out loud. Run transcription with word-level timestamps, store the results in a search index, and keep a stable mapping from each transcript segment back to a file and an offset. That gets you jump-to-the-moment retrieval for interviews, webinars, and panels. It fails on anything visual and silent. B-roll, product shots, gameplay, and sports action produce no useful transcript, so a speech-only index will simply return nothing for those queries rather than returning something wrong. Check what fraction of your archive is speech-driven before committing. ### How is a video indexing API different from a video search API? Indexing is the write path and search is the read path. An indexing API takes a file and produces derived data: transcripts, embeddings, labels, shot boundaries. A search API takes a query and returns ranked results with timecodes. Some vendors sell both behind one endpoint, which is why the terms get used interchangeably. The distinction matters at purchase time because they are priced differently. Indexing is usually billed per minute of source video, once, at ingest. Search is usually billed per query or bundled into a seat. A large archive with light query traffic and a small archive with heavy query traffic end up with very different bills from the same vendor. ### Do I need a vector database to build video search? Only if you are doing embedding-based retrieval. If your index is transcripts and labels, a normal full-text search engine handles it, and adding a vector store buys you complexity you do not need. If you want natural-language queries over what appears on screen, then yes, you need somewhere to store and search embeddings. Postgres with pgvector is usually enough at small scale, and dedicated stores like Qdrant or Milvus make sense once the segment count is large or you need filtered search with heavy metadata conditions. ### How long does it take to build video search on top of an API? Longer than the API integration, which is often a day or two. The work that consumes the schedule is everything around it: an ingest queue that survives failures, a permissions model so people only see footage they are allowed to see, a result interface that lets someone scrub and confirm a hit, and a plan for reprocessing when the model changes. Teams that ship this fastest tend to be ones already running an asset pipeline, so the API slots into existing plumbing. Teams starting from a shared drive usually underestimate by a factor most of a quarter, because the drive is where all the exceptions live.