Parallax

Parallax/Work/Case study

Multimodal content search: an archive, searchable by meaning

A digital publisher had years of articles, images and video that its own teams struggled to find. We prototyped a pipeline that crawls the archive, has a multimodal model describe and tag every piece, and serves search by meaning — then designed the architecture to take it to production scale.

Client
Digital publisher (anonymized)
Type
AI search · internal tooling
What we did
Prototype, platform evaluation, production architecture
Stack
Gemini · Cloud Run · Firestore · sitemap crawling

Case study

The problem

Publishers sit on enormous archives, and most of the value is locked away because search only sees the words someone typed into a CMS field. Nobody tagged the photo of the post-game handshake, or the video segment where the coach explains the new strategy. Editors re-create content that already exists; social and SEO teams can’t find the evergreen piece that would perform today.

The prototype

  1. Crawl from the sitemap. The public sitemap gave a complete, low-friction inventory of the archive without waiting on CMS access.
  2. Describe everything with a multimodal model. Gemini read each article together with its images and video and returned structured tags: topics, people, organizations, events, visual descriptions and a short summary.
  3. Store documents, not just vectors. Firestore held each item with its tags and metadata, so results could be filtered as well as ranked.
  4. Serve search on Cloud Run. A small, stateless search API and a simple UI let editors try natural-language queries against real content within days.
Pipeline: sitemap crawler to fetcher to Gemini multimodal tagging to Firestore to search API on Cloud Run Sitemapinventory Fetchertext + media Geminimultimodal tags Firestoredocs + tags Search APICloud Runbatch backfill today · real-time ingestion in production
Fig. 2 — The prototype pipeline. Each stage is independently retryable, which matters when you’re processing years of content.

Build or buy: evaluating managed search

We also evaluated a managed option, Vertex AI Search, against the custom pipeline. Managed search is fast to stand up; the custom path gave more control over what gets extracted from images and video, how results are filtered, and what it costs at archive scale. Doing both in the prototype meant the decision was made with evidence rather than vendor slides.

The production architecture we recommended

  • Real-time ingestion with an event stream (Kafka, with Flink for processing) so new content is tagged and searchable within moments of publishing, not on the next batch run.
  • Elasticsearch as the search engine for hybrid keyword-plus-semantic ranking, filtering and faceting at scale.
  • A federated GraphQL API so the same search capability can serve editorial tools, the website and partner products without separate integrations.
Tag once, with a model that can see. Every downstream team — editorial, social, SEO — gets a better archive for free.

Updated 2026-09-25

More work

Have a workflow like this? Let’s look at it together.

Start a project →