DEV Community

Cover image for Brand Tagging with VLMs
Ville Vesilehto
Ville Vesilehto

Posted on • Originally published at ville.dev

Brand Tagging with VLMs

#vlm #pytorch #ai #python

TL;DR

Build a two-stage logo pipeline:

  1. Retrieval - generate image embeddings for small crops and match against a logo dictionary with FAISS cosine search. Use SigLIP-2 (NaFlex) so logos are not distorted and small marks still pop.

  2. Verification - for top matches, ask LLaVA-OneVision-1.5 a strict JSON question ("Is this the X logo?") and accept only high-confidence "yes". It's a good model sir.

A bit longer post this time around.

Intro

Brand tagging in real-world video is hard: logos are tiny, partly occluded, moving, and often appear on textured backgrounds. A practical approach is a two-stage pipeline:

  • First retrieve likely logo crops with a fast contrastive image encoder.
  • Then verify each candidate with a vision-language model (VLM) that can read text and reason about shapes and context. Each candidate is a (frame, bbox, brand, retrieval_score) record that we pass to the VLM.

This tutorial combines a modern image-text encoder (SigLIP-2, NaFlex variant) for high-recall retrieval with a VLM from the LLaVA family for precise, structured yes/no verification.

You could train a YOLO-style detector for specific brands. This post, however, focuses on a more flexible "embedding + VLM" approach that adapts quickly to new logos without retraining. Some might find this fun.

Research

Contrastive encoders such as CLIP/SigLIP produce embeddings where similar visuals are close. FAISS makes nearest-neighbor search over many references instantaneous. You can treat logo search as nearest-neighbor lookup instead of training a custom detector.

Verification reduces false positives. A VLM can explicitly answer "Is this the Red Bull logo?" and justify the decision, improving precision on lookalikes, partial views, and blur.

Pointers to the underlying research for the initiated:

Setup

You’ll need to be comfortable with basic Python and PyTorch. No prior experience with FAISS or VLMs required.

Hardware/software requirements:

  • At least one NVIDIA GPU (at least 32 GB VRAM recommended)
  • CUDA 12.x, Python 3.10–3.12
  • Disk: 5–10 GB (models + caches)
  • Tools: ffmpeg for frames

This tutorial was built and tested on a single NVIDIA H200.

Prepare the Python environment as follows:

python3 -m venv .venv && source .venv/bin/activate pip install -U torch torchvision torchaudio \ "transformers>=4.45" accelerate pillow opencv-python faiss-cpu \ numpy pydantic polars
Enter fullscreen mode Exit fullscreen mode

For simplicity we'll use ƒaiss-cpu as it's fine at this scale. It's a single logo after all. GPU for the models.

Example video (Creative Commons):

  • Title: Red Bull Racing Pit Stop Practice (2015), 44 s, 1920x1080.
  • License: CC BY-SA 4.0 - attribute ProtoplasmaKid / Wikimedia Commons.
  • It has tiny, moving logos, occlusion, uniforms, car bodywork, pit rig and acts as a great stress-test.

Architecture 

frames (2–4 FPS) ──▶ crops (multi-scale grid) │ ▼ [Stage 1] Retrieval (SigLIP-2 image features + FAISS cosine) │ └─ top-K per frame/brand with heuristics ▼ [Stage 2] VLM verification (LLaVA-OneVision-1.5 JSON verdict) │ ▼ JSONL evidence
Enter fullscreen mode Exit fullscreen mode

Models used in this post:

SigLIP-2 reports better zero-shot and retrieval performance than prior SigLIP/CLIP models on public benchmarks. NaFlex means the encoder resizes each crop to a grid of flexible patches instead of forcing a fixed square, so long thin logos don't get squashed.

OneVision-1.5 is an open VLM family. Its card reports strong benchmark leads vs other open models.

Both are Apache-2.0 licensed.

So let's do it!

Quickstart

1) Grab the video:

wget https://upload.wikimedia.org/wikipedia/commons/b/ba/Red_Bull_Racing_Pit_Stop_Practice.webm
Enter fullscreen mode Exit fullscreen mode

Then extract frames. Two FPS, scaled down to 1280x720:

mkdir -p frames ffmpeg -i "Red_Bull_Racing_Pit_Stop_Practice.webm" -vf "fps=2,scale=1280:-1:flags=lanczos" -q:v 3 frames/f_%06d.jpg
Enter fullscreen mode Exit fullscreen mode

This yields 88 JPG files.

2) Prepare a logo dictionary

In this example we're only interested in the Red Bull logo. Create a logos directory you want to use. Grab the logo:

mkdir -p logos wget -P logos https://upload.wikimedia.org/wikipedia/fi/a/a5/Red_Bull_logo.png
Enter fullscreen mode Exit fullscreen mode

Disclaimer: This tutorial is for educational purposes only and is not affiliated with or endorsed by Red Bull. Red Bull is a registered trademark of Red Bull GmbH. It's a decent energy drink though.

3) Build the logo index (FAISS, SigLIP-2)

Save to build_logo_index.py:

import json, faiss, torch from pathlib import Path from PIL import Image import numpy as np from transformers import AutoModel, AutoProcessor MODEL_ID = "google/siglip2-base-patch16-naflex" # NaFlex = native aspect ratio OUT_DIR = Path("artifacts"); OUT_DIR.mkdir(exist_ok=True, parents=True) def embed_images(paths, model, proc, batch=16): imgs = [Image.open(p).convert("RGB") for p in paths] feats = [] for i in range(0, len(imgs), batch): chunk = imgs[i:i+batch] inputs = proc(images=chunk, return_tensors="pt") inputs = {k: v.to(model.device) for k, v in inputs.items()} with torch.no_grad(): f = model.get_image_features(**inputs) # (B, d)  f = torch.nn.functional.normalize(f, dim=-1) feats.append(f.cpu()) return torch.cat(feats, dim=0).numpy() def main(): model = AutoModel.from_pretrained(MODEL_ID, dtype=torch.float32, device_map="auto") proc = AutoProcessor.from_pretrained(MODEL_ID) logo_paths = sorted(list(Path("logos").glob("*.*"))) if not logo_paths: raise SystemExit("No logo files found in 'logos' directory.") brands = [p.stem for p in logo_paths] vecs = embed_images(logo_paths, model, proc) d = vecs.shape[1] index = faiss.IndexFlatIP(d); index.add(vecs) faiss.write_index(index, str(OUT_DIR / "logos.faiss")) (OUT_DIR / "logos_meta.json").write_text( json.dumps({"brands": brands, "files": [str(p) for p in logo_paths]}, indent=2) ) print(f"Indexed {len(brands)} brands into {OUT_DIR/'logos.faiss'}") if __name__ == "__main__": main()
Enter fullscreen mode Exit fullscreen mode

SigLIP-2 provides get_image_features and NaFlex dynamic resizing to minimize distortion on non-square inputs — useful for narrow/wide logos.

Run the tool:

$ python build_logo_index.py ... Indexed 1 brands into artifacts/logos.faiss
Enter fullscreen mode Exit fullscreen mode


4) Generate crops + Stage-1 retrieval

Save the following to retrieve.py:

import json, math, os, faiss, torch from pathlib import Path from PIL import Image, ImageDraw import numpy as np from transformers import AutoModel, AutoProcessor EMB_ID = "google/siglip2-base-patch16-naflex" ART = Path("artifacts"); ART.mkdir(exist_ok=True) SIZES = [192, 256, 320] # square windows STRIDE = 0.5 # 50% overlap TOPK = int(os.environ.get("TOPK", "3")) COSINE_TH = float(os.environ.get("COSINE_TH", "0.7")) # keep candidates above this  def _metric_type(index): try: return index.metric_type except Exception: return None def _l2_to_cosine(d: np.ndarray) -> np.ndarray: # If vectors are L2-normalized, cos = 1 - 0.5 * ||a - b||^2  return 1.0 - 0.5 * d def _as_float32(x: np.ndarray) -> np.ndarray: if x.dtype != np.float32: x = x.astype(np.float32, copy=False) return np.ascontiguousarray(x) def grid_crops(im: Image.Image): W, H = im.size for s in SIZES: step = max(1, int(s * STRIDE)) for y in range(0, max(1, H - s + 1), step): for x in range(0, max(1, W - s + 1), step): yield (x, y, s, s) def embed(model, proc, pil_list, bs=24): out = [] for i in range(0, len(pil_list), bs): chunk = pil_list[i:i+bs] inp = proc(images=chunk, return_tensors="pt") inp = {k: v.to(model.device) for k, v in inp.items()} with torch.no_grad(): f = model.get_image_features(**inp) f = torch.nn.functional.normalize(f, dim=-1) out.append(f.detach().to(torch.float32).cpu()) vecs = torch.cat(out, dim=0).numpy() return _as_float32(vecs) def main(): # load logo index  index = faiss.read_index(str(ART / "logos.faiss")) meta = json.loads((ART / "logos_meta.json").read_text()) brands = meta["brands"] try: print(f"[retrieve] index ntotal={index.ntotal}, brands={len(brands)}") except Exception: pass # load embedder  model = AutoModel.from_pretrained(EMB_ID, dtype=torch.float32, device_map="auto") proc = AutoProcessor.from_pretrained(EMB_ID) frames = sorted(Path("frames").glob("f_*.jpg")) print(f"[retrieve] frames found={len(frames)}") out = [] dump_all = bool(int(os.environ.get("DEBUG_DUMP_ALL", "0"))) dbg_all = [] if dump_all else None debug_draw = dump_all and bool(int(os.environ.get("DEBUG_DRAW", "0"))) debug_draw_dir = ART / "debug_vis" if debug_draw: debug_draw_dir.mkdir(parents=True, exist_ok=True) debug_draw_th = float(os.environ.get("DEBUG_DRAW_TH", "-1.0")) debug_draw_max = int(os.environ.get("DEBUG_DRAW_MAX", "0")) # 0 = unlimited  best = {"score": -1.0, "frame": None, "brand": None, "bbox": None} for fpath in frames: im = Image.open(fpath).convert("RGB") boxes, crops = [], [] for (x, y, w, h) in grid_crops(im): boxes.append((x, y, w, h)) crops.append(im.crop((x, y, x+w, y+h))) if not crops: continue vecs = embed(model, proc, crops) D, I = index.search(_as_float32(vecs), TOPK) mt = _metric_type(index) if mt == getattr(faiss, "METRIC_L2", 1): scores_mat = _l2_to_cosine(D) else: scores_mat = D # collect frame-local debug matches as well  frame_dbg = [] if dump_all else None for i, (scores, ids) in enumerate(zip(scores_mat, I)): if dump_all: for r, (score, idx) in enumerate(zip(scores.tolist(), ids.tolist())): rec = { "frame": fpath.name, "bbox": boxes[i], "rank": int(r), "score": float(score), "brand": brands[idx] } dbg_all.append(rec) if frame_dbg is not None: frame_dbg.append(rec) for score, idx in zip(scores, ids): if score > best["score"]: best.update({ "score": float(score), "frame": fpath.name, "brand": brands[idx], "bbox": boxes[i] }) if score < COSINE_TH: continue bx = boxes[i] out.append({ "frame": fpath.name, "bbox": bx, "score_retr": float(score), "brand": brands[idx] }) # draw annotations for this frame if requested  if debug_draw and frame_dbg: # sort by score desc  frame_dbg.sort(key=lambda r: r["score"], reverse=True) if debug_draw_max > 0: frame_dbg = frame_dbg[:debug_draw_max] canvas = im.copy() draw = ImageDraw.Draw(canvas) for rec in frame_dbg: if rec["score"] < debug_draw_th: continue x, y, w, h = rec["bbox"] x2, y2 = x + w, y + h color = (255, 0, 0) draw.rectangle([x, y, x2, y2], outline=color, width=2) label = f"{rec['brand']} {rec['score']:.3f}#{rec['rank']}" # simple text; if background needed, draw a small filled box then text  draw.text((x + 3, y + 3), label, fill=color) out_path = debug_draw_dir / f"{fpath.stem}_debug.jpg" canvas.save(out_path, quality=90) print(f"[retrieve] wrote debug visualization → {out_path}") Path("candidates.jsonl").write_text("\n".join(json.dumps(x) for x in out)) print(f"wrote {len(out)} retrieval candidates → candidates.jsonl") if not out and best["frame"] is not None: print(f"[retrieve] no candidates above threshold {COSINE_TH}. " f"Best observed: score={best['score']:.3f}, frame={best['frame']}, " f"brand={best['brand']}, bbox={best['bbox']} — consider lowering COSINE_TH.") if dump_all and dbg_all is not None: Path("debug_matches.jsonl").write_text("\n".join(json.dumps(x) for x in dbg_all)) print(f"[retrieve] wrote {len(dbg_all)} raw matches → debug_matches.jsonl (DEBUG_DUMP_ALL=1)") if __name__ == "__main__": main()
Enter fullscreen mode Exit fullscreen mode

This is quite a few things so breaking it down:

  • Crops: multi-scale grid over each frame using SIZES and STRIDE, yielding square patches and their bbox tuples.
  • Embeddings: SigLIP‑2 get_image_features on each crop batch, then L2‑normalize features.
  • Search: FAISS over the logo index; uses inner‑product on normalized vectors (cosine). If the index is L2, we convert to cosine (1 − 0.5·L2²).
  • Types: FAISS expects contiguous float32; embeddings are cast/contiguous before index.search.
  • Thresholds: keep top‑TOPK per crop, then filter by COSINE_TH. You can override at runtime:
    • TOPK=10 COSINE_TH=0.65 python retrieve.py
  • Debugging:
    • DEBUG_DUMP_ALL=1 writes every raw match to debug_matches.jsonl (ranked with scores).
    • DEBUG_DRAW=1 also saves artifacts/debug_vis/*_debug.jpg with [brand score#rank] boxes.
    • Optional: DEBUG_DRAW_TH=0.3 (only draw ≥ threshold), DEBUG_DRAW_MAX=200 (cap boxes).
  • Outputs: filtered candidates land in candidates.jsonl and are fed to the verifier stage.
  • Knobs to tune recall/precision: crop sizes, stride, TOPK, COSINE_TH. For higher recall, increase sizes or TOPK; for precision, raise the threshold and later add a margin or temporal smoothing.

Output:

$ python retrieve.py [retrieve] index ntotal=1, brands=1 [retrieve] frames found=88 wrote 25 retrieval candidates → candidates.jsonl
Enter fullscreen mode Exit fullscreen mode

Depending on the source material this probably needs tuning for recall/precision. The debugging knobs are quite nice. Here's an example frame from the video showing how it looks like on frame #7:

Example frame with SigLIP-2 retrieval overlay: Red Bull logo detected at cosine 0.70 (rank #0) on the car sidepod

Frame from Red Bull Racing Pit Stop Practice (2015), ProtoplasmaKid / Wikimedia Commons / CC BY-SA 4.0.

5) VLM verification

Save the following to verify.py:

import json, torch, os from pathlib import Path from PIL import Image from transformers import AutoProcessor, AutoModelForCausalLM VLM_ID = "lmms-lab/LLaVA-OneVision-1.5-8B-Instruct" SYSTEM = ( "You are a logo verification API. " "Given an image crop and a target brand, answer in strict JSON with no extra text." ) def build_user_prompt(brand: str): return ( "Task: Verify whether the crop contains the specified brand's logo.\n" f"Brand: {brand}\n\n" "Return JSON only:\n" "{\n" ' "verdict": "yes" | "no",\n' ' "confidence": 0.0-1.0,\n' ' "visual_cues": "short, literal cues proving the verdict (colors/shapes/text)"\n' "}\n" "Rules: output only the JSON object; no prose before or after. " "Be literal; do not speculate; base confidence on how clearly the logo is visible." ) def _normalize_quotes(s: str) -> str: # Replace smart quotes with ASCII equivalents  return ( s.replace("\u201c", '"').replace("\u201d", '"') .replace("\u2018", "'").replace("\u2019", "'") ) def _extract_json_object(text: str): text = _normalize_quotes(text) # Find first balanced {...} block; handle braces within strings  in_str = False escape = False depth = 0 start = None for i, ch in enumerate(text): if ch == "\\" and not escape: escape = True continue if ch == '"' and not escape: in_str = not in_str escape = False if in_str: continue if ch == "{": if depth == 0: start = i depth += 1 elif ch == "}": if depth > 0: depth -= 1 if depth == 0 and start is not None: candidate = text[start:i+1] try: j = json.loads(candidate) if isinstance(j, dict) and "verdict" in j and "confidence" in j: return j except Exception: pass return None def run_once(proc, model, crop: Image.Image, brand: str, max_new=128): msgs = [ {"role":"system", "content": SYSTEM}, {"role":"user", "content":[{"type":"image"}, {"type":"text", "text": build_user_prompt(brand)}]} ] text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True) inputs = proc(text=[text], images=[crop], padding=True, return_tensors="pt") inputs = {k: (v.to(model.device) if hasattr(v,"to") else v) for k,v in inputs.items()} with torch.inference_mode(): ids = model.generate(**inputs, max_new_tokens=max_new, temperature=0.0, do_sample=False) out = proc.batch_decode(ids, skip_special_tokens=True)[0] j = _extract_json_object(out) if j is None: if os.environ.get("DEBUG_VLM", "0") == "1": Path("vlm_raw.txt").write_text(out) return {"verdict":"no", "confidence":0.0, "visual_cues":"parse_error"} # Coerce fields  verdict = str(j.get("verdict", "no")).strip().lower() if verdict not in ("yes", "no"): verdict = "no" try: conf = float(j.get("confidence", 0.0)) except Exception: conf = 0.0 cues = str(j.get("visual_cues", "")) return {"verdict": verdict, "confidence": conf, "visual_cues": cues} def main(): proc = AutoProcessor.from_pretrained(VLM_ID, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( VLM_ID, dtype=torch.float32, device_map="auto", trust_remote_code=True ) results = [] for line in Path("candidates.jsonl").read_text().splitlines(): c = json.loads(line) im = Image.open(Path("frames")/c["frame"]).convert("RGB") x,y,w,h = c["bbox"] crop = im.crop((x,y,x+w,y+h)) j = run_once(proc, model, crop, c["brand"]) results.append({ **c, "verdict": j.get("verdict","no"), "confidence_vlm": float(j.get("confidence",0.0)), "rationale": j.get("visual_cues","") }) Path("detections.jsonl").write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in results)) print("wrote detections.jsonl") if __name__ == "__main__": main()
Enter fullscreen mode Exit fullscreen mode

Again, breaking it down:

  • Inputs: candidates.jsonl entries with frame, bbox, score_retr, brand.
  • Crop: open the frame and im.crop(bbox) per candidate.
  • Prompting: one system + one user message; user contains the target brand and an explicit "JSON only" schema.
  • Generation: temperature=0.0, do_sample=False, max_new_tokens=128 for deterministic outputs.
  • Parsing: balanced-brace JSON extraction with smart-quote normalization; on failure returns parse_error. Set DEBUG_VLM=1 to dump raw text to vlm_raw.txt.
  • Output: writes detections.jsonl with verdict, confidence_vlm, rationale merged onto each candidate.
  • Cost control: keep retrieval strict (lower TOPK, higher COSINE_TH) to limit VLM calls as that's the main runtime driver.

Output:

$ python verify.py wrote detections.jsonl
Enter fullscreen mode Exit fullscreen mode

And what does detections.jsonl look like? It has items like this (one per line):

{ "frame": "f_000007.jpg", "bbox": [ 384, 288, 192, 192 ], "score_retr": 0.7020304203033447, "brand": "Red_Bull_logo", "verdict": "yes", "confidence_vlm": 0.95, "rationale": "red bull charging bull silhouette, red and yellow colors" }
Enter fullscreen mode Exit fullscreen mode

Here 0.702 is the cosine similarity between the crop and the logo embedding.

Conclusion

I hope this showcases how a simple two-stage recipe - SigLIP-2 retrieval + VLM verification - can turn a semi-noisy video into reviewable brand evidence. I do want to mention that a pipeline like this is meant as a powerful filter, not an oracle. Human in the loop needed.

On this specific 44-second clip, with the thresholds above, I get N true positives, M false positives, and 0 missed clear logos (subjective visual check).

That being said, this is probably not a production-ready detector. We trade some accuracy and runtime for simplicity and transparency. Some improvement ideas I had in mind:

  • float32 dtype could be replaced with lower precision, like bfloat16.
  • Add a margin filter (top1 − top2 ≥ 0.15) and simple temporal smoothing. Confirm across 2–3 adjacent frames.
  • Add a lightweight OCR gate for texty marks to backstop retrieval.
  • Calibrate thresholds per brand, as some logos need higher COSINE_TH or TH_VLM.
  • Expand the logo dictionary and try multi-resolution templates (flat vs curved surfaces).
  • Speed/scale: use FAISS IVF/PQ for larger dictionaries.
  • Quantize the VLM or batch crops.
  • Maybe consider a second encoder for consensus (SigLIP‑2 + CLIP) to reduce lookalikes.
  • For 4K frames, consider either down-scaling first or increasing STRIDE (e.g. 0.75) to avoid generating tens of thousands of crops per frame.

As always, credits: thanks to the SigLIP-2 authors & maintainers and the LLaVA-OneVision team for open releases. And attribution for the example video: ProtoplasmaKid / Wikimedia Commons / CC BY-SA 4.0. 

Top comments (0)