Flywheel of Media Models & Product Velocity#

Rushaan Mahajan (Meta)

• Why: Once AI media models cross the photorealism threshold, evaluation — not generation — becomes the bottleneck. This talk is about VLM-based "Visual Judges" as evaluation infrastructure, an emerging pattern relevant to anyone building perceptual-quality gates for generative systems.
• What: The argument that log files and traditional metrics (which solved LLM evaluation) break for visual media — a pipeline can have zero software errors and flawless prompt adherence yet fail perceptually (bad camera crop, stuttering motion, plastic skin). The work puts VLM-powered Visual Judges into the stack to drive a flywheel that accelerates media products. (from abstract only)
• How: Perceptual quality assessment via visual-language models acting as automated judges, feeding product and user signals back into model-recipe iteration.
• Where applicable: Meta's media-generation products. The VLM-as-judge pattern is broadly applicable wherever perceptual or subjective quality needs an automated gate; the specific judge training and calibration details are Meta's.