To comprehensively evaluate the generated creatives, we design a multi-aspect evaluation suite covering video quality, personalization, product alignment, integration quality, and diversity. For video quality, we adopt Motion Smoothness (MS), Dynamic Degree (DD), Aesthetics Quality (AQ), and Imaging Quality (IQ), as suggested in VBench to assess basic video fidelity. The remaining aspects require multimodal understanding and reasoning, which are hard to capture with conventional quantitative metrics, while human evaluation is costly and difficult to reproduce. Therefore, we leverage the state-of-the-art VLM Gemini 2.5-Flash as an automatic judge: it scores each criterion on a 0-10 scale based on predefined rules and simultaneously provides detailed rationales to support its judgments.
Specifically, for personalization, we present the user’s historically interacted content alongside the generated creative to assess alignment in presentation (e.g., tone, artistic style) and content (e.g., topics, scenes, appeal). For product alignment, we provide product information with the creative to evaluate product identity, brand elements, selling points, and factual consistency. For diversity, we assess similarity among all creatives generated for the same user in visual style, narrative structure, topics, and scenes. For integration quality in the PCI task, we insert the generated creative into the corresponding host video and evaluate contextual flow, thematic consistency, and user experience disruption.
Video QualityMS · DD · AQ · IQ
PersonalizationPresentation · Content
Product AlignmentIdentity · Brand · Selling points · Facts
Integration QualityContextual flow · Theme · Disruption
DiversityStyle · Narrative · Topics · Scenes