Subjective and objective assessment of GenAI content (SOGAI)
This group is investigating subjective methods to evaluate the content produced by generative AI methods.
Related Documents
- Meeting minutes and literature review (GoogleDoc)
Description
The rapid advancement of Generative AI (GenAI) has introduced novel ways of creating video, audio, and image content, opening up transformative opportunities across various applications and industries. GenAI-generated content differs significantly from traditional, camera-captured media, necessitating new approaches for evaluating its quality, fitness for purpose, and user acceptability.
To address these needs, this initiative seeks to standardize both subjective testing methodologies and objective metrics for assessing the quality of GenAI-generated content. Subjective evaluation requires rethinking traditional methodologies, questionnaires, and statistical analyses to accommodate the unique attributes of GenAI content. For example, questions like “Rate the quality of this video” must be expanded to address specific aspects such as translation accuracy, lip-sync quality, style adherence, prompt consistency, and overall plausibility. Furthermore, defining reliable objective metrics as proxies for subjective evaluations is critical, given the high scale and evolving nature of GenAI content production.
The scope of this effort encompasses a range of GenAI paradigms. For example:
-
Prompt-Driven Content Creation: Generating entirely new videos based on user prompts (e.g., “Create a video of a cat dancing on ice”) or transforming existing media to align with specified styles or themes.
-
Automatic dubbing: Translating videos between languages while ensuring accurate lip-sync and voice matching.
-
Style Transfer: Modifying existing videos to adopt the style of a given image or video.
This initiative also acknowledges the challenges posed by subjective testing for GenAI content, such as diverse user expectations, the inherent subjectivity of evaluating “prompt adherence,” and the varying nature of questions across use cases. Engaging the broader GenAI community, including researchers and practitioners contributing to conferences like CVPR and ICIP, will be essential for ensuring alignment and adoption of standardized methodologies.
By establishing robust frameworks for subjective and objective evaluation, this group aims to support the reliable assessment of GenAI content, fostering its broader adoption and enhancing user trust in AI-generated media.