No Reference Metrics (NORM)
The No Reference (NR) Metrics group (NORM) is an open collaborative for developing NR metrics for monitoring visual service quality. NORM encourages knowledge sharing on all aspects of NR metric research and development. Current projects include:
- Open source NR metric
- SI and TI clarification and improvement
- Video quality metadata standard
NORM coordinates work using the VQEG reflector email@example.com
Conference calls are announced on the NORM reflector and listed on the main VQEG webpage.
#1) NR Metric Development—What Is Our Design Goal?
NR metrics use the decoded video at the point of measurement to estimate:
- Mean opinion score (MOS), to assess the overall quality
- Root cause analysis (RCA), to identify the impact of specific visual impairments
We seek a broad scope: camera capture, encoding, decoding, transcoding, scaling, transmission, aesthetics and artistic intent, image enhancement, monitor, and display device.
Users must be able to modify the scope to ignore specific impairments. For example, broadcasters want MOS to ignore artistic intent and aesthetics. The omitted factors should not influence MOS.
The performance goals are robust accuracy with a minimum of operational restriction. Use cases include Video on Demand (VoD), live broadcast services, social media, first responder video, medical, and AI vision systems (autonomous vehicles). Access to the bitstream is beneficial, but not required.
NORM is an open collaborative group that intends to make all of its work public, royalty free.
- No reference resources here.
- This Google Sheet coordinates our work to identify metric components.
- This presentation from 2021 identifies datasets for training NR metrics.
#2) SI and TI Clarification
The ITU-T Rec. P.910 definitions of SI and TI contains ambiguities. Our goal is to eliminate implementation specific inconsistencies. A revised definition would ideally ensure:
- Stable output for the same sequence if converted between limited and full range
- Handling of higher bit depth formats while always keeping output in 0-255 range
- Consistent definition of temporal measurement on the first frame of the sequence
- Availability of reference implementation and test vectors to verify correctness of implementations
Once agreement has been reached, our proposal will be submitted to ITU-T Study Group 12, to revise P.910.
This group is also interest in developing an improved version of SI/TI that assesses the coding complexity of videos. Design specifications include:
- Very light weight algorithm (low coding complexity)
- Estimates a curve that relates coding complexity to bit-rate
- Includes motion estimation
#3) Video Quality Metadata Standard
Full reference video quality metrics are readily available in most
modern transcoding pipelines. Including full-reference video quality metrics as metadata in compressed bitstreams would take very little space and provides a more accurate and “green” way of estimating source video quality.
To realize this vision, we must establish a standard format to save such metadata at both elementary video bitstream level and system layer. Both hardware (device) makers and service providers have a lot to gain by offering such metadata in their compressed bitstreams.
NR metrics would still be needed in situations like the following:
- In the camera front-end, to estimate quality of raw input
- Legacy content (video quality metadata unavailable)
- Some video broadcasting applications (e.g., transmission over
- Non-transcoding image/video applications (e.g., editing, image enhancement)