Video Quality Expert Group - Motivation, Objectives and Rules
1. MOTIVATION
VQEG was born from a need to bring together experts in subjective video quality assessment and objective quality measurement. The first VQEG meeting, held in Turin in 1997, was attended by a small group of experts drawn from ITU-T and ITU-R Study Groups. The general motivation of VQEG is to advance the field of video quality assessment by investigating new and advanced subjective and objective methods for assessing quality. However, with the exception of some recent contributions on subjective assessment methods for multimedia, VQEG has focused, in the last few years, its effort on the validation of new objective quality metrics for standardization purposes.
VQEG is open to all interested parties. There are no fees involved, no membership applications and no invitations are needed to participate in VQEG activities. Subscription to the main VQEG email list (ituvidq@its.bldrdoc.gov) constitutes membership in VQEG. VQEG activities, such as validation tests, are documented in reports and submitted to relevant ITU Study Groups (.e.g., ITU-T SG9, ITU-T SG12, ITU-R WP6C), and other SDOs as appropriate. Several VQEG studies have resulted in ITU recommendations.
2. OBJECTIVES
VQEG provides a forum, via email lists and face-to-face meetings for video quality assessment experts to exchange information and work together on common goals.
VQEG aims at providing a forum where algorithm developers and industry users can meet to plan and execute validation tests of objective perceptual quality metrics. Prior to the formation of VQEG, there was no recognized route for evaluating the predictive performance of objective video quality metrics. Over the years, VQEG has developed a systematic approach to validation testing which typically includes the collection of several subjective databases whose results are to be predicted by the objective video quality models under examination. An important element of the VQEG approach is the formulation of test plans that clearly and specifically define the procedures for performing objective model validation. These test plans describe the format and range of source content, the scope and nature of degradations that may be applied to the source content, the subjective test methods to be used to collect the subjective data, the test laboratories who perform the subjective assessment tests, the type of objective quality models that may be submitted for the purpose of model validation, the submission procedures of the objective quality models, and finally the statistical techniques and model evaluation metrics to be used. Importantly, the test plans are approved by consensus among all VQEG participants - including model proponents, subjective test labs, industry representatives, academia, and representatives of several Standards Developing Organizations (SDOs).
To ensure that the validation process is fair to all proponents of objective metrics and relevant for industry users, the test plans are executed with the collaboration (i.e., under the supervision) of VQEG's Independent Laboratory Group (ILG), which is composed of organizations that do not develop video quality models and, therefore, cannot benefit commercially or otherwise from the results of the validation tests.
As noted, there are no membership fees for participation in VQEG activities. In the beginning, there were also no fees for participating as proponents in validation tests; this was done in order to allow all organizations, irrespective of size or budget, to be able to submit objective models. Through time this no-fee validation position has conflicted with the need to have, whenever possible, subjective assessment datasets produced by members of the ILG. Demands on the ILG are considerable and inevitably this has led to the charge of a small fee made on model proponents for validation to partly compensate for the effort of the ILG members.
It is also the ambition of VQEG to produce open source databases of video material and test results, as well as software tools. The output from VQEG's first validation test was publicly released, including the source and processed test materials and the subjective tests data for each test condition. This freely available set of annotated test materials has become an important resource for model developers, who use these materials for training, testing and refining their models. These materials are also used in comparative studies that are commonly reported in the video quality measurement literature. Unfortunately, VQEG has been unable to release test materials from some validation tests due to the use of copyrighted video sources. VQEG has been forced to proceed with validation tests using materials that have copyright restrictions, since open source video material has been difficult to obtain.
Besides validation of objective measurement methods, VQEG conducts subjective studies of multimedia and 3DTV and provides a place for collaborative model development to take place.
3. RULES
3.1 Representation
3.1.1. VQEG distinguishes three types of organizations: Proponents, Independent Laboratories, and third party organizations.
3.1.2. Chair, Vice Chair and Co-Chair positions are assigned to individual people, regardless of their employment status within a participating organization.
3.1.3. All other roles, commitments, and responsibilities are assigned to organizations. This includes, but is not limited to, voting rights and access to data/information collected by VQEG.
3.1.4. To become a member of the ILG that is implementing one particular test plan, that organization (1) must perform a non-trivial amount of work implementing the test plan, (2) must not submit a model, and (3) cannot have any relationship with a proponent that would indicate a conflict of interest. A "non-trivial amount of work" includes but is not limited to HRC creation, donating SRC, coordinating implementation of the test plan, data analysis, subjective testing, scene selection, model receipt & validation, experiment design, calibration checks, and drafting portions of the final report. The associated Chairs (i.e., VQEG, ILG, and Project Group) decide whether a particular organization is an ILG for a given test plan. This decision will be made for each project.
3.1.5. Organizations who are neither proponents nor the ILG implementing one particular test plan (or do not expect to be so) are encouraged to contribute in other meaningful ways to the effort. This includes but is not limited to drafting the final report, reaching agreement on the test plans, voting during face-to-face meetings, providing the ILG with special knowledge of video service parameters.
3.1.6 Unsolicited advertising, dispatching of brochures, public product presentations etc. are not allowed at VQEG meetings. Excepted from this are of course private discussions between participants, excursions to/through the meeting hosts facilities and invited presentations. Announcements of conferences and related meetings of interest to VQEG members are encouraged at meetings.
3.2 Voting Procedures
3.2.1. Every effort is made to come to consensus on important issues before VQEG. Co-Chairs will often call for an informal show of hands to determine the feeling of the room on a topic. If necessary, a formal vote can be called by a Chair or Vice Chair of the Group or a Co-Chair of VQEG to come to a decision. All decisions must be recorded in the minutes. Decisions from formal votes will include the name and vote of each organization recorded in the minutes.
3.2.2. When votes are required, each organization can only cast one vote.
3.2.3. Unless otherwise (and previously) agreed upon, overturning an existing decision requires a qualified majority of 2/3 of the available votes.
3.2.4. Voting can be done at face-to-face VQEG meetings, via email, or during an audio call.
3.2.5. Any decision reached during an audio call takes effect 1-week (5 business days) after the audio call meeting minutes are posted. People who could not attend the audio call have 1-week to object to decisions reached by posting an email to the relevant VQEG reflector. The Co-Chairs will decide whether the objection warrants a re-vote.
3.3 Use of Data and Material
3.3.1. Private VQEG data are data whose distribution is restricted to a subset of VQEG members. Public data are data open to all VQEG members and the public at large.
3.3.2. The objective data produced by an objective model are considered private until that model appears in an official VQEG report. When VQEG publishes a report that analyzes objective models, the objective data for those models will either remain private or be published (and thus become public), as specified in the test plan. Private objective data can be used or made public only with the explicit consent of the proponent organization that legally owns the objective model.
3.3.3 A proponent may withdraw their model after examining their model's performance. In this case, the VQEG report will include only anonymous mention of the model's existence (e.g., identifying the number of models that were submitted and then withdrawn). That model's objective data will remain private.
3.3.4. The subjective data produced within the context of a specific VQEG project are considered private until that project publishes a report. When the report is published, the subjective data will either remain private or be published (and thus become public), as specified in the test plan. Private subjective data can be used only by the organizations participating in that project, including the proponents of the models under validation and the ILG laboratories implementing the test plan (see & 3.1.4.) Acceptable use of these data includes all uses specified in the project's test plan, as well as uses related to R&D work and scientific communication (e.g., academic publication).
3.3.5. Third party organizations may or may not be allowed access to private VQEG data, depending upon the exact wording of the relevant test plan.
3.3.6. Subjective data produced by an organization outside the scope of a specific project cannot be used without the explicit consent of that organization.
3.3.7. Private VQEG audio/video material is the audio/video material whose distribution is restricted via verbal or written agreements (e.g., SRC video material). Any use of private audio/video material must conform to the requirements specified by the corresponding agreements. Unless otherwise specified, all the audio/video material used by VQEG is to be considered private.
3.4 Use of Email Reflectors
3.4.1. VQEG has established one 'general' email reflector and several 'specific' email reflectors. The general reflector is destined for general discussions of interest to VQEG members, e.g., on audio and video quality measurements. The specific reflectors are to be used only for purposes related to the specific test project they were built for. All reflectors are moderated.
3.4.2. Offensive and threatening language will be punished by removing access to all reflectors.
3.4.3. The advertising of commercial product in any of the VQEG reflectors is not permitted. It is deemed acceptable to ask information about commercial products, as long as the responses are requested and made privately.
3.4.4. The advertising of job positions, which might be of interest to VQEG members, is permitted.
3.4.5. The advertising for conference, journals, or other scientific endeavors, which might be of interest to VQEG members, is permitted.
3.5 Working Methods
3.5.1. VQEG work is organized into Projects which are managed by a VQEG Group (formerly called Ad-Hoc Groups). A Group or Project can be established by consensus at a face-to-face meeting. Proposals for a Group or Project should be submitted as a contribution to a meeting. There should be at least three organizations that intend to devote resources to a proposed Project. Normally there is only one active Project in a Group (e.g., FRTV-1, FRTV-2).
3.5.2. The VQEG working group structure is intentionally dynamic. The expectation is that groups will “die” quickly if inactive, projects are reviewed at each meeting and either confirmed or closed due to inactivity.
3.5.3. The Chair is the point-of-contact for each working group, “one throat to choke”, responsible for body of work, responsible for reporting to VQEG co-chairs during meeting organization, responsible for driving the work.
3.5.4. Vice-chairs provide support for chair, major contributor, step in if chair is unable to perform work
3.5.5. If a working group chair or vice-chair is inactive for two meetings, then the role will be reassigned. For example, if the working group chair does not attend two consecutive face-to-face meetings (i.e., neither remote nor in person attendance), or is unresponsive to emails coordinating the VQEG meeting agenda.
3.5.6. Chairs and vice-chairs can create named roles within their groups, if needed, for special responsibilities (e.g., project leader); and the role is mentioned on the VQEG website.
3.5.7. New working group proposals need a working group title (name), a chair person (someone that wants to drive), and a description for the web about the group.
3.5.8. The collection of Chairs, Vice Chairs and Co-Chairs compose the VQEG Board. The Board is generally consulted when a decision must be made that does not require consultation with VQEG as a whole on the general email list and cannot wait until the next face-to-face meeting or audio call. An example is the agenda for an upcoming meeting.
3.6 Face-to-Face Meetings
3.6.1. VQEG generally has face-to-face meetings twice a year where major decisions are made. These meetings are free to attend and are usually held alternately in Europe, North America, and Asia and last 5 days. There are usually ITU-T Rapporteur meetings held at the same time and location as part of the VQEG meeting. VQEG meetings are held where a host can provide a room to meet and internet access. Some consideration is given to ease and cost of travel and accommodations.
3.6.2. Meeting minutes will be written on GoogleDocs, so that all participants can help write meeting minutes.
3.6.3. VQEG desires that the host has remote participation capability. Prospective hosts should find out whether they can support remote participation and indicate the mechanism before meetings (e.g., Hangouts, Skype). This information will be part of the decision on meeting location.
3.6.4. Goal: establish meeting schedule by 2 months before meeting.
3.6.5. Goal: establish meeting date & location two meetings in advance (roughly 1 year).
3.6.6. The last hour on Friday meeting is devoted to establishing the draft agenda for the next face-to-face meeting.