Return-Path: Delivered-To: alvarochf@traceseis.com Received: from cs510.bluehost.com by cs510.bluehost.com with LMTP id uL5ECA4NlWksHwUAWiBQxw (envelope-from ) for ; Tue, 17 Feb 2026 17:51:26 -0700 Return-path: Envelope-to: alvarochf@traceseis.com Delivery-date: Tue, 17 Feb 2026 17:51:26 -0700 Received: from mx-out-01b.sea3.discourse.cloud ([184.105.99.122]:41161) by cs510.bluehost.com with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1vsVmo-00000001PUk-0F4Z for alvarochf@traceseis.com; Tue, 17 Feb 2026 17:51:26 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=discoursemail.com; s=sea3; t=1771375874; bh=4JEt6gDThbO8+F2/+Bo67G70iXorxcdsjnRW3aIlsCw=; h=Date:From:Reply-To:To:Subject:List-Unsubscribe:List-ID: List-Archive; b=BfhIDwTgPD2AJhkFwrl3FdhHtVn1EUB4NW4ji8qA1MAjCHNIDPF6zMf2LOXiaWq7+ 4Oz0c/hXcAwRcCrlKKC1Dv5ko7a2q/VreWNfDOntOMhKSjvR7g7rKeU/8nG0wKFy6s R58wqyAjxwhMKZnOu0ZIx9p3JfmwC2Apg897SFvdBgZ/CDSGOaN1pa9wlmijBLP5+6 x0LqXZlHIyGeP8lbLMH8WiVRpA/jl/HcVCuNoI5/uOwbY8ObI5Vg1Llhj0AHi2S42u m1AGEKbQ6GIicaFlayBH8qG3CLTfCnzkZwSGfM7gUbAv8MZ44Ez1WiSGBcwtuKErZJ HHFiuG6RQ+Trw== Received: from localhost (unknown [IPv6:2602:fd3f:3:301:0:242:ac11:21]) by mx-out-01b.sea3.discourse.cloud (Postfix) with ESMTP id D5F6F2A00030 for ; Wed, 18 Feb 2026 00:51:14 +0000 (UTC) Date: Wed, 18 Feb 2026 00:51:14 +0000 From: Josafat P Reply-To: Outlier Community To: alvarochf@traceseis.com Message-ID: Subject: =?UTF-8?Q?[Outlier_Community]_[Aether_Approved_Annotators/Project?= =?UTF-8?Q?_Announcements]_=F0=9F=93=A2_Guidelines_for_the_Evals_Projects_-?= =?UTF-8?Q?_h2h_Evaluation?= MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="--==_mimepart_69950d02d0784_3b4f5ae894e7423968c4"; charset=UTF-8 Content-Transfer-Encoding: 7bit List-Unsubscribe: List-Unsubscribe-Post: List-Unsubscribe=One-Click X-Discourse-Post-Id: 8273724 X-Discourse-Topic-Id: 2869021 X-Discourse-Tags: official urgent X-Discourse-Category: Aether Approved Annotators/Project Announcements X-Discourse-Sender: Josafat_P_QM X-Auto-Response-Suppress: All x-ms-reactions: disallow Auto-Submitted: auto-generated Precedence: list List-ID: Outlier Community | Aether Approved Annotators Project Announcements List-Archive: https://community.outlier.ai/t/guidelines-for-the-evals-projects-h2h-evaluation/2869021 Feedback-ID: outliercommunity:user_watching_first_post:discoursemail ----==_mimepart_69950d02d0784_3b4f5ae894e7423968c4 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi team! :smile: In the Multimango platform you will evaluate and compare AI-generated out= puts across multiple media types, including audio, video, image, text, an= d Vision Language Model responses. = These projects can be identified by the **"h2h"** label, indicating that = the task involves comparing two outputs head-to-head. We have just uploaded the **Official Guidelines** for these projects. Thi= s document will be extremely helpful while completing your tasks, so plea= se make sure to **review it carefully**. You can find the link at the bot= tom. --- # :warning: Main Rule for ALL Evaluations Annotators should **PRIORITIZE instruction following** and **realism/natu= ralness** above all other qualities when deciding which media is better. :memo: For example: = > If Media A appears to be higher quality but does **not** follow instruc= tions as good as Media B, then **Media B should be selected**. = > >Resolution alone should not outweigh instruction following unless it is = **explicitly requested in the prompt**. Outputs that look obviously **AI-generated** or **unnatural** should not = be preferred. Common issues to watch for include: * Overly smooth or artificial textures * Unrealistic lighting or proportions * A =E2=80=9Crendered=E2=80=9D or synthetic appearance :prohibited: An AI-generated look does **not** equal better quality. :prohibited: A visually appealing image does **not** automatically mean i= t is better quality. We previously posted an [announcement](https://community.outlier.ai/t/cri= tical-feedback-for-evals-projects-h2h/2865599) for this. --- # :bookmark: [h2h Evaluation Guide](https://app.outlier.ai/en/expert/guid= elines/68dd6e9b668d6876a63ab1d4?componentId=3D6994e57dd486b50a12f17e7a&ty= pe=3Dattachment) :backhand_index_pointing_left: --- [Visit Topic](https://community.outlier.ai/t/guidelines-for-the-evals-pro= jects-h2h-evaluation/2869021/1) or reply to this email to respond. To unsubscribe from these emails, [click here](https://community.outlier.= ai/email/unsubscribe/56da0c2a47db6ed612137920921fdd3fb01d81196c1e7bdb9335= 8b3ca76fdb3d). ----==_mimepart_69950d02d0784_3b4f5ae894e7423968c4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable = =
= =
Josafat_P_QM
February 17

Hi team! 3D":smi=

In the Multimango platform you will ev= aluate and compare AI-generated outputs across multiple media types, incl= uding audio, video, image, text, and Vision Language Model responses.
= These projects can be identified by the =E2=80=9Ch2h=E2=80=9D label, indicating that the task involves
comparing two outputs head-to-head.

We have just uploaded the Offi= cial Guidelines for these projects. This document will be extrem= ely helpful while completing your tasks, so please make sure to r= eview it carefully. You can find the link at the bottom.

3D":warning:" Mai= n Rule for ALL Evaluations

Annotators should PRIORITIZE i= nstruction following and realism/naturalness ab= ove all other qualities when deciding which media is better.

3D":m= For example:

If Media A appears to be hig= her quality but does not follow instructions as good as = Media B, then Media B should be selected.

Resolution a= lone should not outweigh instruction following unless it is expli= citly requested in the prompt.

Outputs that look obviously AI= -generated or unnatural should not be preferred= . Common issues to watch for include:

  • Overly smooth or artificial textures
  • Unrealistic lighting or proportions
  • A =E2=80=9Crendered=E2=80=9D or synthet= ic appearance

3D":prohibited:" An A= I-generated look does not equal better quality.
3D":prohibited:" A visually appealing image does not automatically mean it is better quality.

We previously posted an announ= cement for this.

3D":bookmark:" h2h Evaluation Guide 3D":backhand_in=

Visit Topic or reply to th= is email to respond.

To unsubscribe from these= emails, click here.

                  &nbs= p;                   &nbs= p;                   &nbs= p;
= = = = ----==_mimepart_69950d02d0784_3b4f5ae894e7423968c4--