Data Science Evaluator - AI Work Quality
Posted date [2026-08-19]
(ID: 49856)
 |
YO IT Consulting specializes in recruiting remote professionals for AI training and data annotation projects with global companies. Based in Abu Dhabi Emirate, United Arab Emirates, the company connects qualified talent worldwide with flexible remote opportunities that contribute to the development and improvement of modern artificial intelligence systems. |
Job Description and Requirements
Data Science Evaluator - AI Work QualityJob Snapshot
Role: Data Science Evaluator - AI Work Quality
Location: Abu Dhabi Emirate, United Arab Emirates
Industry: IT and Services
Function: Science / R&D
Experience: Minimum 5 years
Job Type: Contractor
Position Overview
The Data Science Evaluator - AI Work Quality opportunity in Abu Dhabi Emirate, United Arab Emirates is a remote contractor role within IT and Services offered through YO IT Consulting. The evaluator will establish professional quality standards for realistic data science assignments and determine how effectively AI systems and human contributors meet them.
Job Details
Country: United Arab Emirates
City: Abu Dhabi Emirate
Industry: IT and Services
Function: Science / R&D
Salary: 22000-38000
Estimated salary range based on similar jobs in Abu Dhabi Emirate; please confirm the final offer with the employer.
Gender: Any
Candidate Nationality: Any
Job Type: Contractor
Role Context
The assignment focuses on defining and measuring excellent
data science work rather than completing conventional business deliverables. The evaluator will create scoring criteria for specific tasks and apply them to analyses, statistical models, dashboards, experiment summaries, and written recommendations.
Each judgement must be grounded in evidence and explained with enough precision for another qualified reviewer to understand and reproduce the result.
Key Responsibilities
* Analyze realistic data science tasks and identify their essential quality requirements.
* Design detailed grading criteria tailored to the objective and expected output of each assignment.
* Define scoring levels for analytical correctness, methodology, completeness, and business usefulness.
* Review AI-generated and human-produced data science work against approved criteria.
* Score analyses, models, dashboards, experiment readouts, and recommendation documents.
* Provide a clear written justification supporting every assigned score.
* Verify whether analytical methods are appropriate for the available data and business question.
* Examine metric definitions, assumptions, calculations, and interpretation for accuracy.
* Assess experiment design, sample selection, control groups, and A-B testing conclusions.
* Review SQL and Python work for technical correctness and reproducibility.
* Evaluate whether dashboards communicate relevant information without misleading users.
* Determine whether recommendations are supported by evidence and suitable for executive audiences.
* Apply standards consistently across comparable tasks and submissions.
* Identify ambiguous criteria and propose refinements that improve reviewer alignment.
* Incorporate structured feedback from senior reviewers and revise assessments promptly.
* Participate in calibration activities to align scoring decisions with other experts.
* Maintain concise records of assumptions, evidence, and evaluation decisions.
* Protect project information and comply with contractor confidentiality requirements.
Ideal Profile
Candidates should have at least five years of professional data science experience in an industry environment. Experience supporting product, growth, or business-operations decisions within a leading technology company is particularly relevant.
Advanced capability in experiment design, A-B testing, metric development, SQL, Python, and statistical analysis is required. Applicants must also be able to communicate technical findings clearly to executive stakeholders.
The role suits a detail-focused professional who is comfortable receiving critical review and calibrating judgement with peers. Previous work involving AI training, model evaluation, human feedback, or data-quality projects would be beneficial.
Skills Set
* Data science quality evaluation
* Task-specific rubric design
* Evidence-based scoring
* AI output assessment
* Statistical analysis
* Experiment design
* A-B testing
* Metric development
* SQL analysis
* Python analysis
* Predictive model review
* Dashboard quality assessment
* Product analytics
* Growth analytics
* Business operations analytics
* Executive recommendation review
* Reproducibility assessment
* Written score justification
* Reviewer calibration
* Analytical quality assurance
Why Join Us
This remote contract enables experienced data scientists to influence how advanced AI systems are evaluated on genuine professional work. Contributors can apply analytical judgement across varied scenarios, maintain a flexible schedule, and receive weekly payments through Stripe or Wise for completed services.
About the Company
YO IT Consulting is supporting a specialist engagement with an AI research organization. The project brings experienced data scientists into structured evaluation work that helps improve the accuracy, reasoning, and professional usefulness of AI-generated analytics.
Each judgement must be grounded in evidence and explained with enough precision for another qualified reviewer to understand and reproduce the result.
Key Responsibilities
* Analyze realistic data science tasks and identify their essential quality requirements.
* Design detailed grading criteria tailored to the objective and expected output of each assignment.
* Define scoring levels for analytical correctness, methodology, completeness, and business usefulness.
* Review AI-generated and human-produced data science work against approved criteria.
* Score analyses, models, dashboards, experiment readouts, and recommendation documents.
* Provide a clear written justification supporting every assigned score.
* Verify whether analytical methods are appropriate for the available data and business question.
* Examine metric definitions, assumptions, calculations, and interpretation for accuracy.
* Assess experiment design, sample selection, control groups, and A-B testing conclusions.
* Review SQL and Python work for technical correctness and reproducibility.
* Evaluate whether dashboards communicate relevant information without misleading users.
* Determine whether recommendations are supported by evidence and suitable for executive audiences.
* Apply standards consistently across comparable tasks and submissions.
* Identify ambiguous criteria and propose refinements that improve reviewer alignment.
* Incorporate structured feedback from senior reviewers and revise assessments promptly.
* Participate in calibration activities to align scoring decisions with other experts.
* Maintain concise records of assumptions, evidence, and evaluation decisions.
* Protect project information and comply with contractor confidentiality requirements.
Ideal Profile
Candidates should have at least five years of professional data science experience in an industry environment. Experience supporting product, growth, or business-operations decisions within a leading technology company is particularly relevant.
Advanced capability in experiment design, A-B testing, metric development, SQL, Python, and statistical analysis is required. Applicants must also be able to communicate technical findings clearly to executive stakeholders.
The role suits a detail-focused professional who is comfortable receiving critical review and calibrating judgement with peers. Previous work involving AI training, model evaluation, human feedback, or data-quality projects would be beneficial.
Skills Set
* Data science quality evaluation
* Task-specific rubric design
* Evidence-based scoring
* AI output assessment
* Statistical analysis
* Experiment design
* A-B testing
* Metric development
* SQL analysis
* Python analysis
* Predictive model review
* Dashboard quality assessment
* Product analytics
* Growth analytics
* Business operations analytics
* Executive recommendation review
* Reproducibility assessment
* Written score justification
* Reviewer calibration
* Analytical quality assurance
Why Join Us
This remote contract enables experienced data scientists to influence how advanced AI systems are evaluated on genuine professional work. Contributors can apply analytical judgement across varied scenarios, maintain a flexible schedule, and receive weekly payments through Stripe or Wise for completed services.
About the Company
YO IT Consulting is supporting a specialist engagement with an AI research organization. The project brings experienced data scientists into structured evaluation work that helps improve the accuracy, reasoning, and professional usefulness of AI-generated analytics.
