Evaluation Scenario Writer

Vor 3 Tagen

Hamburg, Hamburg, Deutschland Mindrift Vollzeit 24.000 $ - 108.800 $ pro Jahr

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English.

At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. 

What we do

The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

About the Role

We're looking for someone who can design realistic and structured evaluation scenarios for LLM-based agents. You'll create test cases that simulate human-performed tasks and define gold-standard behavior to compare agent actions against. You'll work to ensure each scenario is clearly defined, well-scored, and easy to execute and reuse. You'll need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.

Although every project is unique, you might typically:

  • Designing structured test scenarios based on real-world tasks.
  • Defining the golden path and acceptable agent behavior.
  • Annotating task steps, expected outputs, and edge cases.
  • Working with devs to test your scenarios and improve clarity.
  • Reviewing agent outputs and adapting tests accordingly

How to get started

Simply apply to this post, qualify, and get the chance to contribute to projects aligned with your skills, on your own schedule. From creating training prompts to refining model responses, you'll help shape the future of AI while ensuring technology benefits everyone.

Requirements

  • Bachelor's and/or Master's Degreein Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
  • Background in QA, software testing, data analysis, or NLP annotation.
  • Good understanding of test design principles (e.g., reproducibility, coverage, edge cases).
  • Strong written communication skills in English.
  • Comfortable with structured formats like JSON/YAML for scenario description.
  • Can define expected agent behaviors (gold paths) and scoring logic.
  • Basic experience with Python and JS.
  • Curious and open to working with AI-generated content, agent logs, and prompt-based behavior.
  • You are ready to learn new methods, able to switch between tasks and topics quickly and sometimes work with challenging, complex guidelines.
  • Our freelance role is fully remote so, you just need a laptop, internet connection, time available and enthusiasm to take on a challenge.

Nice to Have

  • Experience in writing manual or automated test cases.
  • Familiarity with LLM capabilities and typical failure modes.
  • Understanding of scoring metrics (precision, recall, coverage, reward functions).

Benefits

Contribute on your own schedule, from anywhere in the world. This opportunity allows you to:

  • Get paid for your expertise, with rates that can go up to $52/hour depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.

  • Internship Engine Parts Engineering

    Vor 6 Tagen

    Hamburg, Hamburg, Deutschland Lufthansa Technik Vollzeit 35.000 € - 45.000 € pro Jahr

    Are you interested in aviation and manufacturing processes? Would you like to get to know and optimize processes?Then apply to us and gain exciting insights into the repair of engine components. After the internship, you will also have the opportunity to write your Bachelor's or Master's thesis.TasksAn internship in engineering for engine components offers a...

  • Revenue Growth Management Analyst

    Vor 5 Tagen

    Hamburg, Hamburg, Deutschland -fb29-444d-aa4d-5cdfc45d7ad1 Vollzeit 40.000 € - 60.000 € pro Jahr

    Job DescriptionLocation: HamburgContract terms: PermanentAbout UsFrom Arthur Guinness to Johnnie Walker, our business was founded on people of great character, and in 250 years, nothing's changed. We're the world's leading premium alcohol company. Our brands are industry icons. And our success is thanks to the strength of our people, in every role. It's why...

  • Product Application Engineer

    vor 1 Woche

    Hamburg, Hamburg, Deutschland Analog Devices Vollzeit 60.000 € - 120.000 € pro Jahr

    Come join Analog Devices (ADI) – a place where Innovation meets Impact. For more than 55 years, Analog Devices has been inventing new breakthrough technologies that transform lives. At ADI you will work alongside the brightest minds to collaborate on solving complex problems that matter from autonomous vehicles, drones and factories to augmented reality...

  • Product Application Engineer

    vor 1 Woche

    Hamburg, Hamburg, Deutschland Analog Devices Vollzeit 60.000 € - 120.000 € pro Jahr

    Come join Analog Devices (ADI) – a place where Innovation meets Impact. For more than 55 years, Analog Devices has been inventing new breakthrough technologies that transform lives. At ADI you will work alongside the brightest minds to collaborate on solving complex problems that matter from autonomous vehicles, drones and factories to augmented reality...

  • Subsititute Teacher Pdsd

    vor 1 Woche

    Hamburg, Deutschland Arizona State Schools for the Deaf and Blind Vollzeit

    JobID: 3515 **Position Type**: Teacher of the Deaf/HH/Substitute Teacher **Date Posted**: 7/25/2025 **Location**: Special Education Phoenix **Arizona State School for the Deaf and the Blind** **SUBSTITUTE TEACHER DEAF/HH** Location: Phoenix Employee Status: Temporary Hours Per Week: As needed Work Calendar: Flex Pay Rate: $21.88-$25.00/hr Other:...

  • Consumer Finance Business Partner

    Vor 6 Tagen

    Hamburg, Deutschland Imperial Brands Vollzeit

    **The Role** We are looking for a Consumer Finance Business Partner to join our dynamic team and act as the strategic co-pilot to the Global Consumer Office and Consumer Marketing teams. Reporting directly to the Finance Director Cluster DACH, this pivotal role supports proactive commercial collaboration and helps drive revenue growth and commercial margin...

  • Senior Operations Controller

    Vor 3 Tagen

    Hamburg, Deutschland The Quality Group GmbH Vollzeit

    The Operations Controller (gn) plays a critical role in ensuring operational excellence across our dynamic, multi-category nutrition business, including powders, liquids, bars, and vitamins. The role partners with operations, supply chain, R&D, and commercial teams to link operational performance with financial outcomes. The mission is to provide actionable...

  • Ch-47f Maintenance Test Pilot

    vor 1 Woche

    Hamburg, Deutschland Amentum Vollzeit

    **The CH-47F MTP -** **CH-47F** **Maintenance Test Pilot (CH-47F)** obtains a detailed knowledge of operating in the AOR airspace. Must be proficient flying in day VFR conditions. Maintenance Test Pilot performs routine acceptance test of aircraft following corrective maintenance. - ** CH-47F MTP** - Maintenance Test Pilot, responsible for all matters...

  • Ch-47 Maintenance Test Pilot

    vor 2 Wochen

    Hamburg, Deutschland Amentum Vollzeit

    **The CH-47 MTP -** **CH-47** **Maintenance Test Pilot (CH-47)** obtains a detailed knowledge of operating in the AOR airspace. Must be proficient flying in day VFR conditions. Maintenance Test Pilot performs routine acceptance test of aircraft following corrective maintenance. - ** CH-47 MTP** - Maintenance Test Pilot, responsible for all matters pertaining...

  • Sales Cloud Product Owner

    Vor 6 Tagen

    Hamburg, Deutschland MAM Gruppe Vollzeit

    Our customer is a proud Global leader with more than 70% market share in a specialised field of saving and extending people’s lives. With a product range that is the number 1 choice for professionals worldwide and a dedicated passionate team who have the vision to make the world a better place. Nominated for one of the top 100 global Innovative companies...

  • Senior Food Scientist

    vor 1 Woche

    Hamburg, Deutschland Ingredion Incorporated Vollzeit

    **About Ingredion**: Join Ingredion, where innovation impacts lives worldwide! Without even realizing it, you’ve experienced our work in your favorite chocolate, your pet’s food, the paper you write on, and your everyday snacks. As a global powerhouse across more than 60 industries, we harness the potential of natural ingredients to transform lives....

  • Working Students Meteorological Consulting

    vor 22 Stunden

    Hamburg, Deutschland Dornier Consulting Vollzeit

    **Dornier Suntrace GmbH**: - The - **Dornier Group** stands for innovative projects and exciting challenges and is a competent and reliable partner for complex consulting projects and demanding engineering services in a national and international environment. We bundle our expertise in the five business units Power and Heat, Nuclear Services, Renewables,...

  • Working Students Meteorological Consulting

    vor 22 Stunden

    Hamburg, Deutschland Dornier Group GmbH Vollzeit

    **Dornier Suntrace GmbH**: The **Dornier Group** stands for innovative projects and exciting challenges and is a competent and reliable partner for complex consulting projects and demanding engineering services in a national and international environment. We bundle our expertise in the five business units Power and Heat, Nuclear Services, Renewables, and...

  • Test Automation Engineer

    vor 3 Wochen

    Hamburg, Deutschland Soji AI UG Vollzeit

    About Us At Soji AI, we’re on a mission to transform the aviation industry with cutting-edge artificial intelligence. Our AI-powered ecosystem is designed to reduce aircraft downtime, improve safety, and make operations more efficient. Headquartered in Germany and Singapore, we’re an international team that thrives on radical transparency, continuous...