Analyst-Developer for the Neuro Quality Assessment Team

Search with Alice is not just about providing links: it creates detailed, structured responses with sections, pictures, and videos. But how do you know if the quality of these responses is good?

You could, for example, apply the classic approach - analyze user behavior. However, the modern internet has become so complex that often online metrics alone are not enough. Therefore, we approach the task comprehensively: we additionally build offline instruments that allow us to answer specific questions before experiments begin. Have the answers gotten better? How often do they contain serious errors? Do they match the queries?

You won't just be analyzing data, but creating rules and metrics that will become a 'quality detector' for responses.

What we strive for

• To create a new-generation search Not just a link provider, but an intelligent assistant capable of solving user tasks on the spot, without needing to go anywhere else.
• To answer not only in Russian We are launching in new regions, where we face challenges related to linguistic and regional specifics.
• To provide detailed answers Our goal is answers where text, video, and pictures work together. We make information come alive so it's memorable at first glance.
• Not to lie! No assumptions or 'creative' interpretations. We strictly ensure that answers are based on verified data, and every statement is backed by reliable sources. We teach models not to fantasize, but to rely on facts - even if it's harder.

It's great here because:

• We work on Search with Alice - a Yandex product based on LLM - and are primarily focused on production results.
• Our tasks are closely connected to both the product design itself and to ML.
• We provide opportunities to develop technical, communication, and management skills.
• Your work will directly influence what Search with Alice becomes in six months.
• We create crowdsource projects unique in complexity, scale, and architecture.
• We are a cohesive team of analysts and quality ML engineers.

What tasks await you

Giving clear form to product requirements Our key task is to formalize the initially abstract requirements of the product team into a set of clear rules and principles. These criteria allow us to objectively determine whether a model's response is good (suitable for the product) or bad (an error in the product), and to justify the decision. First, we develop these rules ourselves, analyzing examples and generalizing observations into instructions, then we teach them to AI trainers and assessors to see improvements in the model's responses in new versions.

Creating complex data labeling projects (crowdsourcing and LLM) Training modern models requires a huge amount of high-quality labeled data. We create projects for such labeling, engaging people through Yandex Crowd or using LLMs: we assemble a task (from instruction to interface), find performers, and train them. Each new task requires an understanding of system interrelationships, building complex architecture, and inventing new combinations of standard labeling approaches.

Improving quality, optimizing, and saving resources We regularly monitor the quality metrics of the obtained labels and look for growth points. To do this, we build detailed dashboards, configure data preparation pipelines, experiment with labeling schemes, and analyze query/response characteristics (topic, structure, etc.). Our task is not just to help the product become better, but to do so within given time or budget constraints.

We expect that you

• Can code in Python and SQL
• Know mathematical statistics and probability theory
• Enjoy working with data and know how to extract practical results from it
• Can interact with the team, clearly express thoughts, understand and persuade colleagues
• Are ready to delve into how and why a product should work

Will be a plus

• Have worked with Toloka or other crowdsource platforms
• Have written instructions and independently launched data labeling projects