Reach out directly about this role
Analyst-Developer for the Neuro Quality Assessment Team
Search with Alice is not just about providing links: it creates detailed, structured responses with sections, pictures, and videos. But how do you know if the quality of these responses is good?
You could, for example, apply the classic approach - analyze user behavior. However, the modern internet has become so complex that often online metrics alone are not enough. Therefore, we approach the task comprehensively: we additionally build offline instruments that allow us to answer specific questions before experiments begin. Have the answers gotten better? How often do they contain serious errors? Do they match the queries?
You won't just be analyzing data, but creating rules and metrics that will become a 'quality detector' for responses.
What we strive for
It's great here because:
Giving clear form to product requirements Our key task is to formalize the initially abstract requirements of the product team into a set of clear rules and principles. These criteria allow us to objectively determine whether a model's response is good (suitable for the product) or bad (an error in the product), and to justify the decision. First, we develop these rules ourselves, analyzing examples and generalizing observations into instructions, then we teach them to AI trainers and assessors to see improvements in the model's responses in new versions.
Creating complex data labeling projects (crowdsourcing and LLM) Training modern models requires a huge amount of high-quality labeled data. We create projects for such labeling, engaging people through Yandex Crowd or using LLMs: we assemble a task (from instruction to interface), find performers, and train them. Each new task requires an understanding of system interrelationships, building complex architecture, and inventing new combinations of standard labeling approaches.
Improving quality, optimizing, and saving resources We regularly monitor the quality metrics of the obtained labels and look for growth points. To do this, we build detailed dashboards, configure data preparation pipelines, experiment with labeling schemes, and analyze query/response characteristics (topic, structure, etc.). Our task is not just to help the product become better, but to do so within given time or budget constraints.
3-5 years
Experience
Full-time
Employment
Hybrid
Work Format
Data Analytics
Specialization
IT & Tech
Industry
Corporation
Company Type
IT & Tech
Industry
Corporation
Company Type